Do what you want and think you can do.
But remember, Job #1 is the guy paying you a wage, probably providing a company car and gas. Their position is that you should not be using their resources (time they pay for, equipment they provide) for another job.
Just understand the risk before you do anything ... because you may lose at least the job that provides the benefits, car, and perhaps the majority of your income.
