Depends on the weather and what you are looking for. On a very cloudy day the time won't matter very much, and I prefer the light any time on a cloudy day to the early morning on a sunny day. If you want dramatic shadows you're better off going nearer the middle of the day, although the shadows are longer at the beginning and end of the day. You should try different times and conditions and see what works for you.
I sometimes start shooting within 2 to 3 minutes of rolling off the rack. I often classify these as "sunrise" shots. At times, I shoot photos within just one or two minutes of falling asleep. I classify these as "late night, drunk shots". And in between the early AM and the late-night stuff, I classify everything else as "daytime shots". I do however, like the Golden Hour, when the light looks as sweet and golden as a finely-brewed glass of beer.
I can't really do the sunrise ones, hubby is out to work early and bairn is still asleep, garden isn't that interesting either!! ...well maybe on a Sunday I can pop out for a bit early on if I'm not too tired from my Saturday night out
Here is a nice website that calculates the "golden hour" for your location every time you visit. Useful for planning when to get up /set up
If you hover your mouse over the yellow bands in the graph at the bottom it will tell you when the golden hour starts and ends.