should I take off the polarizing filter when shooting in early moring light and late afternoon light?
Should I take off the filter when shooting on grey overcast days?
It depends what you're shooting, and whether or not it has characteristics of polarized direct reflection.
When shooting something like landscapes, a polarizing filter will be capable of it's maximum effect when shot at 90 degrees to the suns rays.
In a situation with grey, overcast skies, you probably won't see a difference is the sky, but you may still see a difference on things like glass or water etc.
Keep it on hand, and try it in all situations. You'll see where it will work and where it won't. After a while, you'll know without having to try it every time.
Personally, if it's on the lens, I won't take it off even if I know it's not going to have an affect for that particular shot. I avoid low-end filters so what negative affects it has on the image are pretty slim. It's just not worth it to be taking it off and putting it back on every 5 shots.
Now, if I know that I'll be doing a whole "shoot" and will not need it, then yeah, I'll absolutely take it off. Or if I need the extra shutter speed, then yeah, I'll take it off.
Personally i would only use it when I found it was going to assist me in creating the photo. I.E. removing glare from reflections, enhancing skies, etc.
You lose about 2 fstops of light which may or may not be helpful if not gaining the benefits.
Mike has given you some good tips of when and how, the rest is up to you.