It depends what you're shooting, and whether or not it has characteristics of polarized direct reflection.
When shooting something like landscapes, a polarizing filter will be capable of it's maximum effect when shot at 90 degrees to the suns rays.
In a situation with grey, overcast skies, you probably won't see a difference is the sky, but you may still see a difference on things like glass or water etc.
Keep it on hand, and try it in all situations. You'll see where it will work and where it won't. After a while, you'll know without having to try it every time.