At some point as a photographer, if you're going to pursue this as more than just a hobby you indulge in every now and then, you have to shrug off the self consciousness of "I'm shooting public, everyone is looking at me!" and just do it. It's really nothing more than cognitive behavioral therapy, the more you do it, the less it will bother you and cause you anxiety, and you'll see it's not as scary as you think.
I don't carry my camera with me everywhere I go, and I don't usually just drive around and shoot whatever because I don't really shoot landscapes, or wildlife. My shoots are usually planned to fit in with my schedule, but I do take a lot of pictures with my iPhone 4 when I'm just hanging out or doing a "day in the life" sort of thing. If you're into landscapes, urban cityscapes/life, street portraiture, I definitely think it's important to get in the habit of carrying a camera and capturing what you see, what draws your eye. Photography is like any other skill, no matter how innately good you are at it, you still need to practice, practice, practice. If carrying your camera around and photographing on the spur of the moment is just too much, SCHEDULE time to go shooting, find a spot that appeals to you, and go to town, even if it's just for an hour each weekend.
HTH!