I've been a photographer so long I don't remember starting out, other than I've always loved roaming around taking pictures and if you don't love it don't do it.
I remember the first time I went out on the ice pregame at a hockey game to take pictures of the ceremonial puck drop. I was nervous-excited and all I could think was don't let me fall on my a$# in front of how many thousand people! (The hockey gods watched over me, I didn't.)
I've had kids at games wave and ham it up, I've had drunks at games wave and show off. I've had players shoot pucks at me (and laugh), mascots wave at me (and I'd pretend to take their picture since they were right in front of me not realizing I'm using a telephoto lens). In general I get people looking up/around, looking at me, like they aren't quite sure what I'm photographing.
I think you just get used to it. I've more recently gotten some looks/double takes when I've fired off a Polaroid!
edit - As far as safety, for the type of job I had we did safety training since we went out on home visits in all kinds of neighborhoods (including the ones that are on the news with drug related shootings, etc.). Maybe try to find personal safety techniques to use, like scanning the area before you get out of the car. My personal suggestion is if it looks like a crack house it probably is, just keep going... Seriously, if it looks like anything's going on or something doesn't seem right, find someplace else or another time, it's not going to be worth it.