Are you talking about the piece you put to your eye, or the mirror assembly? If its just the piece that you put to your eye then yes, clean it with a q tip or a soft cloth or whatever really. If its the mirror assembly that you're meaning, then no, stay away from it.
You can also gently brush the mirror and the focusing screen, just be sure and hold the camera so the opening in the body is facing down, so any dust/debris falls out of the mirror box and not just into the bottom of the mirror box where it can eventually get on the image sensor.