The display you are refering to is just that, a display. When in a semi-auto mode (Aperture priority, Shutter priority, or Program shift) it is the display for the Exposure Compensation. The semi-auto modes will always try to expose at the center of what the light meter reads. Exposure compensation is just telling the camera to shift the exposure that it's reading to that amount you set.
In manual mode, there is no exposure compensation. You set the exposure completely with full control over everything. The display now shows what your light meter is reading and you adjust aperture, shutter, or ISO to change the exposure. Change any of those and the display will move.
What I don't know is if there are more than one display with a dSLR. Perhaps the exposure compensation display is on the LCD and the display in the viewfinder is only showing what the light meter is reading. I am only assuming that the display in the viewfinder is the exact same function as the one on the back LCD with what I typed above.