Sounds like maybe you have a dud 80-200 AF-S **OR** the 36-MP resolving ability of the D800 is simply too much for the 80-200. I shot the 80-200 AF-S on the D3x for two years, and it was pretty good on 24-MP FX, whereas the 70-200 VR, the "first version" focused faster, handled more nimbly, but the corners of the image were NOT up to the full-frame, 24-megapixel D3x sensor; the 70-200VR did not really vignette on the D3x, but its corner sharpness was WEAK, even stopped down, so for landscapes and such, it was not a good zoom. The thing was, it was a VERY good crop-sensor lens, with very high sharpness over a smaller image circle, which is the only type of d-slr Nikon had when it was designed and released.
The NEWER 70-200 VR, the "second version" f/2.8 model is supposedly quite good; same goes for the 70-200 f/4 AF-S VR; the f/4 verion is the one I am considering purchasing...SMALLISH, light, handles fast, and frankly, I never shot pictures I wanted to keep at f/2.8.
I think your 80-200 AF-S might be a clunker? Maybe? I dunno...depends I guess on one's expectations and uses for the lens. I shot my 60,90,and 180mm primes this last weekend, alongside three zooms...I was able to spot the prime lens shots by ultimate sharpness on 24-MP D610 frames of landscape stuff at the Oregon coast. Yet STILL, the zooms allowed me to get precise framing and compositions I wanted, so, I shot plenty with the three zooms I had.
I've grown tired of lugging an f/2.8 70-200 around when I never shoot at f/2.8, but start at f/4 or f/4.5 as my "wide" stop, and consider f/5.6 to f/8 to be the most-useful f/stops for me.
As an aside: I've read the fairly new Tamron 70-200/2.8 G-2 is a staggeringly sharp lens...The Angry Photographer, on YouTube, a real lens nut, enthused muchly about the new Tamron's price,value,performance, naming it the #1 Value lens in F-mount a few months back...high praise from him!