Confused about thedigitalpicture's sharpness comparison

[PaulWog]

There are various ways to gather information about a lens. I have been going through DXOMark and The-Digital-Picture's two tools (DXOMark's sharpness charts, and the digital picture's lens image quality comparison tool). It seems like DXOMark's sharpness ratings don't always line up with MTF charts and information from other reviews (ie. "Stopped down to f8 the lens appears sharpest" <-- says one review... while DXOMark might indicate that the lens is sharpest at f4, given their measurements). Alright, we can write off DXOMark as possibly producing skewed and irrelevant numbers... or can we say that? Their system seems to generally be correct, but some things just seem oddly off.

Moving on to The-Digital-Picture: Most of their images seem to be about right (sharpness-wise)... but I've looked at some very expensive lenses and seen some really poor sample images out of their comparison tool. For example (a prosumer lens which I own): The Sigma 18-35 Art. At any aperture at 35mm, the image looks fairly soft on their image quality tool:
The Digital Picture Link

Going through some of their lens shots, some lenses which are touted as soft look very sharp... and some lenses that are touted as the sharpest of sharp, look much duller than one would expect.

I'm just curious about why I'm seeing these discrepancies. Clearly what really matters is real-world performance, but my question is about the theoretical measurements / theoretical target-chart sharpness shots.

 

[Solarflare]

Why online tests are so different is fairly obvious: sample variation. Tests that only run one lens or worse only the lens provided by the original company to the press are less reliable than tests that check out a couple random samples bought on the free market.

Unfortunately MOST of the online tests are just like that.

 

[PaulWog]

Why online tests are so different is fairly obvious: sample variation. Tests that only run one lens or worse only the lens provided by the original company to the press are less reliable than tests that check out a couple random samples bought on the free market.

Unfortunately MOST of the online tests are just like that.

Seems to be just that. Even looking through MTF charts, I'm seeing the 70-300 VR has a rating of about 35 lp/mm at 300mm and f8 on lenstip, and yet the 300mm f4 Nikon prime (old version) gets 45 lp/mm... the difference is *definitely* more staggering than that. My Sigma 150-600 Contemporary is rated at between 30 and 33 lp/mm at 600mm, and I know for a fact it is sharper at 600mm than the Nikon 70-300 VR at 300mm (not by leaps and bounds, but by a margin).

Thanks for the answer. This won't really effect my photography (obviously), but I enjoy looking at the technical nit-picky end of things... and it's frustrating seeing so much difference between one review and another.