How reliable are DxO Mark scores?

[sovietdoc]

While following the never-endless debate of "Canon vs Nikon", I've stumbled upon a number of people who criticized DxO Mark for being wrong. I don't usually care about scores produced by any benchmarking suites as they usually don't reflect real life performance, but because a lot of people actually base their opinion on the numbers produced by DxO Labs, I had to look into it deeper.

Apparently, their Lens benchmarking tool shows 70-200 f/2.8 mark I at 21 points and mark II at 18. Now, everyone who shot with the Mark I and then went to Mark II will tell you that mark II is better in every single way, yet DxO places it lower by a significant margin.

DxO shows Mark II at a lot worse resolution, worse distortion, worse vignetting and better CA compared to Mark I. This is rubbish as many many people have cropped RAWs from either lens and found that mark II was better than mark I in every one of those cases.


This is just one example of obvious error in DxO Mark tests, there are others as well.

So the question is: Why do people think DxO's scores are so significant that they use DxO scores to justify their purchases or in an argument of what's better? I would seriously doubt any number produced by DxO Labs based on what I've seen.

 

[analog.universe]

I agree, browsing through DxO I've seen lots of results that are either obviously wrong, like the example you point out, or don't match up with other review sites. When I buy a lens I read every test I can get, and the entire internet usually agrees on the numbers, except DxO. They so frequently contradict ever other test that I've stopped reading them at all. I never know if I'll get something bogus...

 

[Trever1t]

Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?

 

[o hey tyler]

Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?

He's referencing a difference in scores of the same Canon lens (MKI and MKII). This isn't a Canon v. Nikon thing... It wasn't even brought up.

 

[Trever1t]

oops.....mybad! Sorry, I totally misread!

 

[KmH]

DXO Mark uses industry standard testing methods.

DxOMark - Test result reliability

 

[Derrel]

Funny, all the Canon users were touting the benchmarks WHEN Canon tested higher?

He's referencing a difference in scores of the same Canon lens (MKI and MKII). This isn't a Canon v. Nikon thing... It wasn't even brought up.

It's implied...Canon has been gettig its ASS kicked by Nikon products since 2009...and now, Canon users are questioning the validity of the DxO Mark scores....BUT, when Canon was scoring higher, they were touting the DxO Mark scores all the fricking time, all over the web...

The better question might be, "How reliable is Canon's lens quality control???"

Here is what Axel, at DxO Mark had to say about the test results: Canon EF 70-200mm f/2.8L IS II USM vs its predecessor :If we compare these 2 generations of L series telephoto lenses on a Canon 5D Mark II, we can see the following:

• These 2 lenses are very similar in term of transmission, distortion and vignetting
• The version II gives slightly less chromatic aberration
• The Canon EF 70-200mm f/2.8L IS II USM offers slightly less resolution with 51 lp/mm compared to the excellent 61 lp/mm of the Canon EF 70-200mm f/2.8L IS USM
• The slightly less good resolution result of the Canon EF 70-200mm f/2.8L IS II USM comes from less homogenous behavior across the field.

.."after checking with all our experts in the lab, there isn’t really a mistake.

Indeed, the review was a bit harsh with the new version of the Canon EF 70-200mm f/2.8L IS II USM and we modified it: DxOMark - Canon EF 70-200mm f/2.8L IS II USM measurements and review.
But, overall the Mark 1 has a slightly higher and more homogeneous resolution. So, it scores better on a full frame camera, like the Canon 5D Mark II used in the review.
But, as we also added to the review, if you make the same comparison on a APS-C, it’s the Canon EF 70-200mm f/2.8L IS II USM that comes out slightly ahead. This may be the reason why the results are in other reviews made on other cameras.


So, what it means is that both lenses are really really close, and if you have a full frame it is better go for the older version and if you have an APS-C the newer version is a better choice.


Axel from DxO Labs"

So...they stand by their results on the 5D-II; the older lens performs better across a wider range of settings. ALSO, the 70-200 Mark II is alleged to be non-parfocal, meaning its focus point shifts as the lens is zoomed....that is a VERY,VERY bad quality to have in a lens like a 70-200/2.8 zoom...

The differences in lens scores are not particularly significant on the 5D-II they tested it on...I would love to see the OP verify his assertions that there are "other errors" in DxO Mark's tests....if so, let's SEE some URL's and some actual proof.

 

[analog.universe]

DXO Mark uses industry standard testing methods.

DxOMark - Test result reliability

Nothing there seems to explain why so many of their results are either self contradictory, or contradict the results of other respected testers.

Does it not seem odd that the Samyang 14 and 35mm, as well as the Canon 85mm 1.8, all report higher resolution figures than any macro lens they tested? They claim the Tamron 18-270mm f/3.5-6.3 has a higher resolving power than the Canon 300mm f/2.8 IS II (a lens which broke records on most testing sites). You could go on and on with the absurdity... It doesn't matter how objective they claim to be, their figures are in disagreement with reality.

 

[DorkSterr]

Yup those guys over at DxO know what their doing.

 

[Frances69]

Nothing there seems to explain why so many of their results are either self contradictory, or contradict the results of other respected testers.

 

[chuasam]

Reliable, accurate and absolutely pointless in real world applications.

 

[jaomul]

Reliable, accurate and absolutely pointless in real world applications.

I believe this to be true but- I have to point out that i use Canon so i may be a bit defensive because they make such bad cameras and i wasted my money on 2 of them. It was pointed out on a post above by a very informed source that Nikon are kicking canons ass since 2009. I dont doubt this, the facts say so. This leads me to believe that the nikon cameras before 2009 were even more crap than the crap canon cameras on the market now. I should go get a sight test because i have seen loads great images from older Nikon cameras from before that. How is this. i am confused now

 

[Crollo]

about as reliable as the american government

 

[sovietdoc]

I think this says it all: Customers worldwide use DxO Analyzer

 

[2WheelPhoto]

they are hiring too

DxO - Careers