Interpretation of DxOMark results

[enzodm]

While I'm glad to have an independent and quantitative evaluation of sensors and lenses like DxOMark, I'm somewhat puzzled with some of their data. As an example, please look here: DxOMark - Compare sensors
(comparison among 1000D, 50D, 60D).
Sooner or later I wish to upgrade from my 1000D (XS) to 60D or even 50D to save some money. Put apart the better body, 50D seems only slightly better than 1000D from the sensor point of view and worst if looking at ISO performance (perhaps related to different pixel density).
If I look at details, it seems that the 1000D is better than both 50D and 60D at 100ISO, and then more or less the same. If I look at noise, it seems like they are more or less the same, unless the fact 50D and 60D allow for greater ISO, but with quality much lower than 1000D 1600ISO. I'm not sure how these differences translate in real picture quality - it seems to me to have seen better low light pictures coming from 50D than what I would expect from those data.

So, please give me directions on how interpret DxOMark results for a more informed choice, thanks.

 

[Overread15 3]

The problem with DxOMark results is that they never publish the methods or criteria they use for testing, just the results. This leads to a lot of results they publish being somewhat suspect by the general population as they suggest things that - in the field and in real world experiences - are either different or more marginal than the scores from DxO would suggest.

Personally I would take it with a pinch of salt and read regular reviews and opinions to get a more balanced viewpoint.

 

[analog.universe]

^ +1

If you want to compare image quality from 2 different cameras, look at the actual images they've taken, not some mathematical deconstruction that attempts to explain them. I've had good experience with Digital Camera Resource Page | Unbiased Digital Camera News and Reviews. They take essentially the same shots of the same subjects for every camera they review, so comparisons are pretty straightforward.

As you mentioned though, due to the size of the pixel on the sensor, all other things being equal (which they pretty much never are), higher pixel count = higher noise.

 

[subscuck]

^ Hmmm, didn't we just go thru this last night on another thread? DXO Mark is a waste of reading time.

 

[Overread15 3]

DxO is however not all bad as their processing software that they make is very popular - esp with users of wide angle lenses looking to reduce distortion from the lens.

 

[enzodm]

The problem with DxOMark results is that they never publish the methods or criteria they use for testing, just the results.

ok, this is really a problem. No methods, no sensible interpretation.
I always looked at reviews (Analog: for me, DPreviews gives good comparisons), but numbers are tempting, at least for my education.

 

[subscuck]

I'm not sure I've seen it here, but on dpreview, which seems to have a lot more DXO Mark discussion, I've clicked on links people have provided to DXO Mark showing consumer and prosumer lenses with better numbers than L or Gold Ring glass. As Overread said, without knowing how these results were obtained, it's just not possible to interpret the numbers in any meaningful way. And vis a vis the thread from last night, raw numbers are meaningless, final output is what counts.

 

[KmH]

DxOMark - Why trust DxOMark?

Why trust DxOMark?

When looking for digital camera image quality evaluation data and benchmarks, you want to be sure that the provided measurements have been openly and objectively performed by a completely independent testing operation that uses state-of-the-art equipment and follows industry-standard methods.
DxOMark means objective, independent, comprehensive image quality data
It is our constant focus at DxO Labs to ensure that dxomark.com is a source of information that photographers can fully trust for objective evaluations and comparisons of digital cameras and lenses.
DxOMark measurements are performed on unprocessed RAW files (the only image format that reflects the intrinsic performance of digital cameras and lenses)
All published measurements are RAW sensor-based and performed prior to any digital image processing (except processing that may happen on the sensor chip itself). RAW format is analogous to a film negative in traditional photography, and it is the format by which objective performance for digital cameras is most effectively evaluated, as it does not depend on the efficiency of the RAW conversion applied.
RAW is also intrinsically more relevant for measuring lens performance, given that no further processing was used that might have smoothed or hidden any intrinsic optical limitations or lens defects.
Overall, for both sensor and lens, DxOMark RAW-based measurements and ranking scales provide an objective and essential evaluation and comparison tool. Read more about using RAW to measure image quality.
DxOMark lens measurements are performed separately for each compatible camera body and under all shooting conditions
DxOMark lens measurements are unique in that they are performed on RAW and are specific to all camera bodies for which each lens is designed. The test results published on dxomark.com conclusively show that the same lens performs differently when mounted on different camera bodies. Many sensor characteristics, such as physical size, pixel pitch, and microlenses, influence the optical performance of lens/camera combinations (read more on how lens performance is affected by the sensor).
On dxomark.com, the camera body is always referenced for lens measurements, and you can easily browse through the compatible bodies for a given lens to view how the different combinations perform.
Measurements are performed for each lens-camera combination using a wide range of shooting parameters (aperture, ISO speed, object distance, focal length, etc). Up to a thousand shots of various targets under different exposure conditions may be required for each combination, generating hundreds of thousands of measurement data reflecting the lens performance under all shooting conditions and for the compatible camera bodies (read more about How we test).
DxOMark measurement methods and protocols are open and compliant with international and industry standards
All dxomark.com measurements follow methods and protocols described in ISO (International Standard Organization) standard documentation or developed by industry players and considered as industry standards. DxO Labs image scientists and engineers have studied, developed and implemented such standard methods and protocols and then packaged this know-how and tools in a commercial solution, DxO Analyzer. This solution is now widely used in the digital camera and component industry worldwide by such manufacturers as Nikon, Olympus, Panasonic and Samsung, to name just a few. The product is also used to perform certain tests by many web or print photo magazines such as as Popular Photography, Nippon Camera, Chasseur d’Images, DIWA Labs, Imaging Resource, Réponse Photo, Let’s Go Digital, Digital Camera Magazine Japan, Digital Photo Japan, and more than 30 other press partners.
Further, DxO Labs has put in place a very thorough data validation process. Once data are produced, they are analyzed in detail and compared with those in our existing image quality measurements database (thousands of digital cameras, lenses and imaging systems have been tested and analyzed over time). This process has led us to develop specific protocols or tools to deal with all required difficult measurement issues such as best focus optical performance and sensor spectral sensitivity.
DxO Labs is an active participant in international imaging standardization activities. For example, we have been elected as Technical Editor of the International Imaging Industry Association (I3A) CPIQ standard in defining new methods to assess photographic quality from the consumer point of view, and DxO Labs image scientists regularly present their results at such renowned digital imaging conferences as the Electronic Imaging Conference, and Image Sensors (Europe and US).

 

[enzodm]

[... (read more about How we test).
DxOMark measurement methods and protocols are open and compliant with international and industry standards...

ok, thanks: so methods (at least the basis) are public, and I will read them with care.
But I have to come back to the original question: at this point how do I interpret the comparison between 1000D and 50D? Are they really so close that, from the sensor point of view, it is not worth to "upgrade" from one the the other?

 

[KmH]

Maybe!

You have to consider the other camera features and functions too.
Things like the fact the 1000D doesn't have Spot metering mode, while the 50D does.

How about the frame burst rate? How about the auto focus system? how do those compare.

DXO Mark does not compare bells and whistles. DXO Mark only compares the basic Raw image metrics.

 

[Garbz]

I've clicked on links people have provided to DXO Mark showing consumer and prosumer lenses with better numbers than L or Gold Ring glass.

While in general the thought is counter intuitive don't automagically assume that a colourful ring at the end of the lens means it necessarily outperforms another lens without the coloured ring. I can think of several Gold Ring lenses which are outperformed in sharpness and CA by my kit lens. In general though my kit lens doesn't have an f/2.8 aperture though, which is what I paid for with my gold ring lens. There are some truly horridly performing gold ring lenses, as well as some truly exceptional "cheap" ones.

But I have no comment on DxO's results on the matter.

 

[Derrel]

I've seen loads of Canon users who try and discredit DxOMark results because so many recent Canon models perform so poorly against the new Nikon and Pentax cameras that have sensors made by Sony in them...they cannot imagine that their higher MP-count Canons cameras could score worse than lower MP count Nikons and Pentaxes...in three metrics: Dynamic Range, High ISO, and Color Depth...it's almost epidemic, the constant attempt to discredit DxOMark results among Canon fanatics. Last year, we had numerous Canon nuts (extremists who have for the most part, left this site) constantly trying to discredit an entire company's worth of people and testing equipment as biased...because Canon products were being out-scored by competing cameras. The list of canards these guys pout forth about DxOMark was a Fear Uncertainty and Doubt campaign worth of Microsoft in the 1990's.

One thing to keep in mind right now, in mid 2011: SONY has developed a new generation of sensors that are now being paired with a new generation of noise-reduction and sensor read-out electronics, and the "problems" that Canon has had with its 17.8 megapixel sensor in its 7D and 60D bodies has been circumvented by Sony with its new slightly larger in area, 16.2 MP sensor. Right now, in May of 2011, the SONY-made sensors used in the Nikon D5100, D7000, and Pentax K5 are of an entirely new and more-advanced generation than those Canon is currently putting into its mid-range cameras...the ACTUAL, real-world results mirror DxOMark scoring. After the testing dPreview conducted they wrote this about the Pentax K5: "One of the best dslrs in terms of High ISO image quality." and "Exceptionally low shadow noise in RAW files at low ISOs." The little Nikon D5100 has image quality that matches the D7000; these three cameras have the best sensor,noise reduction, and read-out electronics in the consumer segment.

The one thing that many people are unaware of right now, is that the newest Sony sensors have managed to achieve extremely low noise levels, so much that GROSS under-exposures, like the exposure for ISO 51,200 can be used with the camera set to ISO 200, and the image successfully "rescued" using some of the more-advanced RAW converters on the market. I have seen some demo frames on the web, exposed with ISO values of over 50,000 (fifty thousand ISO!!) but with the camera actually being set to ISO 200--and the image these new Sony sensors makes is so low in noise, that EXTREME Positive Exposure Compensation can be applied to the RAW file data and an actually decent, usable image can be pulled out. Combine this ability of the new sensors Sony, Nikon, and Pentax are using, and then add in the new read-out electronics hardware, and the image processing SOFTWARE that the camera makers build, and we've come to an entirely new frontier in sensor development, as well as in camera-building. Because DxOMark examines RAW image quality, there is a huge advantage for the newer Sony-sensored cameras, like the Nikon D3x versus the Sony A900 and A 850. All three cameras use the SAME sensel, the same light-sensitive portion of the sensor, but the Nikon D3x has vastly,vastly superior read-out electronics and vastly better image processing capabilities than the Sony A900 and A850. Same thing is happening now with the very-newest cameras, the Nikon D5100, D7000,and Pentax K5---the same newer, better, lower-noise-in-RAW sensor technology that Sony developed in its 24.6 megapixel full frame sensor two years ago, has now been migrated to the new 1.5x 16.2 sensor...meanwhile the "other" camera makers are a generation behind, and their cameras are getting clobbered when measured against newer technology cameras.

SONY has advanced sensor technology to a new level. Canon has not yet caught up

 

[subscuck]

I've clicked on links people have provided to DXO Mark showing consumer and prosumer lenses with better numbers than L or Gold Ring glass.

While in general the thought is counter intuitive don't automagically assume that a colourful ring at the end of the lens means it necessarily outperforms another lens without the coloured ring. I can think of several Gold Ring lenses which are outperformed in sharpness and CA by my kit lens. In general though my kit lens doesn't have an f/2.8 aperture though, which is what I paid for with my gold ring lens. There are some truly horridly performing gold ring lenses, as well as some truly exceptional "cheap" ones.

But I have no comment on DxO's results on the matter.

I'm with you 100%. I love my 85 1.8 and see no reason to ever upgrade. It's IQ is outstanding. My point of contention is on DXOMark's lens rankings. For instance, the Canon 50 1.8, on the 1DsIII, at f2.8, outscores both the Canon 1.4 and 1.2L. Really?

 

[Derrel]

While I'm glad to have an independent and quantitative evaluation of sensors and lenses like DxOMark, I'm somewhat puzzled with some of their data. As an example, please look here: DxOMark - Compare sensors
(comparison among 1000D, 50D, 60D).
Sooner or later I wish to upgrade from my 1000D (XS) to 60D or even 50D to save some money. Put apart the better body, 50D seems only slightly better than 1000D from the sensor point of view and worst if looking at ISO performance (perhaps related to different pixel density).
If I look at details, it seems that the 1000D is better than both 50D and 60D at 100ISO, and then more or less the same. If I look at noise, it seems like they are more or less the same, unless the fact 50D and 60D allow for greater ISO, but with quality much lower than 1000D 1600ISO. I'm not sure how these differences translate in real picture quality - it seems to me to have seen better low light pictures coming from 50D than what I would expect from those data.

So, please give me directions on how interpret DxOMark results for a more informed choice, thanks.

Keep this in mind" the 50D launched in the fall of 2008, and its sensor and electronics were probably developed 18 months earlier. The 50D is now a camera based on, basically, 5 year old technology and capability. The 50D was a very controversial camera among Canon users, and caused a HUGE disagreement among the Canon community about cramming too many pixels onto too small of a sensor area; its High ISO score of 696 is bettered by the EOS 60D which scored 813, and which was launched a full two YEARS later, with a sensor that has 17 MP. The Rebel XS is also a mid-2008 camera launch. So, you're comparing two 2008 model Canons with a 2010 model. In terms of Dynamic Range and Color Depth, the two newer Canon cameras are almost neck and neck. Why? Because they are still made using "old" sensor technology.

If we eliminate the 10 MP Rebel, and instead drop in the new Pentax K5, we see HUGE improvements in overall sensor score, color depth, Dynamic Range, and Low-Light ISO performance...from a camera that premiered September 20th, 2010, just three weeks before the Canon 60D. As evidence in support of my claims in Post #12 above, I include this three-camera comparison of the Pentax K5, Canon 50D,and Canon 60D. DxOMark - Compare sensors

 

[enzodm]

Maybe!

You have to consider the other camera features and functions too.
Things like the fact the 1000D doesn't have Spot metering mode, while the 50D does.
How about the frame burst rate? How about the auto focus system? how do those compare.

Keith: in fact I would upgrade for having some of those bells&twistles, including better viewfinder, but possibly having more sensitivity when needed. I have the possibility to buy a used 50D for about half the price of a new 60D... which would be my natural choice. I could resist without upgrading for some time, and wait for cheaper prices of 60D.

Derrel: I know your thoughts about Canon (including your blog post on Canon and Nikon) but I do not think here is matter of religion. As I told, I'm naturally inclined in trusting quantitative evaluations more than samples, but within reason.
Remaining in Canon will cost me less than changing (by the way, here K5 is 200€ more than 60D). However, in case of change I would have no problems in choosing Pentax - one reason I did not start with it (Km) is that was difficult to find one place selling it. When I bought my camera, I had chances to try d3000, Alpha 230/330 and a couple of Olympus. At that time, restricted to entry level, my choice was not bad even considering DXOMark data (although I discovered it later).