freixas
No longer a newbie, moving up!
- Joined
- Jun 11, 2010
- Messages
- 132
- Reaction score
- 29
- Location
- Portland, Oregon
- Can others edit my Photos
- Photos NOT OK to edit
I’ve purchased a new Canon camera, one whose RAW format is not supported by my ancient Adobe CS6 tools. One option around buying new software is to convert the new RAW files to DNG format. I started to educate myself on what that might mean. This post is my research, should it prove useful to anyone else. I would also welcome corrections.
Honestly, if it weren’t for the lack of support in CS6 for the new camera, I wouldn’t give this issue a second look. Nor would I if Adobe’s software were available as other than subscription.
RAW File Contents
Let’s begin by describing the contents of RAW files:
The sensor data in a DNG file is typically stored in a lossless (JPG) compressed format, but could be uncompressed data or lossy, compressed data. The sensor data can contain 12-, 14-, or 16-bit Bayer mosaic data or de-mosaiced data.
Metadata
A RAW file is created with camera and image metadata, but you may wish to add additional metadata (e.g. keywords) or alter existing metadata (e.g. copyright). If we make adjustments to the image (e.g. alter the exposure), the adjustments are also metadata, usually of value only to the software that made the adjustments.
XMP data (EXtensible Metadata Platform) is one of several containers for storing metadata. DNG RAW files can have XMP data embedded directly within them. Many applications avoid altering the original RAW file and place any added or modified metadata in a “sidecar” file or in a database.
There is a small set of metadata that transports well from one application to the next, but only when stored in RAW files or in sidecar XMP files (keywords are the main one I’m thinking of, but I’m sure there are others). For example, ON1 produces an ON1 sidecar file, but if you add a keyword to an image, the keyword is only accessible to other applications if it is stored in an XMP sidecar.
Does conversion to DNG lose metadata?
The quick answers are yes and maybe.
I tested with a Canon CR2 file from my EOS 70D. There are clear differences in the metadata of a CR2 file and its DNG equivalent. Whether these differences are in any way significant is a harder question to answer.
For example, the CR2 has an “Image Width” of 5472 where the DNG has 5568. On the other hand, both have a “Cropped Image Width” of 5472. The images open up in Adobe Camera RAW with a width of 5472. Most of the programs I tried reported a width of 5472. I suspect the difference is an artifact of differences in the way the sensor data is record in the DNG.
There are other similar, puzzling changes. The “Aperture” for both is 5.6, but the “Aperture Value” is 5.7 in the CR2 and 5.6 in the DNG. The “Depth of Field” in the CR2 is “0.03 m (3.89 - 3.91 m)” but “0.03 m (3.82 - 3.84 m)” in the DNG. The “Flashpix Version” in the CR2 is 0100, but is missing entirely in the DNG—on the other hand, flash was not used. Similarly, the GPS Version is omitted in the DNG, but no GPS data was actually recorded by the camera.
The DNG includes some keywords not in the CR2.
The XMP format can accommodate vendor keywords and there seems to be little reason for Adobe to throw away useful data; however, it would require more knowledge than I have to tell whether any useful data was actually discarded. Some people believe that the vendors have totally proprietary and undocumented metadata; it’s possible, particularly if the metadata would somehow reveal some intellectual property that the companies would rather not reveal. I don’t know if the suspicions are founded or fall into the conspiracy-theory realm.
There is a possibility that applications may ignore some vendor-specific metadata when it does not come from the vendor’s RAW file. This sounds more like an application bug than a defect in the DNG format; however, if it affects you, it may not matter whose fault it is.
Does conversion change the sensor data?
Again, the answer is not simple to determine.
Adobe has some really good software engineers. They have no incentive to remove useful data. Just the suspicion that this might occur has been enough to dissuade many people from converting to DNGs.
I do know of several cases where data is altered:
The Procedure
I started with an unmodified (straight off the camera’s SD card) CR2 from my Canon EOS 70D. I copied this to a folder named Original. I then converted it to DNG using the default conversion parameters. This new file had the DNG extension, of course, but I modified the name as well to prevent any confusion with sidecar files.
I then created copies of this folder, one per application.
I loaded both images into various RAW applications. I applied no adjustments, but exported the file as a 16-bit PSD (or 16-bit TIFF if PSD was not supported) with the ProPhoto RGB color space. I opened the files in Photoshop and overlaid one on the other using Difference mode. I used various tools to verify that all pixels were zero (or to see how far off they were from 0).
Adobe Camera RAW CS6
As expected, the two images were identical.
This proves that it is possible to produce identical images from both a CR2 and its matching DNG, so in some sense the two images are “equivalent”. It is possible, of course, that performing the same adjustments to the two images might not have produced equivalent results.
Capture One v12.0.3
With Capture One, I selected each image and right-clicked to export the default variant as a 16-bit PSD using the ProPhoto RGB color space. I turned off cropping and output sharpening and included all metada.
The exported files did not have the same dimensions. In order for both images to come out with the same size and geometry, I had to set the lens profile of the CR2 to “Generic” (setting both to the correct lens produced images that were slightly different).
With the “Generic” lens profile, the two images were again absolutely identical.
DxO PhotoLab v2.2.2
PhotoLab doesn’t have PSD export, so I exported 16-bit TIFFs. There were no differences in the generated images.
Affinity Photo 1.6.5.135
Affinity Photo is more of a Photoshop replacement with a built-in RAW processor. I was not able to get the colors to match—the CR2 had a warmer look.
ON1 Photo RAW 2019.2
ON1 was unable to produce PSD files that matched—one was brighter than the other. I exported the files from the browser, in the hopes that the same default adjustments would be applied to both. To support this belief, no ON1 sidecars were produced.
I tried once more, this time going into the “Edit” module and either turning off or zeroing every setting. Not only were the colors still mismatched, but I noticed some differences in lens correction even though I had specifically turned off this setting for both images
Conclusion
It’s clear that it is possible for applications to create identical images from a vendor-specific RAW file and its DNG equivalent. This doesn’t mean that they will. Some do, some don’t and some can but only with some fiddling.
As far as I know, there is no standard for how a RAW image should be rendered. If you can accept the different renderings that come from switching to a different application, then the differences between RAW and DNG might be moot. The DNG rendering might be inferior or superior (or both, if evaluated using different criteria).
Are DNG files the only RAW files that are read/write?
Files are files. Unless protected, they can be read and written.
Most applications avoid modifying RAW files, even DNG RAW files. Some exceptions:
Advantages and Disadvantages of DNG files
Pluses
References
Honestly, if it weren’t for the lack of support in CS6 for the new camera, I wouldn’t give this issue a second look. Nor would I if Adobe’s software were available as other than subscription.
RAW File Contents
Let’s begin by describing the contents of RAW files:
- A header that defines the structure of the file.
- The sensor image data.
- The camera sensor’s metadata, used to interpret the sensor data.
- Image metadata—information about the image (such as shutter speed).
- A (typically full-size) JPG interpretation of the sensor data.
- A thumbnail, presumably a smaller version of the above.
The sensor data in a DNG file is typically stored in a lossless (JPG) compressed format, but could be uncompressed data or lossy, compressed data. The sensor data can contain 12-, 14-, or 16-bit Bayer mosaic data or de-mosaiced data.
Metadata
A RAW file is created with camera and image metadata, but you may wish to add additional metadata (e.g. keywords) or alter existing metadata (e.g. copyright). If we make adjustments to the image (e.g. alter the exposure), the adjustments are also metadata, usually of value only to the software that made the adjustments.
XMP data (EXtensible Metadata Platform) is one of several containers for storing metadata. DNG RAW files can have XMP data embedded directly within them. Many applications avoid altering the original RAW file and place any added or modified metadata in a “sidecar” file or in a database.
There is a small set of metadata that transports well from one application to the next, but only when stored in RAW files or in sidecar XMP files (keywords are the main one I’m thinking of, but I’m sure there are others). For example, ON1 produces an ON1 sidecar file, but if you add a keyword to an image, the keyword is only accessible to other applications if it is stored in an XMP sidecar.
Does conversion to DNG lose metadata?
The quick answers are yes and maybe.
I tested with a Canon CR2 file from my EOS 70D. There are clear differences in the metadata of a CR2 file and its DNG equivalent. Whether these differences are in any way significant is a harder question to answer.
For example, the CR2 has an “Image Width” of 5472 where the DNG has 5568. On the other hand, both have a “Cropped Image Width” of 5472. The images open up in Adobe Camera RAW with a width of 5472. Most of the programs I tried reported a width of 5472. I suspect the difference is an artifact of differences in the way the sensor data is record in the DNG.
There are other similar, puzzling changes. The “Aperture” for both is 5.6, but the “Aperture Value” is 5.7 in the CR2 and 5.6 in the DNG. The “Depth of Field” in the CR2 is “0.03 m (3.89 - 3.91 m)” but “0.03 m (3.82 - 3.84 m)” in the DNG. The “Flashpix Version” in the CR2 is 0100, but is missing entirely in the DNG—on the other hand, flash was not used. Similarly, the GPS Version is omitted in the DNG, but no GPS data was actually recorded by the camera.
The DNG includes some keywords not in the CR2.
The XMP format can accommodate vendor keywords and there seems to be little reason for Adobe to throw away useful data; however, it would require more knowledge than I have to tell whether any useful data was actually discarded. Some people believe that the vendors have totally proprietary and undocumented metadata; it’s possible, particularly if the metadata would somehow reveal some intellectual property that the companies would rather not reveal. I don’t know if the suspicions are founded or fall into the conspiracy-theory realm.
There is a possibility that applications may ignore some vendor-specific metadata when it does not come from the vendor’s RAW file. This sounds more like an application bug than a defect in the DNG format; however, if it affects you, it may not matter whose fault it is.
Does conversion change the sensor data?
Again, the answer is not simple to determine.
Adobe has some really good software engineers. They have no incentive to remove useful data. Just the suspicion that this might occur has been enough to dissuade many people from converting to DNGs.
I do know of several cases where data is altered:
- Camera calibration data is removed and “baked” in to the DNG sensor data. This sounds undesirable until you understand that many camera makers also “bake” the data into their native RAW formats. The reason for not doing so is usually because of limited processing power. Since the cameras with the most processing power are usually the most advanced, it sounds like these would be the least likely to include the calibration data “unbaked”. I’m trusting Adobe’s explanation of this—does anyone know of any advanced camera that includes calibration data and any software that takes advantage of this data? I’m not talking about mosaic data, which the DNG can retain and which has value.
- Some operations (demosaicing) are helped by having some edge pixels. Some cameras don’t actually have physical edge pixels and use random data in their place. These are removed as they have no value.
- Canon SLR S-RAW and M-RAW formats are not mosaic files and the sensor data are stored in a quasi-JPEG like YCbCr space. The DNG conversion maps these to RGB space, so you might lose some precision. On the other hand, software dealing directly with these RAW formats will probably convert them to a standard internal form (with the same loss of precision), so Adobe may be saving them some work. In any case, if you use S-RAW and M-RAW, you are not in a position to argue about preserving the value of your sensor data
- The DNG converter cannot currently convert dual-pixel RAW files in a way that preserves the dual-pixel features.
The Procedure
I started with an unmodified (straight off the camera’s SD card) CR2 from my Canon EOS 70D. I copied this to a folder named Original. I then converted it to DNG using the default conversion parameters. This new file had the DNG extension, of course, but I modified the name as well to prevent any confusion with sidecar files.
I then created copies of this folder, one per application.
I loaded both images into various RAW applications. I applied no adjustments, but exported the file as a 16-bit PSD (or 16-bit TIFF if PSD was not supported) with the ProPhoto RGB color space. I opened the files in Photoshop and overlaid one on the other using Difference mode. I used various tools to verify that all pixels were zero (or to see how far off they were from 0).
Adobe Camera RAW CS6
As expected, the two images were identical.
This proves that it is possible to produce identical images from both a CR2 and its matching DNG, so in some sense the two images are “equivalent”. It is possible, of course, that performing the same adjustments to the two images might not have produced equivalent results.
Capture One v12.0.3
With Capture One, I selected each image and right-clicked to export the default variant as a 16-bit PSD using the ProPhoto RGB color space. I turned off cropping and output sharpening and included all metada.
The exported files did not have the same dimensions. In order for both images to come out with the same size and geometry, I had to set the lens profile of the CR2 to “Generic” (setting both to the correct lens produced images that were slightly different).
With the “Generic” lens profile, the two images were again absolutely identical.
DxO PhotoLab v2.2.2
PhotoLab doesn’t have PSD export, so I exported 16-bit TIFFs. There were no differences in the generated images.
Affinity Photo 1.6.5.135
Affinity Photo is more of a Photoshop replacement with a built-in RAW processor. I was not able to get the colors to match—the CR2 had a warmer look.
ON1 Photo RAW 2019.2
ON1 was unable to produce PSD files that matched—one was brighter than the other. I exported the files from the browser, in the hopes that the same default adjustments would be applied to both. To support this belief, no ON1 sidecars were produced.
I tried once more, this time going into the “Edit” module and either turning off or zeroing every setting. Not only were the colors still mismatched, but I noticed some differences in lens correction even though I had specifically turned off this setting for both images
Conclusion
It’s clear that it is possible for applications to create identical images from a vendor-specific RAW file and its DNG equivalent. This doesn’t mean that they will. Some do, some don’t and some can but only with some fiddling.
As far as I know, there is no standard for how a RAW image should be rendered. If you can accept the different renderings that come from switching to a different application, then the differences between RAW and DNG might be moot. The DNG rendering might be inferior or superior (or both, if evaluated using different criteria).
Are DNG files the only RAW files that are read/write?
Files are files. Unless protected, they can be read and written.
Most applications avoid modifying RAW files, even DNG RAW files. Some exceptions:
- Adobe will embed XMP data into a DNG if it can. Make the DNG file read-only, and Adobe will create a sidecar XMP.
- Canon’s DPP software writes its adjustments into the CR2 file.
- exiftool can alter metadata in a RAW file.
Advantages and Disadvantages of DNG files
Pluses
- DNG files can be produced with more compression that a camera might have time for, so they might be smaller in size. As processing power increases, this advantage might be reduced (for some advanced cameras, this may already be true).
- DNGs have a checksum to validate for corruption.
- DNG files offer you some options for the size of the JPG preview (this might produce both a space and time savings).
- XMP data is usually written into a DNG file. This makes it easier to deliver a single file with all adjustments. The downside is it may create bigger backups and it seems to be a feature only used by Adobe software. I couldn’t find anyone else writing into a DNG file. As stated above, you stop this by making DNG files read-only. Some other vendors have their own packaging format (Capture One has one called EIP).
- Camera vendors that offer software (such as Canon’s DPP) typically ignore all files except those produced directly by their cameras. Once in DNG format, you cannot use these tools.
- Converting from another format to DNG takes time and adds an extra step to the workflow.
References
Last edited:
