DNG Files: A summary of my research

Discussion in 'Beyond the Basics' started by freixas, May 1, 2019.

Tags:
  1. freixas

    freixas TPF Noob!

    Joined:
    Jun 11, 2010
    Messages:
    113
    Likes Received:
    22
    Location:
    Portland, Oregon
    Can others edit my Photos:
    Photos NOT OK to edit
    I’ve purchased a new Canon camera, one whose RAW format is not supported by my ancient Adobe CS6 tools. One option around buying new software is to convert the new RAW files to DNG format. I started to educate myself on what that might mean. This post is my research, should it prove useful to anyone else. I would also welcome corrections.

    Honestly, if it weren’t for the lack of support in CS6 for the new camera, I wouldn’t give this issue a second look. Nor would I if Adobe’s software were available as other than subscription.

    RAW File Contents
    Let’s begin by describing the contents of RAW files:
    • A header that defines the structure of the file.
    • The sensor image data.
    • The camera sensor’s metadata, used to interpret the sensor data.
    • Image metadata—information about the image (such as shutter speed).
    • A (typically full-size) JPG interpretation of the sensor data.
    • A thumbnail, presumably a smaller version of the above.
    A DNG (Digital NeGative) contains all the above and is a RAW file. It is unique in that other RAW formats can be converted to DNG RAW. Adobe created the DNG format and provides a free tool to convert other RAW formats to DNG RAW.

    The sensor data in a DNG file is typically stored in a lossless (JPG) compressed format, but could be uncompressed data or lossy, compressed data. The sensor data can contain 12-, 14-, or 16-bit Bayer mosaic data or de-mosaiced data.

    Metadata
    A RAW file is created with camera and image metadata, but you may wish to add additional metadata (e.g. keywords) or alter existing metadata (e.g. copyright). If we make adjustments to the image (e.g. alter the exposure), the adjustments are also metadata, usually of value only to the software that made the adjustments.

    XMP data (EXtensible Metadata Platform) is one of several containers for storing metadata. DNG RAW files can have XMP data embedded directly within them. Many applications avoid altering the original RAW file and place any added or modified metadata in a “sidecar” file or in a database.

    There is a small set of metadata that transports well from one application to the next, but only when stored in RAW files or in sidecar XMP files (keywords are the main one I’m thinking of, but I’m sure there are others). For example, ON1 produces an ON1 sidecar file, but if you add a keyword to an image, the keyword is only accessible to other applications if it is stored in an XMP sidecar.

    Does conversion to DNG lose metadata?
    The quick answers are yes and maybe.

    I tested with a Canon CR2 file from my EOS 70D. There are clear differences in the metadata of a CR2 file and its DNG equivalent. Whether these differences are in any way significant is a harder question to answer.

    For example, the CR2 has an “Image Width” of 5472 where the DNG has 5568. On the other hand, both have a “Cropped Image Width” of 5472. The images open up in Adobe Camera RAW with a width of 5472. Most of the programs I tried reported a width of 5472. I suspect the difference is an artifact of differences in the way the sensor data is record in the DNG.

    There are other similar, puzzling changes. The “Aperture” for both is 5.6, but the “Aperture Value” is 5.7 in the CR2 and 5.6 in the DNG. The “Depth of Field” in the CR2 is “0.03 m (3.89 - 3.91 m)” but “0.03 m (3.82 - 3.84 m)” in the DNG. The “Flashpix Version” in the CR2 is 0100, but is missing entirely in the DNG—on the other hand, flash was not used. Similarly, the GPS Version is omitted in the DNG, but no GPS data was actually recorded by the camera.

    The DNG includes some keywords not in the CR2.

    The XMP format can accommodate vendor keywords and there seems to be little reason for Adobe to throw away useful data; however, it would require more knowledge than I have to tell whether any useful data was actually discarded. Some people believe that the vendors have totally proprietary and undocumented metadata; it’s possible, particularly if the metadata would somehow reveal some intellectual property that the companies would rather not reveal. I don’t know if the suspicions are founded or fall into the conspiracy-theory realm.

    There is a possibility that applications may ignore some vendor-specific metadata when it does not come from the vendor’s RAW file. This sounds more like an application bug than a defect in the DNG format; however, if it affects you, it may not matter whose fault it is.

    Does conversion change the sensor data?
    Again, the answer is not simple to determine.

    Adobe has some really good software engineers. They have no incentive to remove useful data. Just the suspicion that this might occur has been enough to dissuade many people from converting to DNGs.

    I do know of several cases where data is altered:
    • Camera calibration data is removed and “baked” in to the DNG sensor data. This sounds undesirable until you understand that many camera makers also “bake” the data into their native RAW formats. The reason for not doing so is usually because of limited processing power. Since the cameras with the most processing power are usually the most advanced, it sounds like these would be the least likely to include the calibration data “unbaked”. I’m trusting Adobe’s explanation of this—does anyone know of any advanced camera that includes calibration data and any software that takes advantage of this data? I’m not talking about mosaic data, which the DNG can retain and which has value.
    • Some operations (demosaicing) are helped by having some edge pixels. Some cameras don’t actually have physical edge pixels and use random data in their place. These are removed as they have no value.
    • Canon SLR S-RAW and M-RAW formats are not mosaic files and the sensor data are stored in a quasi-JPEG like YCbCr space. The DNG conversion maps these to RGB space, so you might lose some precision. On the other hand, software dealing directly with these RAW formats will probably convert them to a standard internal form (with the same loss of precision), so Adobe may be saving them some work. In any case, if you use S-RAW and M-RAW, you are not in a position to argue about preserving the value of your sensor data
    • The DNG converter cannot currently convert dual-pixel RAW files in a way that preserves the dual-pixel features.
    Even if the sensor data is exactly equivalent, applications may not process the different formats to yield the same images. I decided to test this, as I have a number of trial applications on my system right now.

    The Procedure
    I started with an unmodified (straight off the camera’s SD card) CR2 from my Canon EOS 70D. I copied this to a folder named Original. I then converted it to DNG using the default conversion parameters. This new file had the DNG extension, of course, but I modified the name as well to prevent any confusion with sidecar files.

    I then created copies of this folder, one per application.

    I loaded both images into various RAW applications. I applied no adjustments, but exported the file as a 16-bit PSD (or 16-bit TIFF if PSD was not supported) with the ProPhoto RGB color space. I opened the files in Photoshop and overlaid one on the other using Difference mode. I used various tools to verify that all pixels were zero (or to see how far off they were from 0).

    Adobe Camera RAW CS6
    As expected, the two images were identical.

    This proves that it is possible to produce identical images from both a CR2 and its matching DNG, so in some sense the two images are “equivalent”. It is possible, of course, that performing the same adjustments to the two images might not have produced equivalent results.

    Capture One v12.0.3
    With Capture One, I selected each image and right-clicked to export the default variant as a 16-bit PSD using the ProPhoto RGB color space. I turned off cropping and output sharpening and included all metada.

    The exported files did not have the same dimensions. In order for both images to come out with the same size and geometry, I had to set the lens profile of the CR2 to “Generic” (setting both to the correct lens produced images that were slightly different).

    With the “Generic” lens profile, the two images were again absolutely identical.

    DxO PhotoLab v2.2.2
    PhotoLab doesn’t have PSD export, so I exported 16-bit TIFFs. There were no differences in the generated images.

    Affinity Photo 1.6.5.135
    Affinity Photo is more of a Photoshop replacement with a built-in RAW processor. I was not able to get the colors to match—the CR2 had a warmer look.

    ON1 Photo RAW 2019.2
    ON1 was unable to produce PSD files that matched—one was brighter than the other. I exported the files from the browser, in the hopes that the same default adjustments would be applied to both. To support this belief, no ON1 sidecars were produced.

    I tried once more, this time going into the “Edit” module and either turning off or zeroing every setting. Not only were the colors still mismatched, but I noticed some differences in lens correction even though I had specifically turned off this setting for both images

    Conclusion
    It’s clear that it is possible for applications to create identical images from a vendor-specific RAW file and its DNG equivalent. This doesn’t mean that they will. Some do, some don’t and some can but only with some fiddling.

    As far as I know, there is no standard for how a RAW image should be rendered. If you can accept the different renderings that come from switching to a different application, then the differences between RAW and DNG might be moot. The DNG rendering might be inferior or superior (or both, if evaluated using different criteria).

    Are DNG files the only RAW files that are read/write?
    Files are files. Unless protected, they can be read and written.

    Most applications avoid modifying RAW files, even DNG RAW files. Some exceptions:
    • Adobe will embed XMP data into a DNG if it can. Make the DNG file read-only, and Adobe will create a sidecar XMP.
    • Canon’s DPP software writes its adjustments into the CR2 file.
    • exiftool can alter metadata in a RAW file.
    No one seems to change sensor data, not even in a DNG file. View a DNG with a non-Adobe application or view a CR2 with something other than DPP and you will see the original image.

    Advantages and Disadvantages of DNG files
    Pluses
    • DNG files can be produced with more compression that a camera might have time for, so they might be smaller in size. As processing power increases, this advantage might be reduced (for some advanced cameras, this may already be true).
    • DNGs have a checksum to validate for corruption.
    • DNG files offer you some options for the size of the JPG preview (this might produce both a space and time savings).
    Plus/Minus
    • XMP data is usually written into a DNG file. This makes it easier to deliver a single file with all adjustments. The downside is it may create bigger backups and it seems to be a feature only used by Adobe software. I couldn’t find anyone else writing into a DNG file. As stated above, you stop this by making DNG files read-only. Some other vendors have their own packaging format (Capture One has one called EIP).
    Minuses
    • Camera vendors that offer software (such as Canon’s DPP) typically ignore all files except those produced directly by their cameras. Once in DNG format, you cannot use these tools.
    • Converting from another format to DNG takes time and adds an extra step to the workflow.

    References



     
    Last edited: May 2, 2019
    • Winner Winner x 1
    • Informative Informative x 1
  2. GrantWoj

    GrantWoj TPF Noob!

    Joined:
    Apr 11, 2019
    Messages:
    15
    Likes Received:
    3
    Can others edit my Photos:
    Photos OK to edit
    Thanks for the write up, this was super informative for me. I just recently bought my first camera and ran into this issue trying to use CS6, I converted it but don't have enough knowledge/experience to investigate/understand the differences, this was a great write up to clear up my confusion. This makes me feel less concerned about if I should try to get Lightroom or a version of PS that supports RAW editing right out of the camera.
     
    • Like Like x 1
  3. freixas

    freixas TPF Noob!

    Joined:
    Jun 11, 2010
    Messages:
    113
    Likes Received:
    22
    Location:
    Portland, Oregon
    Can others edit my Photos:
    Photos NOT OK to edit
    Thanks! I appreciate hearing that this was useful.
     
  4. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    44,843
    Likes Received:
    17,052
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    On "some types" of raw data, converting to .DNG really saves a lot of file storage space...specifically on Fuji S2 Pro .RAF files. Converting from .RAF to .DNG cuts file storage requirements by approximately 50-60%.
     
  5. freixas

    freixas TPF Noob!

    Joined:
    Jun 11, 2010
    Messages:
    113
    Likes Received:
    22
    Location:
    Portland, Oregon
    Can others edit my Photos:
    Photos NOT OK to edit
    Good note, Derrel. You were careful to highlight "some types" as I'm sure you realize that it's not true in all cases.

    For my 70D test image, the original was 23,776 KB and the DNG was 23,310 KB—not much difference. My suspicion is that this becomes more typical with the pro cameras as they have more horsepower to compress their RAW files. One of the comments I read while doing my research said that D810 NEF files shrank a lot as DNGs, but that D850 NEF files were about the same size (I'm relying on my memory of the camera models). As the DNG converter is free, it's pretty easy for anyone to check whether this would be an advantage for them.

    I converted the RAW with a full-size JPG preview. A smaller preview would also save some space.
     
  6. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    44,843
    Likes Received:
    17,052
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit
    Yeah, file storage savings by converting to .DNG are not uniform from camera-to-camera..as I mentioned, the Fuji S2 used a weird 'honeycomb" sensor. and up-rezzed all raw images 12-megapixel, in-cam...from a 6-MP sensor.

    Perhaps that is why the substantial file storage savings on .RAF files?
     
  7. Derrel

    Derrel Mr. Rain Cloud

    Joined:
    Jul 23, 2009
    Messages:
    44,843
    Likes Received:
    17,052
    Location:
    USA
    Can others edit my Photos:
    Photos OK to edit


    808 D800 files from last winter--23.8 Gigabytes as NEFs..

    The SAME 808 raw images, converted to .DNG, 18.78 gigabytes... about a 5 megabyte size saving per image, give or take Screen Shot 2019-05-02 at 4.36.50 PM.jpg
     

Share This Page

Search tags for this page

should i convert to dng