Exposing to the right

[Ysarex]

The Red Green Blue graphs are my real data, but it's showing JPEG? I thought the raw capture was my real data. So you're saying the RGB histogram is NOT wrong just that it's not my raw data which therefore must not be real data so the JPEG must be my real data. But if my raw data is my real data and the RGB histogram is showing JPEG then it must be wrong for my real data. Sounding very confused here.

Joe

If you shoot Raw, you get Raw, and the only problem is, the rear LCD shows RGB, and cannot show Raw. Nor can our computer monitors show Raw. Our tools show RGB.

So Raw files include a small embedded JPG with the camera properties in it (WB, contrast, etc), to be shown on the rear LCD. And since Raw files are not RGB, my notion is the RGB histogram obviously comes from that JPG too.

Raw does not have gamma either, but gamma only changes the data between 0 and 255, and a quirk of exponents (anything to the power of 0 or 1 is 0 or 1) is that gamma cannot change 0 or 255 endpoints ( so no concern about gamma, it cannot induce clipping).

But WB can shift the dickens out of it, esp red and blue channels in opposite directions, and so WB can shift data to cause clipping. Raw could not care less about WB yet, but unless your JPG is ballpark WB, the histogram could be a bit different than what the final RGB produce later from Raw. Histogram shows that shifted JPG histogram.

Auto WB is fairly poor, but I use it with Raw, as a quicky way to see ballpark color on the rear LCD, to be unconcerned about clipping. It is halfway close, unimportant if it is exactly right or not.

So if I think the raw capture is my real data then the RGB histograms displayed on the camera are not showing me my real data. They are graphs of how the camera processing software has turned my real data into a JPEG.

Joe

 

[WayneF]

So if I think the raw capture is my real data then the RGB histograms displayed on the camera are not showing me my real data. They are graphs of how the camera processing software has turned my real data into a JPEG.
Joe

OK, if you want to get picky. The JPG has no effect, that is just compression. But yes, the JPG is showing the camera settings, like WB, contrast, etc... the way the camera would process the Raw. And the Raw later in computer will show a histogram of the final RGB the way you process it in Raw. The JPG won't show any other ways you may later process it either. If "same as" is your goal, then shoot JPG. Hopefully, you intend to make it better than the camera could.

My previous point, WB does not have many proper choices, so if Auto WB or other gets it close, then it is close. No concern to me. The point of Raw is to make it look you want it to look.

And the earlier point, if you are watching the single luminosity histogram, none of the later subtleties are of the slightest interest to you.

 

[Ysarex]

So if I think the raw capture is my real data then the RGB histograms displayed on the camera are not showing me my real data. They are graphs of how the camera processing software has turned my real data into a JPEG.
Joe

OK, if you want to get picky. The JPG has no effect, that is just compression. But yes, the JPG is showing the camera settings, like WB, contrast, etc... the way the camera would process the Raw. And the Raw later in computer will show a histogram of the final RGB the way you process it in Raw. The JPG won't show any other ways you may later process it either. If "same as" is your goal, then shoot JPG. Hopefully, you intend to make it better than the camera could.

My previous point, WB does not have many proper choices, so if Auto WB or other gets it close, then it is close. No concern to me. The point of Raw is to make it look you want it to look.

And the earlier point, if you are watching the single luminosity histogram, none of the later subtleties are of the slightest interest to you.

Yep, picky is good here because the Red, Green and Blue histograms displayed on the camera are in fact not histograms of the raw sensor capture and are not showing the status of my real data.

Later in the computer I can see histograms of my raw data before I process it to a final RGB photo as well as histograms of that final RGB photo.

I don't know what you mean by "if 'same as' is your goal," but of course my processed photos from the raw files are better than the camera could process them.

I thought your previous point was, "And the one showing the three Red, Green, Blue graphs, which is your real data." But now we know that's only if you're not picky.

Joe

 

[apaflo]

The camera actually embeds a full sized JPEG in the RAW file, as well as a small version for preview on the LCD.

I cannot see inside the NEF file, but I don't see any evidence of that.

Here is NEF only, and NEF + Fine. On fixed tripod, on unchanged subject.

First line is NEF, followed by NEF + Fine. Second NEF is 106 KB larger (why, I don't know).

If the JPG is 30 MB, and if it is in there, why isn't the second NEF 30 MB larger?

Apparently you wanted something to show there, but I don't see whatever it was.

Regardless, NEF files actually have perhaps three different sized JPEG images embedded. There is a full sized JPEG, there is a smaller JPEG for preview, and there is an even smaller one used when multiuple images are shown on the LCD screen.

EXIFTOOL is a program that can extract the embedded JPEG images. From the NEF file, using options "-b -JPGfromRAW" extracts a fullsized JPEG, options "-b -OtherImage" extracts a smaller JPEG, and "-b -preview" extracts and even smaller JPEG. From a Nikon D4 the sizes are 4928x3280 for the fullsized, 1632x1080 for the small, and 570x375 for the even smaller JPEG. If the camera produces a JPEG file it is possible to extract the same 570x375 preview file from the fullsized JPEG too. I'm pretty sure there is also a 120x80 image embedded as a comment in one or more of those JPEG images but I can't remember the convoluted method to extract it.

Other brands of camera, and for that matter older models of Nikon cameras, have different embedded JPEG images in the RAW files, but as far as I know virtually all of them do have something there.

 

[table1349]

 

[bribrius]

Why would zeroing out the meter give unexpected exposures? I've always used the meter to let me know if I'm getting a proper exposure, which to me means that I'm getting the proper amount of light coming in to the camera. How I get there is where shooting manual comes into it - I can adjust the aperture, or if I want to maintain a certain depth of field I can adjust the shutter speed, or vice versa, or some of both.

There might be situations where you'd want to adjust from what the meter is indicating, if the meter is fluctuating when the light is changeable (sun in and out) etc. When I'm shooting B&W film sometimes I bracket shots to make sure I have a negative that isn't too thin or too dense, but I have yet to figure out why people or websites say to always under or over expose.

When I'm shooting digitally (I'm a film photographer too) I usually shoot Raw and all manual and I don't get blinkys etc. so I go by what the meter indicates as to whether or not I'm getting the exposure I need, then adjust from there.

That's happened to me and not just here. "its under exposed" someone says. Im thinking yeah, no ****, it was taken at night it is supposed to look dim lighted. I bring up the light. Then someone else says " you blew out the highlights". well yeah, no ****. There is only so much you can jack up the light on something taken at night to make it look like it was taken during the day to appease those who might think it is underexposed.
People throw this stuff around quite a bit. which is why I pay attention on here to learn, but I don't take it all seriously? Because unless they are with you, when you take it, to see it first hand what the scene is. Then they are really guessing about some of this stuff and haven't a clue what they are talking about. NOt to mention I had zero comments on a photo I put on here (must have thought it was bad?) that is now hanging on someones wall. Another I got negative reviews on here, (but watch me sell it). So I listen for what I can learn and ignore a good portion of the rest. Happy to see this thread it turned out well. Im picking up some good info on here just reading along. im going to read it through again when I have more time.

 

[sk66]

If you think the histograms (RGB or Luminance) or blinkies shown in the camera are accurate you would be mistaken. Quite possibly significantly. Even using a Uni-WB will only get you close... and I found it not worth the hassles.

In fact the histograms are precisely accurate! They just aren't necessarily displaying what photographers want displayed. They are accurately showing exactly what is in the camera generated JPEG image, not what is in the RAW sensor data; nor is it what will be in a RGB image (JPEG, TIFF or whatever) generated with an external computer program. In particular White Balance, Brightness, and Contrast parameters will all drastically change how an histogram looks.

If you want to see an histogram that accurately displays what is in the RAW sensor data, in particular to use for ETTR or even just generally setting exposure, it is a fact that UniWB can be made as accurate 1/10th of an fstop. And yes that can be a hassle, given that you give up being able to use the in camera JPEG as a preview (don't use that JPEG to show Granny what she looks like, or you'll get tossed out the door!)

Commonly what is worth doing in many cases but not all, is adjusting the camera JPEG generator for low relatively contrast, and the using the brightness level to cause the histogram to be very close in terms of the exposure. It will usually be within 1/4 of a stop, and virtually always within 1/2 a stop. Hence adjusting it to be about 1/3 a stop high in brightness will provide a RAW file that never clips highlights and is always easily processed to provide nearly the maximum dynamic range the camera is capable of.

I can agree with all of that... I found when using a Uni-WB I *also* had to adjust the Jpeg settings in order to get it relatively close.
Since ETTR is not of much use to me (mostly wildlife/action in variable lighting) I found it not worth the hassles.

 

[WayneF]

Later in the computer I can see histograms of my raw data before I process it to a final RGB photo as well as histograms of that final RGB photo.

No, Adobe (or whatever Raw editor) is not showing a histogram of the Raw data. Nothing does. The Raw editor is not showing Raw data at all. It is necessarily converting Raw to a RGB image, to be able to compute a RGB histogram, and to be able to show anything on your RGB monitor. You can guide that conversion, as you please. Camera has the same issue on its RGB rear LCD screen, Raw cannot be shown.

So, I guess you can think of the "real data" as being any point you please, but RGB is all you can see (it is necessarily your own only starting point).

What is clear is that the single gray histogram in the camera is not conceivably real, and can be extremely misleading.

 

[WayneF]

Apparently you wanted something to show there, but I don't see whatever it was.

Regardless, NEF files actually have perhaps three different sized JPEG images embedded. There is a full sized JPEG, there is a smaller JPEG for preview, and there is an even smaller one used when multiuple images are shown on the LCD screen.

EXIFTOOL is a program that can extract the embedded JPEG images. From the NEF file, using options "-b -JPGfromRAW" extracts a fullsized JPEG, options "-b -OtherImage" extracts a smaller JPEG, and "-b -preview" extracts and even smaller JPEG. From a Nikon D4 the sizes are 4928x3280 for the fullsized, 1632x1080 for the small, and 570x375 for the even smaller JPEG. If the camera produces a JPEG file it is possible to extract the same 570x375 preview file from the fullsized JPEG too. I'm pretty sure there is also a 120x80 image embedded as a comment in one or more of those JPEG images but I can't remember the convoluted method to extract it.

Other brands of camera, and for that matter older models of Nikon cameras, have different embedded JPEG images in the RAW files, but as far as I know virtually all of them do have something there.

I did read your post wrong, I cannot find now what I thought it said (regarding outputing BOTH file types). I was expecting more size from a sum. Compression is unknown, but we see Fine JPG is 30MB, and we know uncompressed 12 bit Raw is 1.5x larger than uncompressed 8 bit RGB, and so the 50MB sum seems insufficient for both.

I use ExifTool, it is all I have found that can show most of the newer D800 Exif format. I had not bothered with the command line functions, but looking today, I see this (-ee) in a Raw file:

Subfile Type : Reduced-resolution image (so 160x120 RGB uncompressed)
Image Width : 160
Image Height : 120
Bits Per Sample : 8 8 8
Compression : Uncompressed
Photometric Interpretation : RGB


Subfile Type : Reduced-resolution image (Size not specified, but it says reduced resolution)
Compression : JPEG (old-style)
X Resolution : 300
Y Resolution : 300
Resolution Unit : inches
Jpg From Raw Start : 1001984
Jpg From Raw Length : 3982880
Y Cb Cr Positioning : Co-sited


Subfile Type : Full-resolution Image (the Raw data)
Image Width : 7424
Image Height : 4924
Bits Per Sample : 14
Compression : Nikon NEF Compressed
Photometric Interpretation : Color Filter Array
Strip Offsets : 4985344
Samples Per Pixel : 1
Rows Per Strip : 4924
Strip Byte Counts : 41391624
X Resolution : 300
Y Resolution : 300
Planar Configuration : Chunky
Resolution Unit : inches
CFA Repeat Pattern Dim : 2 2
CFA Pattern 2 : 0 1 1 2
Sensing Method : One-chip color area


Subfile Type : Reduced-resolution image (same again? no size)
Compression : JPEG (old-style)
X Resolution : 300
Y Resolution : 300
Resolution Unit : inches
Other Image Start : 176640
Other Image Length : 825308
Y Cb Cr Positioning : Co-sited

I suppose extracting them would show a size.

I am wondering if "extract a full size JPG" means it creates one for us, or if it was already in there. I wonder where the bytes come from? I guess I am not yet a believer about a full size JPG.

 

[sk66]

The RGB histogram is NOT wrong (255 is NOT ambiguous). But yes, it is showing JPG with camera WB, when you may be using Raw with your own WB.

The single Luminosity graph is not even close to anything.

Here's a test I did for the D800 using an image designed to push histograms from Cambridge in Color website.

Here's the histograms in camera. Jpeg settings set so that the image looked as accurate as possible to the original.


Here's the same image opened in PS (raw; LR looks very similar). The histograms here look very much as the test image was designed to make them (some minor differences due to not capturing the entire original image)


These are both of the exact same file. Some will note that there is obviously some interpretation being done to the raw file in order to display it... I did everything possible to make it "neutral" (i.e. adobe standard, as shot, etc.). These differences are simply a fact of life and can't be entirely avoided. Even using a Uni-WB I wasn't able to get the Jpeg histograms "accurate," but I could get the luminance "closer."

The D800 seems particularly bad, my D4 is MUCH closer without any tweaking. This may be due to the "level" the camera is targeted to...I don't know.
I believe a lot of what we call "headroom" in the raw file is due to the manufacturers' intent to protect the harsh clipping characteristic of a digital sensor (i.e. the REI/reported ISO being higher than actual sensitivity); and many cameras do not use 255 as the point for "clipping." (item 9 in the link). They use something more like 245-250.

IMHO, this is not necessarily "bad;" whites above ~245 and blacks below ~10 are not of much good for display/print. The D800 also seems to be applying some "perceptual bias" to the histograms and shifting them based upon their "perceptual weight" (30/59/11% R/G/B). and the luminance histogram is more of a "stack up" of the others instead of a true luminance graph.

What do the manufacturers have to say about this? Here's the disclaimer in the Nikon owner's manual: "camera histograms are intended as a guide only and may differ from those displayed in imaging applications." I.e. differ from "the real ones." Canon doesn't even bother to mention it.


edit to add: it is interesting to note that upon initial opening into PS the histograms will show the camera jpeg "cached" histograms with an alert/update icon. LR does not show the "cached" histograms.

 

[chuckdee]

Understanding the basics of exposures is the first thing you should learn. Once you learn that, everything will start to make a lot more sense and will make you a better photographer. Do a search for F16 sunny rule. Print out a chart and keep it with you when you go and shoot. When you know what exposures are right for certain conditions, you can then make on the fly adjustments and can pinpoint a certain area of your frame for the correct exposure.

 

[Gavjenks]

If you shoot Raw, you get Raw, and the only problem is, the rear LCD shows RGB, and cannot show Raw. Nor can our computer monitors show Raw. Our tools show RGB.

Many programs on the computer can and do show the RAW histogram (the cameras could too, they just don't).
See for yourself. Set your camera to some really ridiculous settings like +maximum contrast and +maximum saturation (so that the jpeg histo will be noticeably much different than the RAW), then snap a photo in RAW+jpeg mode.

Open the jpeg in photoshop and check out the histo, then open the RAW in different RAW editors and look at the starting histo. In any program I've ever used, they are quite different, suggesting that the RAW converter really is showing the RAW histogram. And there's no good reason why it wouldn't do that, since the histogram is just counting up numbers of dots with each color/lightness, and the pattern of them doesn't matter.

And if you want an accurate histogram in camera, just set all your settings to "neutral" or "zero" or whatever so that it makes minimum adjustments when converting to jpeg and thus minimal differences between the RAW and the LCD histogram. It still won't be absolutely perfect, but almost. The higher bit depth of the RAW doesn't matter for the LCD, because the LCD doesn't have that kind of resolution anyway.


Of course, if you ever shoot jpeg only, then don't set all your settings to zero, because you'll get horrible looking images. it's only if you know you'll be editing from RAW later ("custom shooting modes" are good for this to switch all the settings instantly)

 

[apaflo]

I suppose extracting them would show a size.

I am wondering if "extract a full size JPG" means it creates one for us, or if it was already in there. I wonder where the bytes come from? I guess I am not yet a believer about a full size JPG.

Exiftool will not create or generate anything. It extracts data from tags.

But it doesn't take much to realize there necessarily has to be a full sized JPEG available while the RAW file is available on the camera... to allow the magnified view available on the LCD.

 

[AlanKlein]

I use a thirty-year-old Mamiya RB67 Professional medium format film camera for my landscape photography. All manual. I use a hand held meter, usually in incident mode, to calculate the correct exposure. I then dial in the aperture, shutter and focus adjusting for depth of field. I then release the shutter in mirror up position to reduce vibrations.

And then I bracket two more - one stop above and one stop below. Just to be sure.

Isn't manual terrific?

 

[Ysarex]

Later in the computer I can see histograms of my raw data before I process it to a final RGB photo as well as histograms of that final RGB photo.

No, Adobe (or whatever Raw editor) is not showing a histogram of the Raw data. Nothing does. The Raw editor is not showing Raw data at all. It is necessarily converting Raw to a RGB image, to be able to compute a RGB histogram, and to be able to show anything on your RGB monitor. You can guide that conversion, as you please. Camera has the same issue on its RGB rear LCD screen, Raw cannot be shown.

So, I guess you can think of the "real data" as being any point you please, but RGB is all you can see (it is necessarily your own only starting point).

What is clear is that the single gray histogram in the camera is not conceivably real, and can be extremely misleading.

I didn't say Adobe (or whatever Raw editor) was showing me a histogram of the Raw data. You're making false assumptions. I said, "Later in the computer I can see histograms of my raw data before I process it to a final RGB photo..."; see illustration below.

So I guess I do think the sensor raw capture is my "real data" and I do examine it carefully all the time to make sure I'm getting the best possible exposures. It would be nice if I could do that in the field but I'm happy with a process where I review my "real data" later and use that review to inform my work behind the camera. For me it's a more precise method than relying on the Red Green and Blue histograms displayed on the camera which we know have been interpreted by the camera processing software and do not directly represent my "real data."

This thread began with a question about ETTR. To practice ETTR you need a way to examine the raw data as directly as possible to assess how you're exposing the sensor. Looking at data derived from the camera processed JPEG (camera manufacturer histograms of whatever flavor) is indirect and although it may be useful and possible to interpolate from that information it's off mark to suggest that it's anywhere near the "real data."

Joe