Challenge: Noticeable differences between RAW and a jpeg edited in 16 bit mode?

[Ysarex]

Yes okay there are like 30 pixels of clipping when you mask it like that. I really disagree that that amount of clipping you show in the clipping mask you made above is anything very significant to worry about. I am sitting there myself straining to look at that exact part of the image, knowing full well what I'm supposed to be seeing that's so "butt ugly" and am unable to even MAKE myself see it as bad looking at all. It's a white cloud, 0.5% of it is pure white, and the areas around that part gradient smoothly into that pure white, not abruptly . So what?

The camera seems to have done a pretty darn good job in general in choosing a sampling range from the RAW in order to meet your request of very low contrast

However:

The lighting contrast can get much higher and from this point on your camera can only fail worse.

This is a fair point. Based on the fact that it had any clipping here, I imagine that if you used 0,-4,0,0 settings on that first photo you took of the backlit tree, it would indeed have failed pretty badly even at those maxed out negative 4 settings.

I'll try to test it later when I get home with my camera, with a bright lamp in front of a black window shade or something, but I suspect that you're right, and it'ss going to be fairly bad.

That's a lot more than 30 pixels in the full res version of the photo. Bottom line: Camera software can't do it. We've known this for a long time. You can repair a JPEG with decent success if it's taken under less extreme lighting conditions but really why? Having a uniform workflow is helpful. Many of us find not having to deal with anything except evaluating the lighting and nailing the exposure is in fact a real advantage. And I would much rather process a raw file than try to repair a JPEG. I find it easier and I get it done faster. I like faster.

It was obviously a mistake to give you a photo where the camera only failed slightly. Yesterday when I went to take that photo I had to also take out the trash. So I started out in the back alley and the first thing I did was photograph my neighbor's backlit roses in the backyard. The camera software failed miserably, but I went ahead around front to get you that shot of the house where the camera fail was less obvious. My mistake. Here's the roses:



I'm not spending a lot of time with this so I just popped the raw file through Photo Ninja made sure the highlights weren't clipped (cause they're not in the raw file) did a slight rotate and crop and got the contrast about right. The camera JPEG is on the right. Compare the sky and the clouds in the sky. There is no clipping in the raw file and the exposure is excellent. The camera software failure is massive. You want to verify the JPEG (or process it) here it is: roses.

If so, then yes, that would lead me to the conclusion that shooting RAW is helpful in normal circumstances practically. However, this doesn't seem like it SHOULD be the case theoretically. Rather, it is a failing of Canon's camera settings that you can't make them strong enough to meet your desires. I.e., it only lets you have some small proportion of the influence that you should have over the RAW->Jpeg sampling pattern, for no apparently good reason.

It's not just Canon. I can assure it's all of them. Their position is reasonable. The point of a JPEG is that it's finished. If you're using the camera to deliver a finished JPEG then there have to be appropriate limits to how far the software adjustment controls can be pushed before you end up with something that doesn't look at all finished. For those who require the control, insist on being able to shoot difficult conditions and intend to edit their photos the camera manufacturers have provided the appropriate access.

Joe

 

[amolitor]

Joe makes an excellent point.

JPEG is not intended as a format that "contains all the sensor information, so that you can process it into a final image", there's already a format for that which is much better suited to that role. It's intended as a SOOC picture format, or at least to render acceptably SOOC.

 

[Gavjenks]

In light of the apparent failure of software, I propose a new intermediate format available from camera software, described in this thread:
http://www.thephotoforum.com/forum/...w-nor-jpeg-hypothetical-idea.html#post2962371

Check it out!

As mentioned, jpeg is intended as a finished SOOC. RAW is intended for hardcore editing with no compromises or any attempt at all to cut down on pointless data.

Which leaves a pretty obvious unfilled niche of an in between format that compresses things which don't matter, but intentionally preserves data in the areas that do. See thread for a proposal to do exactly that.

It would require no fiddling in the field to work right.

 

[o hey tyler]

Facepalm.jpeg

 

[480sparky]

Facepalm.jpeg

 

[Benco]

In light of the apparent failure of software, I propose a new intermediate format available from camera software, described in this thread:
http://www.thephotoforum.com/forum/...w-nor-jpeg-hypothetical-idea.html#post2962371

Check it out!

As mentioned, jpeg is intended as a finished SOOC. RAW is intended for hardcore editing with no compromises or any attempt at all to cut down on pointless data.

Which leaves a pretty obvious unfilled niche of an in between format that compresses things which don't matter, but intentionally preserves data in the areas that do. See thread for a proposal to do exactly that.

It would require no fiddling in the field to work right.

For there to be a niche there would need to be something wrong/difficult/compromised about raw...but there isn't.

 

[Helen B]

There's already compressed raw, which works out at about 8 bits per pixel - and if it is raw from a Bayer or X-Trans sensor there is only one channel per pixel.

 

[Derrel]

Who here knows anything about Kodak's ERI or extended range imaging JPEG format? They had that in the 14n, and I think the SLR/n and SLR/c models as well.

It's been over a decade, but it seemed like ERI JPEG format was going places. Buuuut then again, Kodak was "going places" back then too...like down the drain...

Rob Galbraith DPI: Kodak ERI technology brings power of RAW format to JPEGs

 

[Ysarex]

In light of the apparent failure of software, I propose a new intermediate format available from camera software, described in this thread:
http://www.thephotoforum.com/forum/...w-nor-jpeg-hypothetical-idea.html#post2962371

Check it out!

As mentioned, jpeg is intended as a finished SOOC. RAW is intended for hardcore editing with no compromises or any attempt at all to cut down on pointless data.

Which leaves a pretty obvious unfilled niche of an in between format that compresses things which don't matter, but intentionally preserves data in the areas that do. See thread for a proposal to do exactly that.

It would require no fiddling in the field to work right.

No data is pointless until a human being has evaluated it and made a determination. In the end this is really all about just one thing. F****ing software is f****ing blind. I can see, software can't. Keep the f****ing software the h*ll away from my data. Thank you.

Joe

edit: had a couple beers.

 

[Garbz]

Which leaves a pretty obvious unfilled niche of an in between format that compresses things which don't matter, but intentionally preserves data in the areas that do.

You mean like Compressed NEF which even offers you the choice of a reduced bitrate to save space and a choice between lossy and lossless RAWs and is available on Nikon cameras, and I'm sure Canon has an equivalent?

I don't understand your need to re-invent the wheel. Either you want the data for editing or you don't. If you want a little data then compress the RAW and drop the bitrate down.

 

[Gavjenks]

Guys... I linked to the thread about this topic, and you're discussing it here. that's not a great way to get responses back on comments or questions.

Very briefly: Compressed NEF and other RAW compressions are not at all the same. They are either lossless, or blind to the image details (like ERI or the block based cRAW), and thus their loss can potentially be catastrophic compared to RAW if it happens to be the wrong kind of image that doesn't optimize them. I am not particularly surprised that they failed, for this reason.

What I am suggesting is a lossy compression, but where the loss is custom targeted to the areas of the data that have few or no pixels in them and that therefore don't actually matter or require high data density. The data that was being "lost" was data that wasn't needed by the photographer for anything. This is the only compression format mentioned here thus far, as far as I can see, that actually uses the image to intelligently figure out where to compress, instead of just applying the same algorithm to every pixel or every 16 pixels or whatever.

For example, if you have 3500 different values of green shades that NO PIXELS in your image have, then why the hell are you taking up bandwidth (in the form of requiring larger color codes to potentially code for that color) when don't even exist in your image??? None of the methods mentioned above would compress that out.

Getting rid of those would be considered "lossy" in hardcore computer science terms, but would not in fact correlate to any loss of data that the photographer actually cares about. This is how my algorithm works (though very slightly more aggressively, as it doesn't have to be compeltely empty of pixels to be compressed, but pretty close to this, honestly).


Anyway, further discussion of that thread should go in that thread, and I don't intend to check back here to follow up on this post (where in fact another equally promising idea already exists in a pixel compression format instead of rgb/l based format). Link again:
http://www.thephotoforum.com/forum/...w-nor-jpeg-hypothetical-idea.html#post2962371