"Nonlinear bitrate" image format, neither RAW nor Jpeg. Hypothetical Idea

[Gavjenks]

So RAWs are at 12-14 bits or so. Jpegs are at 8.
The difference ends up being a difference of a factor of 16 or 64, respectively, in the number of different steps you can have along color or luminosity values. This gives you extra latitude in editing, because you can pick and choose which areas you want to sample more densely (like almost blown out highlights) and which to sample less densely in going from RAW to the final version you send to print or post on the internet, etc. In other words, more dynamic range to choose from (even though you end up with the same dynamic range).

However, RAWs take up way more space and take longer to process, etc. Why not have a format that compromises between the two? Why are camera companies ONLY offering you either a completely unedited format (aimed at people who want high image technical quality), OR an inflexibly edited version (aimed at people who want to use images SOOC)? Surely some people want something in between (I am one).

There's no reason why they couldn't also offer a format that flexes as needed, preserving PORTIONS of the dynamic range of RAW only where they are needed based on the resulting histogram from an image. Giving you the size and speed of a jpeg, but not sacrificing any of the precious data where you need it. Only where you DONT need it.

It could go something like this:

1) The camera calculates a luminance histogram from the RAW itself, or at least the closest rectangular grid approximation. (all these steps would work equally well for color. I'm just using luminance as an example). Simple arithmetic.
2) The camera superimposes the shape of the histogram so that its average value falls on the dotted line on the graph shown below. Also simple arithmetic.
3) The camera adjusts the shape of the histogram to account for the fact that you can't go higher than 12 bits (this involves clipping the parts above 12, and then pulling up the average value of every other point on the curve so that the average is once again right on the dotted line. Also simple arithmetic.).



4) Okay so now the camera is going to save it in an 8 bit format. Which means 256 lightness values. BUT, critically, each of those 256 values does not have to cover an equal portion of the dynamic range that RAW provides. They can cover VARIABLE portions of the dynamic range. As in, value 125 might cover a range of 50 different lightness values from RAW, whereas value 126 might only cover 35 values from RAW, etc. So we need to determine how big each bin is. How? Well, we already did. That's what that red line does for us.
5) The height of the red line at each point corresponds to what portion of RAWs dynamic range that value covers. Let's say that your camera has RAWs with 12 bits. If so, then if the red line is on the dotted line, then that value (for instance, approximately value 30 in the example above?) will average across 16 lightness values in the RAW (12 bit - 8 bit = 2^4 = 16). If the red line is at maximum (12 bits), then one value in the file will correspond to one value in the RAW (NO loss of data). If it is at 4 bits, then one value will correspond to 256 values of lightness in the RAW (12 - 4 = 2^8 = 256).
6) Save the data along with a "key" that tells you what the value of the red line was at each point (this only adds a couple of kilobytes or whatever to your image)
7) The highlights and very darkest shadows could, by default, be biased to have a bit denser sampling.

When you're done, the overall file size will equal the size of a normal 8 bit jpeg, but it will be nonlinearly sampled, to best suit your image. So in this example, the highlight will be at maximum density sampling with the RAW, meaning you will have almost just as good of an ability to recover highlights or do any other editing as if you had shot RAW. but in a jpeg sized file.


There are no decisions to be made in the field, because the density of data will auto-scale to where it's needed. Any large smooth areas that need to have smooth gradients will be densely.

When you load it into a computer, the computer can then use the key to decode it back to a normal grid with almost the same RELEVANT data density as the original RAW did.

A toy example:


The bars on top are a histogram. Imagine that the top row of boxes are a "RAW" file in an imaginary system where the data is 3x as dense in RAW as in the imaginary jpeg-style format. Each box = one value of lightness. This is how it samples currently, more or less. Each lightness value corresponds to an equal number of values in the other format. This is true regardless of the density of the pixels. Notice that the first of the compressed boxes doesn't correspond to even one pixel! Yet it is still using up some of our data bandwidth. At the same time, alll of the actual pixels in the image are being crammed into one single lightness value, and all that data is getting pointlessly thrown away. Wasteful.

In the proposed system, it would map like this:



The first box covers the whole empty area, and then the second two boxes are devoted to storing the majority of the data that is actually in the photo at a higher than otherwise resolution. The very top lightness value is actually equally as dense as the RAW data, where density and precision are most relevant to this particular photo.

Thoughts?




P.S.: two other random details:
1) You could select in camera menu how much you wanted to compress (doesn't have to be 8 bits. If we are gonna make our own new format anyway, might as well let people select how much compression they want).
2) In reality, the red line above would only go in steps of 1 bit at a time, for technical computer reasons.

 

[Overread]

My initial thoughts

1) Most of the time most people will go to the extremes. They either want the full range of data to edit with or they want a straight out of the camera print ready and easy to use file.

2) How much faster is this going to be over RAW? I mean in a serious world example since I can, if I want to be fast with RAW, just copy-paste them into my computer and run them through Lightroom (or any RAW processing software that does pre-set settings and batch work) and its done. Few moments and its sorted. If I want to spend longer editing chances are I'll be back to point 1 in that I'll want the whole file not bits of it .

 

[Gavjenks]

It would be vastly faster than RAW in the field, in terms of transferring to your data card. And it would use up much less space. These are the main advantages.

It would not be any particularly faster than RAW in post processing. Essentially it would just decompress itself INTO a raw when you load it on the computer (this step goes very very quickly, faster than almost any edit you could do. Almost as fast or faster than loading a jpeg in photoshop). Then everything you do after that would just take a normal RAW amount of time.

 

[Helen B]

You know that compressed raw (cRAW) already exists? It can be compressed into 8 bits per pixel, using compression based on small blocks, or lines. The first pixel of the block is 16 bit, then the difference for the rest of the block is encoded at 8 bits per pixel.

 

[slow231]

because not every raw conversion is just about filling the bins equally. sometimes you want a non-average exposure, sometimes you want dead bands, sometimes you want significant areas of highlights (or low lights), sometimes you want compressed binning in one place and not in others. it all depends on the image. a pure raw file lets you determine that non-linear mapping after the fact. my raw conversions are all over the place depending on what fits that particular image, i would hate to have only one standard conversion algorithm!

or you can think of it this way. your proposed algorithm will tend toward trying have the same amount of pixels in each bin (flat histogram), look at any raw conversion (done manually, made to look as good as possible to the eye) and you'll see that the histograms are not necessarily flat!

also in your example, consider the 3 "compressed" bins as black, grey, and white. The original image is pretty bright, with almost the majority of the pixels in the very bright range, and just a bit in the grey. using your mapping algorithm, now you now have black, and a whole lot more grey in the image. this is not at all an accurate representation and could very well lead to a pretty unwanted rendition of the scene.

 

[Gavjenks]

You know that compressed raw (cRAW) already exists? It can be compressed into 8 bits per pixel, using compression based on small blocks, or lines. The first pixel of the block is 16 bit, then the difference for the rest of the block is encoded at 8 bits per pixel.

Okay, but:
1) What cameras is that in?
2) It sounds like it would compress equally small, but if you do it by blocks of 16 pixels, it is not as well tailored to the image. Areas without much detail don't even need 8 bits for change from the key pixel (like the sky), and areas with lots of detail need more than 8 bits. Mine is better able to adapt to the image.

Of course, you could also just do what you're describing, but with variable amounts of bitrate used to hold change data, depending on what is needed, for each block of pixels. That would be similar to what I'm suggesting, but based on pixels instead of lightness and RGB values, and would probably work just fine, yes.

because not every raw conversion is just about filling the bins equally.

I think there's some misunderstanding here. It will indeed attempt to make a flat histogram for STORAGE temporarily. Not for your final image. Once you decompress it again in computer, it will go back to being a very very non flat histogram, almost exactly like the original RAW was.

This isn't making your image flatter or grayer. it has nothing to do with artistry or changing the look of your image. It's purely a compression technique, which if successful is designed to make your image more or less indistinguishable from a RAW (even the ability to recover highlights, etc.), since the compression is coming from bandwidth that you wouldn't have been using anyway for that image if you had a RAW.

In fact, the reason the compression idea would work is precisely BECAUSE the histogram that you actually want is NOT flat.

 

[slow231]

I see, you're just compressing with the intent to uncompress back to the full raw resolution. the confusing part is that you're determining your binning by looking at histograms which show the number of pixels (over the entire frame) at each tonal range. and yes if you keep track of your original histogram you can recreate your original histogram (in it's raw tonal resolution), but the histogram contains no information about where those pixels were located.

let's take your example. the original raw file has 9 tonal values (1-9). and let's just estimate the heights of the bins based on your picture. lets say 1 pixel @ a value 4, 5 pixels @ a value 7, 6 pixels @ a value 8, and 15 @ a value 9. we compress into 3 tonal values (mapped as you have shown) let's call them I, II, and III. we now have an image with 1 pixel @ I, 11 pixels @ II, and 15 pixels @ III. Now if we kept track of the mapping we know that the pixels of value III correspond to a raw value of 9, and this works out ok. Similarly the pixels of value I correspond to raw values between 1-6, but since we kept track of the raw histogram we know there is only one non-empty raw bin in this range, so II corresponds to raw value 4, and this works out ok. the problem comes in when you compress multiple raw bins into one compressed bin (as with bin II). in this case we know bin II corresponds to raw values 7 + 8. and since we kept track of the raw histogram we know that of these 11 pixels now marked II, 5 of these should have value 6, and 6 should have value 7. but which of these 11 pixels were originally 6 and which were 7? that's where the information loss occurs.

your method takes advantage of sparsity in the tonal resolution range. and this will work (losslessly) only if the number of non-sparse raw bins <= the number of bins you are compressing to (not sure how often would actually occur). but once you start combining multiple raw bins into one compressed bin, information is lost.

 

[Benco]

It would be vastly faster than RAW in the field, in terms of transferring to your data card. And it would use up much less space. These are the main advantages.

It would not be any particularly faster than RAW in post processing. Essentially it would just decompress itself INTO a raw when you load it on the computer (this step goes very very quickly, faster than almost any edit you could do. Almost as fast or faster than loading a jpeg in photoshop). Then everything you do after that would just take a normal RAW amount of time.

I can only speak for myself but I find that writing raw in camera is just not an issue, using decent cards it's fast enough that I never find myself waiting for it to write, when things get slow is when lightroom is grinding its way through the raw files and as you say there'll be no particular advantage to be gained there anyway.

If I want to get images out of camera and away fast then I'll use JPG and the editing and max IQ can go hang. As in your previous thread on raw vs 16 bit jpg you're apparently seeing a problem where none exists.

 

[Garbz]

It would be vastly faster than RAW in the field, in terms of transferring to your data card. And it would use up much less space. These are the main advantages.

We have these things called buffers and large memory cards.

Seriously your threads are starting to sound like complaining because you don't want to pay / can't afford a better camera. I have some news for you, the cheaper the camera the more crippled the software. This is the reason why Magic Lantern exists. Why would anyone implement yet another non-standard solution like this when they already have their own RAW formats most which include compression when it seems like the only customer who isn't currently happy is you?

 

[Gavjenks]

I see, you're just compressing with the intent to uncompress back to the full raw resolution. the confusing part is that you're determining your binning by looking at histograms which show the number of pixels (over the entire frame) at each tonal range. and yes if you keep track of your original histogram you can recreate your original histogram (in it's raw tonal resolution), but the histogram contains no information about where those pixels were located.

let's take your example. the original raw file has 9 tonal values (1-9). and let's just estimate the heights of the bins based on your picture. lets say 1 pixel @ a value 4, 5 pixels @ a value 7, 6 pixels @ a value 8, and 15 @ a value 9. we compress into 3 tonal values (mapped as you have shown) let's call them I, II, and III. we now have an image with 1 pixel @ I, 11 pixels @ II, and 15 pixels @ III. Now if we kept track of the mapping we know that the pixels of value III correspond to a raw value of 9, and this works out ok. Similarly the pixels of value I correspond to raw values between 1-6, but since we kept track of the raw histogram we know there is only one non-empty raw bin in this range, so II corresponds to raw value 4, and this works out ok. the problem comes in when you compress multiple raw bins into one compressed bin (as with bin II). in this case we know bin II corresponds to raw values 7 + 8. and since we kept track of the raw histogram we know that of these 11 pixels now marked II, 5 of these should have value 6, and 6 should have value 7. but which of these 11 pixels were originally 6 and which were 7? that's where the information loss occurs.

your method takes advantage of sparsity in the tonal resolution range. and this will work (losslessly) only if the number of non-sparse raw bins <= the number of bins you are compressing to (not sure how often would actually occur). but once you start combining multiple raw bins into one compressed bin, information is lost.

What? No it keeps track of where each pixel is by recording them in order... its just that once it gets to a pixel, it only stores its rgb and lightness with 8 bit codes that map to the original values according to my algorithm.

Of course you may also use a lossless compression on top of that, like png or gzip, if you have many now-identical pixels in a row. Png is fast.

And yes information loss occurs. Im not a moron. But its smartly allocated to not matter. Where you would actually need data gor editing like blown out highlights, you have plenty of it. Where you dont need it it is thrown out

 

[Gavjenks]

It would be vastly faster than RAW in the field, in terms of transferring to your data card. And it would use up much less space. These are the main advantages.

We have these things called buffers and large memory cards.

Seriously your threads are starting to sound like complaining because you don't want to pay / can't afford a better camera. I have some news for you, the cheaper the camera the more crippled the software. This is the reason why Magic Lantern exists. Why would anyone implement yet another non-standard solution like this when they already have their own RAW formats most which include compression when it seems like the only customer who isn't currently happy is you?

Thats right. So what? We would still all be using view cameras exclusively if customers never demanded technologically possible improvements in more affordable models. This post makes no sense. They can do it at almost no cost and make some expensive things cheaper thus beating competition. So they should as a smart business. Simple as that.

Are you concerned that you might not get to be as snobby anymore if your camera's abilities start to cost closer to what they should and become available to average people? Because that's the only logical reason I can think of for somebody being annoyed at the prospect of cheaper gear with more capabilities.

Also more memory and buffer are inherently wasteful and inefficient brute force solutions that will fail to work when we hit the limits of things like flash memory miniturization. Smarter algorithms can function beyond those barriers though and with less waste. This an ultimately more forward thinking solution.

 

[Garbz]

Ok challenge for you. Code up your algorithm in C and apply it to a RAW file. Let see if the end result is computationally simpler and uses less storage than a compress NEF at reduced bitrate. I'm not saying we should have better features in cheaper cameras. I'm saying what you're proposing is a waste of time in the presence of oh so many alternatives.

If compressed NEF isn't small enough for you then petition manufacturers to implement 16bit JPEG2000. With lossy compression you can get similar quality/filesize ratios to actual JPEGs and we don't need to deal with yet another non standard completely unsupported esoteric format that no one uses or knows about.

 

[Garbz]

Quoting from your other thread:

I am not particularly surprised that they failed, for this reason.

Failed at what? Most manufacturers implement compressed RAW, all software supports it, and many people use it which doesn't sound much like a failure to me and actually some of the compression algorithms don't sound too dissimilar from what you're suggesting here.

What I am suggesting is a lossy compression, but where the loss is custom targeted to the areas of the data that have few or no pixels in them and that therefore don't actually matter or require high data density. The data that was being "lost" was data that wasn't needed by the photographer for anything.

Nikon's NEF compression works in a similar way already by applying a custom quantisation curve to the data and cramming the result into something like 9.5bits of data. The decoding curve is then embedded into the RAW file. The difference is you're trying to guess what data is important and other compression algorithms work on visual perception of data. e.g. JPEG trashes the blue channel because it's not important in colour pictures which aren't all blue. The above mentioned NEF compression favours shadows over highlights as the non-linear nature of our vision means there's an excess of data in one area vs another. But hey RAW = bad right?

I think you're re-inventing the wheel.

 

[Gavjenks]

I dont own a nikon anymore but when I did, the raw files were all exactly the same size plus or minus a couple kilobytes. And they were something like 4x as large as jpegs still. If they were using any sort of aggressive compression then files would differ significantly in size, and it doesnt have to be nearly 4x as large.

A quick googling confirms a mild lossless compression for nef format which isnt even available on all models.

So they could still benegit from a cheap lossy algorithm alternative in between jpeg and nef using smart compression, like combinations of optimizations you mentioned and the algorithm in the OP.

Also I never claimed no such similar thing exists in the universe. However ive never seen it offered in any actual cameras (in a way that actually accomplishes useful results) as of writing this post. What camera actually has the ability to select variable user selected smart compression designed for later editing, in camera?

 

[Garbz]

If by mild you mean in lossless there's a 50% reduction in file size, and in lossy and bit reduced 75% reduction in file size, vs a JPEG's 85% all while having 769 more possible data points per channel then I agree with you. By the way this isn't from google. I just took a few pictures at different settings to check. A compressed NEF is only about double the size of a JPEG.

Lets see I have the choice of NEF, NEF Lossy, NEF Lossless, NEF Reduced bitrate Lossy, NEF Reduced bitrate Lossless, TIFF, JPEG L, JPEG M, JPEG S and if you're desperate on file size then each of the JPEGs and TIFFs break down into Large, Medium and Small. This gives me the choice of files ranging from about 100MB down to 1MB. Why the hell would I want yet another choice?

The reason you idea won't take off is because frankly there is no point. As mentioned compressed RAW already biases the data where it's needed in a way that makes it useful for most photographers and gets you within 10% of JPEG's compression ratio. You're proposing yet another format in between for what purpose again? Slightly faster write speeds to the camera?

You idea doesn't fail on technical grounds. It's a good one. But it fails social grounds. I think I speak for many here when I say I'd much rather see DNG implemented natively in cameras than some esoteric compression algorithm which I'm still struggling to find a use for.