To help better illustrate my point, I believe this is the situation... as I've illustrated below. The Normal Histogram we see is misleading in that it implies data is clipped when it is not. I guess I'm wondering why we can't have a proper RAW histogram that actually shows when data is really being clipped? Is this technically impossible for some reason or are camera and software companies lazy, or am I the only one that sees a need for this?
Technically lazy isn't exactly the case. There are multiple aspects to it, and if anything marketing departments are the lazy ones. They try to make things too digestible at times, which results in a lot of inaccurate intuition.
I by no means know the answer and am by no means competent in the matter. I think it's because of the way the RAW files are made. Say current Canon cameras use 14bit raw files. However, a camera like a 7d spreads it's dynamic range over something like 12.x bits. (not sure where I read that *citation needed) So if the RAW converter showed you the full 14 bits you could have a clipped image that doesn't look clipped on the histogram. Also the dynamic range of the camera isn't set in stone, you know how you can still get data from an over exposed image but it's ugly and noisy, where do you draw that line? Also that lines tends to move with the ISO at which the image was shot and probably temperature (not to mention between same model cameras).
To add to this, the number of bits stored is independent of the range described. It's just how many values are used to describe that range. It can get a bit confusing at times when you look at things like 32 bit floating point modes in different software where they don't just reduce rounding errors but provide a way to store values over a greater range. What often confuses people is that this is a function of the way the format is set up rather than the number of bits that describe it. They just need a lot of bits to make such a descriptive method work without banding. In the case of digital cameras, you're looking at camera data with a dynamics range that exceeds typical working profiles and greatly exceeds display gamuts. The raw data isn't gamma corrected. It's not necessarily precisely linear in its nature, but it's not baked/profiled to a specific gamma encoding. Anyway a lot of cameras have a greater range than typical working profiles, so sometimes you have leverage. If they're really blown out, sometimes 1-2 channels may still contain detail. De-bayering data has some interpolation involved either way. In that case it could be way more, thus the reduced quality. Blah I wish I could better organize this.
The problem is more complex and there are a few dfferent cases
1) The JPG file is "clipped" and has some pure white areas while the raw file is not clipped
It's partly that they have different clipping points. You'll sometimes see it referred to as clamped data as well.
Agreed, but it doesn't explain why we're only given a JPEG histogram to work with in any and all tools.
I'm not sure. It could be they want you to see the results as they should normally look. It's typically more extreme lighting situations where you would need to pull out a lot of extra range at the edges, although raw processing has some tools for dealing with things such as extremely saturated details where the resulting color values would otherwise cause clipping in some channels with default processing and relative colorimetric conversion.
The histogram represents the in-camera JPEG conversion - which has a contrast curve applied to it, not the Raw file - which has linear gamma.
The camera is "squashing" your highlights into a JPEG.
Well gamma encoding isn't necessarily evil. Without that how would you really view things meant to appear continuous in tone over a limited dynamic range?
I suspect you're right, but we should start demanding better tools. Wouldn't it be helpful to have the choice of histograms presented in post #3? So you could tell what is clipped in the JPEG, what is recoverable from RAW data, and what is clipped in RAW?
The tools have evolved somewhat relative to computing power. Ideally you'd be able to access that entire range while a program like photoshop merely interprets it to display as gamma 2.2 Adobe RGB or sRGB or whatever else you choose (not opening that debate). It's not like your display can handle the number of values that would be needed to properly show something without heavy gamma correction. The fact that the most popular raw processors haven't come up with something like an export to 32 bit .EXR or .HDR probably means it's a niche request.