Basic question about way retina-optimized apps work

Basically, it works as follows. The core idea is to differentiate between 'points' (which are the logical pixels all applications are aware of) and native pixels. On a traditional screen, in a traditional OS 1 point = 1 pixel. That is, if you render a 60x20 points button, it will have the dimensions of 60x20 pixels on the screen and if you render a full-sized Full HD video it will have the dimensions of 1920 × 1080 pixels on the screen.

Now, on a HiDPI (retina) screen, each point is 2x2 pixels. The applications still work with points though . So the same button stays 60x20 points as far the application is concerned, but the OS renders it as 120x40 pixels. Because pixels are so small, this retina button actually has the same size as a non-retina 60x20 button. Something very similar happens to the video: the application just tells the OS "hey, I want to display 1080p content, so give me a 1920x1080 window". But because dimensions are points and not pixels, this window will be 3840x2160 in the retina mode and the video will be upscaled (on scaled 1920x1200 resolution the OS X renders into a '4x supersampled' internal buffer 3840x2400 and then downscales it to to the native 2880x1800).

This video upscaling happens because the application is not 'retina'(HiDPI)-aware or just not care and thus asks for a 1920x1080 points display area. A HiDPI-aware application can detect that point ≠ pixel and ask for actual 1920x1080 pixels (which would be 960x540 points). This is what Apple demonstrated with Final Cut Pro.

BTW, per default, OpenGL 3d content is rendered pixel-doubled to improve performance. So if the application asks for an 500x500 points OpenGL window, the OS will always give it a 500x500 pixels window and than upscale (pixel-double) it to 1000x1000 when the window is drawn. Again, HiDPI-aware applications can control this as well.

Now, as to the question how many applications out there are already HiDPI-aware - I have no idea.

Hope it could clear things up for you a bit.

The Idiot's Guide to UI Elements on a Retina Screen:
User Interface (UI) elements in non-Retina application are scaled once in "Best for Retina" mode, and twice in the "Scaled" modes.

Let's take a hypothetical UI element which is 10 pixels tall and 10 pixels wide. In "Best for Retina" mode this element would be stretched to 20 by 20. Each individual pixel in the original element will be interpolated into four pixels on the stretched version. Interpolation introduces "blurriness", but because of the super-high resolution of the Retina screen, this is offset to some degree.

In Scaled mode the situation is somewhat worse. The non-Retina UI element undergoes its doubling (to 20 by 20), and then undergoes a non-integer scaling to convert the virtual 1680 screen to the real 2880 screen. Our doubled 20 by 20 element becomes a rather unwieldily 17.143 by 17.143, along with inheriting another level of blurriness. 1 pixel of the original UI element (or any non-Retina image - www for instance) is now mapped to 1.7143 pixels of the screen, through two blur-inducing resampling stages.

Again, this is offset somewhat by the Retina effect of the screen (where pixels have less importance than previous). Some people will notice this added blurriness, some will not.

The thing that worries me is that the 2880 Retina screen is not a standard size, so Apple cannot take advantage of the specialist hardware built into the rMBP's GPUs. All of this resampling is done in software. At the moment this software resampling is - supposedly - done on the CPU, but Mountain Lion will move this load onto the GPUs.

Rendering the original 10 by 10 UI element directly on the 2880 screen would make it unusable small, and Apple are using the Retina effect to give "virtual" resolutions to avoid this problem.

Retina-aware applications are displayed in a 1-to-1 fashion on the 2880 screen, but they start off twice the size of non-Retina applications, so they appear at a usable size on the screen (just much clearer). However, under the scaling-modes even the Retina-aware applications go throughout the same (second) non-intager scaling as the non-retina applications to convert from the virtual resolution to the real 2880 screen.

DiGriz said:

The Idiot's Guide to UI Elements on a Retina Screen:

Click to expand...

Do you call it 'Idiot's Guide' because it is full of misconceptions or is there another reason for this?

DiGriz said:

Let's take a hypothetical UI element which is 10 pixels tall and 10 pixels wide. In "Best for Retina" mode this element would be stretched to 20 by 20. Each individual pixel in the original element will be interpolated into four pixels on the stretched version. Interpolation introduces "blurriness", but because of the super-high resolution of the Retina screen, this is offset to some degree.

Click to expand...

Wrong (or, mostly wrong). Usually, UI elements are vector-rendered (for example, buttons or text). Vector rendering is always crisp, there is no interpolation and no blurriness. Only bitmap images, OpenGL views and custom-drawn views are interpolated (pixel-doubled) - and that also only for non retina-aware applications. As I mentioned before, retina-aware applications can choose finer levels of control.

DiGriz said:

In Scaled mode the situation is somewhat worse. The non-Retina UI element undergoes its doubling (to 20 by 20), and then undergoes a non-integer scaling to convert the virtual 1680 screen to the real 2880 screen. Our doubled 20 by 20 element becomes a rather unwieldily 17.143 by 17.143, along with inheriting another level of blurriness. 1 pixel of the original UI element (or any non-Retina image - www for instance) is now mapped to 1.7143 pixels of the screen, through two blur-inducing resampling stages.

Again, this is offset somewhat by the Retina effect of the screen (where pixels have less importance than previous). Some people will notice this added blurriness, some will not.

Click to expand...

Again, this is not entirely correct. On the 1680x1050 HiDPI resolution, the OS renders everything into a big offscreen 3360x2100 buffer. So your 20x20 UI element will be first drawn to this buffer. Then, the buffer is linearly interpolated (downscaled) to the native 2880x1800 screen. Since downscaling preserves more details than upscaling the quality stays very good, although, as you correctly point out, some blurriness may be perceived as the result of interpolation.

DiGriz said:

The thing that worries me is that the 2880 Retina screen is not a standard size, so Apple cannot take advantage of the specialist hardware built into the rMBP's GPUs. All of this resampling is done in software. At the moment this software resampling is - supposedly - done on the CPU, but Mountain Lion will move this load onto the GPUs.

Click to expand...

No idea what this is supposed to mean. There are no "standard sizes" in GPUs since I don't know... the 2000 or so? The GPU will happily interpolate the non-power of two textures for you, and this is what OS X uses it for. This interpolation is also very fast. Even the Intel IGP can do it at very fast rates (several hundred times per second). If the interpolation were done at the CPU, you'd see much higher CPU usage values in the activity monitor.

DiGriz said:

Retina-aware applications are displayed in a 1-to-1 fashion on the 2880 screen, but they start off twice the size of non-Retina applications, so they appear at a usable size on the screen (just much clearer). However, under the scaling-modes even the Retina-aware applications go throughout the same (second) non-intager scaling as the non-retina applications to convert from the virtual resolution to the real 2880 screen.

Click to expand...

Here I again have no idea what you are talking about. If you are talking about 'native' 2880x1800 mode (which you can't select via standard pref pane), the retina-aware and non-retina aware apps behave exactly the same way under it - they are both unusably small. Here, logical pixels = real pixels and both are very small.

If you are talking about 'best for retina' mode, this is HiDPI 1440x900 (where each logical pixel is represented via 2x2 real pixels). Again, retina and non-retina aware apps are both 2x2 scaled, to appear the same size as they would at the 1440x900 monitor. The only thing that retina-aware (we really should start calling them HiDPI-aware, btw.) applications can do is recognize when they are run on a HiDPI display and adjust accordingly (e.g. render custom UI at higher internal resolution). If the application does not adjust explicitly, the OS will pixel-double the content that needs adjusting (again, these are bitmap images, OpenGL views and custom-drawn views).

leman said:

Do you call it 'Idiot's Guide' because it is full of misconceptions or is there another reason for this?

Click to expand...

Yes. Specifically aimed at people who think they know more than they do...

leman said:

Wrong (or, mostly wrong). Usually, UI elements are vector-rendered (for example, buttons or text).

Click to expand...

Last time I checked all UI elements were still bitmaps. The screen grabs of non-retina applications with blurry UI elements would lend credence to this. Surely if Apple had move everything to vector then all applications would be - by default - Retina ready; but they're not, ergo you are talking out of an orifice not usually used for such.

leman said:

Again, this is not entirely correct. On the 1680x1050 HiDPI resolution, the OS renders everything into a big offscreen 3360x2100 buffer.

Click to expand...

...which is the "virtual 1680" screen. Are you being deliberately obtuse, or is this genuine?

leman said:

Then, the buffer is linearly interpolated (downscaled) to the native 2880x1800 screen.

Click to expand...

...which is exactly what I said.

leman said:

Since downscaling preserves more details than upscaling the quality stays very good...

Click to expand...

...and punch in the arm is better than a kick in the balls; It's still better to have neither. Anyway, you're just plain wrong; downscaling loses more detail than upscaling, it's just not as obvious. And the net result is still 1.714 upscale, so you've still lost-out twice.

I hate to think how badly you're being fiddled by your accountant.

leman said:

No idea what this is supposed to mean. There are no "standard sizes" in GPUs since I don't know... the 2000 or so?

Click to expand...

Really, so the fact that everyone else conforms to the standard resolutions is just a mere fluke of probability?

leman said:

Here I again have no idea what you are talking about.

Click to expand...

I know, I clearly pitched the "idiot's guide" too high.

leman said:

If you are talking about 'native' 2880x1800 mode (which you can't select via standard pref pane)...

Click to expand...

Why would I be talking about that in simplified guide? Seriously?

leman said:

If you are talking about 'best for retina' mode, this is HiDPI 1440x900 (where each logical pixel is represented via 2x2 real pixels).

Click to expand...

I think there's an echo in here.

leman said:

Again, retina and non-retina aware apps are both 2x2 scaled...

Click to expand...

Um, not they're not. Non-Retina applications are pixel-doubled, Retina applications are already rendered at twice the size. That's why non-retina applications are blurry and Retina ones are sharp. If you had to upscale the Retina applications, they too would be blurry. You can't invent detail, you've been watching too much CSI. E n h a n c e !

DiGriz said:

Last time I checked all UI elements were still bitmaps. The screen grabs of non-retina applications with blurry UI elements would lend credence to this. Surely if Apple had move everything to vector then all applications would be - by default - Retina ready; but they're not, ergo you are talking out of an orifice not usually used for such.

Click to expand...

Then stop 'checking' and start reading Apple developers docs. Buttons, frames, text etc. are all vector-rendered.

This page might help for starters: http://tinyurl.com/d7kr95d

DiGriz said:

...and punch in the arm is better than a kick in the balls; It's still better to have neither. Anyway, you're just plain wrong; downscaling loses more detail than upscaling, it's just not as obvious. And the net result is still 1.714 upscale, so you've still lost-out twice.

Click to expand...

No it is not - and the reason is exactly because (vector) UI is rendered at a higher resolution in the first place (see above). It does make a difference if you upscale a low-res image or downscale a high-res one - the later will usually look better. Of course, you are right about the bitmap context.

DiGriz said:

Really, so the fact that everyone else conforms to the standard resolutions is just a mere fluke of probability?

Click to expand...

Modern RAMDAC modules can work with custom resolutions. The RAMDAC on the RMPB must be a particularly fast one to be able to output that many pixels. But I am sure that you don't want to say that the RMBP bypasses the RAMDAC and uses the CPU to send the data directly to the display.

DiGriz said:

Um, not they're not. Non-Retina applications are pixel-doubled, Retina applications are already rendered at twice the size. That's why non-retina applications are blurry and Retina ones are sharp. If you had to upscale the Retina applications, they too would be blurry. You can't invent detail, you've been watching too much CSI. E n h a n c e !

Click to expand...

Again, see first part. If the application uses Cocoa-provided controls and does not do anything exotic like drawing its own controls or using offscreen-rendering, it is already retina-enabled and no pixel-doubling will occur.

Overall, I see that you tried to keep the things simple, which is a good thing. Unfortunately, some of your explanations can be understood in the wrong way. Basically, you sound like if a application which is not specially compiled with some sort of retina-specific code will be pixel-doubled. This is plainly wrong. Most applications will look perfect on the retina screen, because OS manages the HiDPI rendering automatically. You only need to adjust the software if a) you use bitmaps (only images), b) you do some sort of exotic things like custom control drawing.