Apple Patent for Resolution Independent User Interface

[MacRumors]

http://www.macrumors.com/images/macrumorsthreadlogo.gif (http://www.macrumors.com)

Macsiumumnews reports (http://www.macsimumnews.com/index.php/archive/apple_patent_is_for_resolution_independent_user_interface_design/) on a recently published patent application for "Resolution Independent User Interface Design". Resolution Independence has been described by Apple (http://www.macrumors.com/pages/2006/10/20061024174114.shtml) as a feature for the upcoming version of Mac OS X 10.5 (Leopard). The reasoning is described:

The old assumption that displays are 72dpi has been rendered obsolete by advances in display technology. Macs now ship with displays that sport native resolutions of 100dpi or better. Furthermore, the number of pixels per inch will continue to increase dramatically over the next few years. This will make displays crisper and smoother, but it also means that interfaces that are pixel-based will shrink to the point of being unusable. The solution is to remove the 72dpi assumption that has been the norm.

According to the patent application, traditional user interface designs are created for specific resolutions. In order to support arbitrary resolutions, images have to be up or down-scaled which may introduce blurring or artifacting. They propose a method to describe the user interface objects in a "procedural" and resolution independent manner.

[OdduWon]

wow could have much to do with the new ipods and isight enabled cimema displays:D

[Xavier]

Was actually just thinking of this today, because my 15 inch screen is soo small. What would happen when I upgrade to the next generation of 50" screens?

[koobcamuk]

I just worry things are going to get too small for us to read - unless we have 50" screens to compensate. One friend already commented on how things on my MacBook feel smaller and higher res compared to his iBook.

Furthermore, I think we will see a lot more ctrl + Zooming going on in the future - but an automated style - that didn't pixelate or break up (if you understand this). i.e. I hope it kind of focuses. Like spaces, but all in one place. Just an idea. Who knows. I could be talking ass.

[trudd]

Makes me wonder how hardware intensive that would be. Would it be more CPU or GPU based? If its GPU based, it makes me glad I bought the MBP...

[Aldyn]

niceeeeee

i use the zooming feature all the time when laying in bed and using my imac from a distance.

[CTYankee]

as a photographer this concerns me. Right now I have control of how people see the images on line for proofing or porfolio. I sure don't want them to look bad because of some automatic resolution changes. Just a small change in resolution can often make a good image look not so good.

[aspro]

This will be a great improvement, especially in a few years down the track my Macbook screen is about the limit I would like.

[Mudbug]

god help all of us professional designers out there trying to explain resolution to clients... I hope this makes sense when/if it comes to see the light of day.

[technocoy]

this is cool, but it's going to be t-totall HELL for us who do web design... dear god, it's already hard enough trying to explain web resolutions to clients as it is...

hahaha.

oh well.

[displaced]

I just worry things are going to get too small for us to read - unless we have 50" screens to compensate. One friend already commented on how things on my MacBook feel smaller and higher res compared to his iBook.

Furthermore, I think we will see a lot more ctrl + Zooming going on in the future - but an automated style - that didn't pixelate or break up (if you understand this). i.e. I hope it kind of focuses. Like spaces, but all in one place. Just an idea. Who knows. I could be talking ass.

That's exactly what a resolution-independent GUI is intended to address!

As resolutions increase, there's more pixels on-screen, so GUI elements of a fixed number of pixels per physical inch become smaller (the problem you're describing, and how OS X's GUI currently operates).

A resolution-independent GUI has the ability to cleanly re-scale or dynamically recreate GUI elements to compensate. As resolution increases, the number of pixels used to draw each GUI element increases so that elements have the same relative size between resolutions. Obviously, at higher resolutions they'll look much crisper, since there's more pixels available to describe them.

Think of it as being similar to a game like Doom 3. As you crank up the resolution, the area you can see doesn't increase so that individual elements are smaller. Rather, the clarity with which characters and environments increases.

[Max Payne]

Is this good or bad?

[geerlingguy]

this is cool, but it's going to be t-totall HELL for us who do web design... dear god, it's already hard enough trying to explain web resolutions to clients as it is...

hahaha.

oh well.

It seems this will apply to the user interface, but not necessarily things like Safari web rendering or things like that. But how will buttons and web forms be handled? We'll see.

This is probably something Apple's been working on for at least a few years, so I'm sure they will take into account all the potential problems web designers and the like could have.

Is this good or bad?

Count me as one for 'good.' I would rather not have to squint to find the scroll bars and other buttons in the user interface when I'm using a 20"+ display with huge resolutions.

[koobcamuk]

I thought I understood - but I don't think I do.

[ChrisA]

this is cool, but it's going to be t-totall HELL for us who do web design... dear god, it's already hard enough trying to explain web resolutions to clients as it is...

No this is going to be great. Currently web designer do things like say and image is 400x200 pixels. This was always stupid. Now we will be able to say 4 inches by 2 inches and then if we supply a large file the result looks good or a small file the result looks worse

Basically what they are talking about is making the user interface "postscript-like" where you send a command to draw a line and let the system rasterize it. OK maybe it is OpenGL like, but either way, executable instructions to draw raster then a pre-drawn raster.

This change is at the GUI library level, programmers will see this directly. end users, photographers, web surfers and so on will only see the effect - non-pixelated zooms.

Another case of "technology going backwards". back in the "old days" before RAM was cheap graphic terminals only accepted vector drawing commands. The CRT did not scan it actually moved the beam to trace out each line. It would draw characters like a pen plotter. This was when a megabyte of RAM cost as much as an apartment building. Displays were very clear and scalable in the 60's with not a chance of pixels showing, because there were no pixels. What other 60's tech can they bring back? Multi-core processors? Oh wait..

[CubeHacker]

I thought I understood - but I don't think I do.

Currently, our UI is pixel based. Look at the "down" arrow on your scroll bar to the right. Its a 10x10 pixel box with an arrow in it. Because its pixel based, when the DPI of your monitor increases (resolution/size), those pixels become very very small, making it difficult to see and difficult to click on. If you wanted to increase its size, you would get a huge pixelated square that looks horrible.

A resolution independent UI is vector based, not pixel based. That 10x10 box of pixels is now a mathematical formula for drawing a square. As a vector shape, it can be enlarged or shrunken to any size without distortion, kinda like how modern fonts on the computer work. Thus, if the dpi of your display makes things look to small, you can now increase the size of the UI without making things look ugly.

[D0ct0rteeth]

this is cool, but it's going to be t-totall HELL for us who do web design... dear god, it's already hard enough trying to explain web resolutions to clients as it is...

hahaha.

oh well.

Obviously it isnt an issue yet, but I imagine we will keep developing for the common denominatior and the early adopters are going to be just a fringe group and they will continue seeing whatever they see.

Then as the technology evolves then it will continue to just go that way... Maybe in 2-3 years we are building 150DPI images... or maybe flash becomes a tool to allow images to be dynamically scaled - but I will absolutely say that there wont be a time where mass design firms are designing 3 or 4 sites for multiple resolutions. There will be sites built for the common denominator/target market until technology builds tools to make this development complete by making web design/development able to address this fully.

[JoeG4]

They can't patent this!

Vectorized interfaces have been around a while, and Windows 95 had the same exact implementation of UI scaling that they're talking about now (without the bitmapping). WTF Apple?

Yes, it even had it with the slider that you'd move to scale the DPI of the GUI up. IIRC, I remembered that and suggested it as a simple way to make aqua scalable (yay!)

Still, they can't patent that! Prior art omg.

[thejadedmonkey]

this is cool, but it's going to be t-totall HELL for us who do web design... dear god, it's already hard enough trying to explain web resolutions to clients as it is...

hahaha.

oh well.

Does this mean for any websites I do in the future, I would be better off specifying width in CM or IN instead of PX?

[thejadedmonkey]

They can't patent this!

Vectorized interfaces have been around a while, and Windows 95 had the same exact implementation of UI scaling that they're talking about now (without the bitmapping). WTF Apple?

Yes, it even had it with the slider that you'd move to scale the DPI of the GUI up. IIRC, I remembered that and suggested it as a simple way to make aqua scalable (yay!)

Still, they can't patent that! Prior art omg.

I never remember Windows 95 having scaling. Heck, 98, 2k, me, and XP don't either AFAIK. What can be done, is to enlarge the elements. It's still not completely res independant though, just small/med/large.

[balamw]

They can't patent this!
Read their application. It's a particular implementation of a general idea.

B

[Stella]

Another example of stupid patents.

Abuse of.

Why software should not be patentable ( spelling ).

Nokia are adding similar to their phones.

[shawnce]

I thought I understood - but I don't think I do.

Say you have a line 1 inch long that is 1/72 of an inch thick. On Mac OS X (and Mac OS before it) that one inch line would be defined as 72 points long and one point thick since Mac OS X defines it point space (what you draw in) as having 72 points per inch.

Currently Mac OS X defines that one point maps to one pixel on a display.

So if you had a display that had 72 pixels per inch (physical pixels crammed into one inch of length along the display's surface) then a 72 point long line would appear to be 1 inch long on that display. Now if you had a display with 144 pixels per inch then a 72 point long line would appear to be 1/2 inch long on that display and on a 288 pixel per inch display it would appear to be 1/4 inch long.

So as you get displays that can cram more and more pixels per inch your on screen drawing will appear to be physically smaller and smaller unless you adjust the scale of what you are drawing. In other words as the fidelity of displays approaches that of common laser printers you need to adjust what you draw to use more pixels so it maintains the same appearance in terms of physical dimensions.

This is what resolution independent UI (http://developer.apple.com/releasenotes/GraphicsImaging/RN-ResolutionIndependentUI/index.html) is about... Mac OS X in the near future will allow you to (or possibly automatically) set the scale factor for the visual environment to match the physical size of pixels of your display.

For example if you had a 144 pixel per inch display you would ideally set the scaling factor to 2x. That would mean a 72 point long line (aka 1 inch long line) that is 1 point tall would be displayed on your screen using a rectangle that is 144 pixels long (72 x 2) and 2 pixels tall (1 x 2). In other words that 1 inch long line would appear as 1 inch long on your display instead of just a half inch long and it would be displayed using twice as many pixels... which means you have a higher fidelity image.

It should be noted that since day one Mac OS X has utilized a resolution independent drawing environment (Quartz / Quartz2D) and that is how what you see on screen is appropriately scaled when you print it out on say a 600+ DPI printer without out much of a through by a programmer (at least for those using modern APIs based on Quartz or using Quartz directly). What Apple is currently doing is starting to utilizing this existing capability to scale what is drawn to the screen since screens are improving in DPI.

[shawnce]

Does this mean for any websites I do in the future, I would be better off specifying width in CM or IN instead of PX?

Review... High DPI Web Sites (http://webkit.org/blog/?p=55), High DPI (Part 2) (http://webkit.org/blog/?p=56), and CSS Units.

[D34thPwny]

mmm... innovation... tasty.

[MongoTheGeek]

Gee sounds a lot like visual postscript...

[MrCrowbar]

LOL, funny to see how few people know what resolution independence does. LCD-Display all have a certain number of dpi (dots per inch). Usually it's 72, 75 or 100 for laptop displays. When you look closely at your screen, you can see those little squares, each of those squares has 3 rectangles (red, green and blue, mixed together you get a square in every color you want). As it is now, cranking up the resolution makes everything smaller, to a point where it's hard to read, click buttons etc. And changing the resolution of a LCD-Display to something different than its native resolution (the one that corresponds to the number of pixels the display actually has), looks terrible.

Resolution independence makes it possible to scale things without changing the resolution. If you have to squint sometimes, make it bigger, if you wish to have more space for windows, make it all smaller. Resolution independence lets you zoom in without getting blurry and loosing details. Try that zoom function (CTRL+scroll up) and you'll see how bad it looks. With resolution independence, it would still be super sharp (until a certain degree I guess). You can't convert everything into vectors or have pictures with infinite resolution, but it should be enough to double or triple the resolution on LCD-Displays. You can't really buy any of those displays yet because you wouldn't be able to read anything without a magnifying glass right now, but who knows, Apple might present new high res displays after Leopard is out giving you super sharp pictures.

[shawnce]

I never remember Windows 95 having scaling. Heck, 98, 2k, me, and XP don't either AFAIK. What can be done, is to enlarge the elements. It's still not completely res independant though, just small/med/large.

Yeah Windows versions prior to Vista have a poor mans version of resolution independence since the drawing environment is generally pixel based for those operating system versions.

Vista is switching to a drawing environment similar to what Mac OS X has (actually since Display PostScript (http://en.wikipedia.org/wiki/Display_PostScript) in NeXT OS). This type of drawing environment allows applications to draw using a fixed coordinate space (for example in terms of points with a fixed value of some number of points per inch) and then have the drawing environment, based on output context/device, map points to pixels (rasterize) as needed without the application having to care.

Also in some cases the output context could be a vector based device (for example a PDF file) and in that case the drawing environment wouldn't raserize any thing but instead preserve the vector information.

[shawnce]

Gee sounds a lot like visual postscript... I assume you mean Display PostScript (DPS)? Mac OS X has had Quartz which provides similar (and greater) capabilities as DPS... nothing new is really being added in this space to allow for resolution independence... the changes are mostly at the Carbon and Cocoa levels to break away from the traditional one point equals one pixel for display output devices. Carbon is the framework that will have the most issues with this... unless folks are using the newer HiView stuff.

[koobcamuk]

Say you have a line 1 inch long...

Thanks for that. Slightly helping. Hope this means that things look nicer! I am not dumb either - just not an artist so wondering how this affects average Joe. Or adam. Or Sam... etc

[shawnce]

god help all of us professional designers out there trying to explain resolution to clients... I hope this makes sense when/if it comes to see the light of day. The control for resolution independence has been available to developers since 10.4 (with various bugs in the Cocoa and Carbon frameworks however stand in the way of full testing).

[shawnce]

Thanks for that. Slightly helping. Hope this means that things look nicer! I am not dumb either - just not an artist so wondering how this affects average Joe. Or adam. Or Sam... etc

It means in the future the text, buttons, other UI widgets, etc. you see on your screen will look like a printed page... nice and crisp.

[MacQuest]

I just worry things are going to get too small for us to read...

:eek:

That is the whole point of this technology.

[koobcamuk]

:eek:

That is the whole point of this technology.

I meant before this comes about.

[shawnce]

as a photographer this concerns me. Right now I have control of how people see the images on line for proofing or porfolio. I sure don't want them to look bad because of some automatic resolution changes. Just a small change in resolution can often make a good image look not so good. I wouldn't worry... any software worth its salt used for viewing images will likely have a way to view the image with a one to one pixel mapping.

[MrSmith]

So a one-inch line will now appear as a one-inch line. Why the negative comments from photographers/designers here?

[portent]

as a photographer this concerns me. Right now I have control of how people see the images on line for proofing or porfolio. I sure don't want them to look bad because of some automatic resolution changes. Just a small change in resolution can often make a good image look not so good.

No. If scaling an image makes it look bad, there are two possiblities:
1. Lousy scaling algorithms, which can make an image look blurry or jagged
or
2. Lousy image.

A good photo scaled by a good algorithm will look decent at all resolutions.

I wouldn't worry... any software worth its salt used for viewing images will likely have a way to view the image with a one to one pixel mapping.

I sure hope not. What happens when we get 150-ppi displays? 300-ppi displays?
Your "full-screen" image will become a thumbnail.

Any designer worth his/her salt will have to learn to work at the resolution of the device. It's not terribly difficult: print designers have been doing this for decades. Unfortunately, web browsers are only beginning to allow the necessary controls.

[shawnce]

No. If scaling an image makes it look bad, there are two possiblities:
1. Lousy scaling algorithms, which can make an image look blurry or jagged
or
2. Lousy image.

A good photo scaled by a good algorithm will look decent at all resolutions.

I sure hope not. What happens when we get 150-ppi displays? 300-ppi displays?
Your "full-screen" image will become a thumbnail.

Recall that many mainstream and most/all professional digital cameras capture images at resolutions that current generations of displays cannot fit fully on screen. So when we get 144 or 288 DPI displays we likely still want to see those images without any scaling so we can fit all of the images pixels on the display.

Mac OS X has some of the best image scalers yet developed (resampler) built-in and accelerated using vector operations (AltiVec & SSE). For example Lanczos (http://en.wikipedia.org/wiki/Lanczos_resampling)5, Lanczos3, etc.

If you want pixel accurate image display you would display it with a one-to-one mapping (yeah it may be small but it is accurate) otherwise you would scale it using a good scaler if you didn't care about accuracy. Basically my point is don't worry... all options remain open to developers and they will provide sensible features to customers given their needs.

[shawnce]

So a one-inch line will now appear as a one-inch line. Why the negative comments from photographers/designers here?

Well in the case of web designers a lot of what they have made and the tools they use work in terms of pixels. So what they have made may not look as good as it could if they had defined things in terms of "points". In general text and layout should be OK because the browser can cover for them in most cases... the main issue is all of those small highly compressed images that are in use... those won't look good scaled.

[portent]

...
If you want pixel accurate image display you would display it with a one-to-one mapping (yeah it may be small but it is accurate) otherwise you would scale it using a good scaler if you didn't care about accuracy. Basically my point is don't worry... all options remain open to developers and they will provide sensible features to customers given their needs.
Ahh, I agree. Pro-level Image editing software will always let you view at one-to-one.

[killmoms]

I wouldn't worry... any software worth its salt used for viewing images will likely have a way to view the image with a one to one pixel mapping.

I'd imagine most image software would do what it does now—map each pixel of the image to each pixel of the screen at 100%. Just that more of the image could fit on the display at once and it would all be smoother and clearer, and the interface around the image could be any size you like.

These are all pretty elementary concerns people—anyone with a solid grasp of the concept of fixed vs. variable resolution and screen vs. print resolution knows what the issues are when programming. I wouldn't worry about it. :cool:

[sushi]

god help all of us professional designers out there trying to explain resolution to clients... I hope this makes sense when/if it comes to see the light of day.
So true!

When the Mac first came out with 72dpi monitors, what you saw was exactly what was printed. If you took a ruler and measured both (monitor and printout) of an object they would be identical. It was truly WYSIWYG.

I hope this change is very transparent to the user. Especially as someone above mentioned with photos and artifacts. Jesus, I can just see it now for those trying to say oh don't worry about what it looks like on the screen, it will be perfect in print. :eek:

[shamino]

as a photographer this concerns me. Right now I have control of how people see the images on line for proofing or porfolio. I sure don't want them to look bad because of some automatic resolution changes. Just a small change in resolution can often make a good image look not so good.
If you rasterize your large image to a small number of pixels, it will look bad no matter what you do. Scaling a low resolution image doesn't reflect a problem with scaling, but a problem in the image.

Right now, your images probably already have a DPI value of some kind in them. So if you want to put up a 2" square, you should scale your image to 2" (Photoshop and many others already allow this), leaving the DPI alone. The pixel-dimensions will probably be larger than 144x144, and that's fine. In your web page, specify the dimensions as 2"x2". The web browser (if properly written) will scale the image to the specified size, showing as many or as few pixels as it takes.

Sure, there may be some artifacts if your pixel-resolution is very low, but for normal pictures, it shouldn't be a big deal. Photographs tend to scale very well (compared with line drawings and renderings that don't use antialiassing.) Of course, some programs scale images better than others. Fortuntely, the scaling built-in to Quartz does a very good job most of the time.
this is cool, but it's going to be t-totall HELL for us who do web design... dear god, it's already hard enough trying to explain web resolutions to clients as it is...
Your pages should specify sizes in terms of real-world units (like inches, milimeters, points, etc.) and not in terms of pixels. Leave the conversion up to the web browser. Then you simply have to provide images with enough DPI so they won't get all jagged when shown on a high-DPI display.

[displaced]

Then you simply have to provide images with enough DPI so they won't get all jagged when shown on a high-DPI display.

I'm just wondering if its possible that different elements of an application could have varying DPIs.

The current resolution independence in Tiger shows that individual windows can have their own DPI values. I wonder if this could be replicated down to individual Cocoa views? So Safari's GUI could be at 120dpi, the actual web page content could be at something lower... Or individual graphic views could be at 72dpi within a 120dpi window?

Interesting.... :D

[Peel]

I'd assume to do this as an automated process the monitor would have to report what it's native pitch is (i.e. dpi). Either that, or when you choose your resolution from the system preferences, you'd have a seperate pop-up menu to select the diagonal screen size, so that the software could calculate your dpi for you.

One of the great things that this will allow is when you choose 100% in a layout program, the visual representation of a layout (for example a 3.5"x 2" business card) will actually scale correctly on the screen. This would be a great help. I can't tell you the number of times that I print something out and am mildly surprised at the actual size of it, after viewing it all day on the screen in what I'm told is scalled at 100%.

[shawnce]

I'm just wondering if its possible that different elements of an application could have varying DPIs.

The current resolution independence in Tiger shows that individual windows can have their own DPI values. I wonder if this could be replicated down to individual Cocoa views? So Safari's GUI could be at 120dpi, the actual web page content could be at something lower... Or individual graphic views could be at 72dpi within a 120dpi window?

Interesting.... :D I don't think I can get into specifics (given NDA and Apple not publishing API docs) but again I will say... don't worry ... Apple is giving developers what they need to deal with these types of issues (including time).

Note some of the API (http://developer.apple.com/documentation/GraphicsImaging/Conceptual/drawingwithquartz2d/dq_affine/chapter_6_section_6.html) is listed (stuff available in 10.4 at least)...

[shamino]

I'll be very interested to see how well this works out in the finished product.

Eventually, I'm sure all operating systems and web browsers will do this, and most pages will be written to render properly, but that won't be the case in the short term.

Right now, we have a lot of images with incorrect DPI values, because everyone assumes that browsers will render the images at one-pixel-to-one-pixel. So someone may scan a 5" CD cover at 300 dpi (1500x1500), and scale it down to 100x100 pixels, leaving the DPI at 300. When shown today, it renders at 100x100. When shown on a resolution-independent UI, however, those 100 pixels at 300dpi will end up being 1/3 of an inch, displaying as 33x33 on a 100dpi display.

This is going to tick off a lot of people until content creators learn what to do. I this particular case, they need to determine the display size in real units (like 1" square), then scale the image to 1"x1". This would be 300x300 at 300dpi, or 100x100 at 100dpi - the important thing being to set the DPI correctly at the same time the image is scaled. This way, it will render at the same 1" square on all displays.

Of course, overall quality when displayed, will depend on the source image. On a 100dpi display, a 300x300@300 image may look better or wose than a 100x100@100 image (because the scaling algorithm in the displaying-system may be better or wose than the scaling algorithm used to generate the image file). But on a 200dpi display (where 1" square is 200x200), the 300x300@300 image will look a lot better than the 100x100@100 image, because scaling-up tends to produce worse artifacts than scaling-down.

I suppose, as a workaround before web sites are changed, browser software will have to have a setting somewhere to tell it to ignore DPI values in images. Maybe to render all images as if they were 72dpi, or only do this for images that don't have DPI values in them. And maybe allow this setting to be overridden on a per-site basis, to keep resolution-independent-savvy pages looking their best.

[shamino]

I'm just wondering if its possible that different elements of an application could have varying DPIs.
Why not?

I should be able to draw some images at 72dpi, and composite them with some 300dpi scanned images and the system should render the results correctly.

Every bitmap used should have a DPI value of some kind associated with it. And non-bitmap data should already be working in terms of physical units. I think Mac OS has always defined its UI as using points and not pixels, so the hard part will be dealing with developers that didn't understand the difference. (Historically, they were the same for displays, but different for printers. I suspect developers who have written printing code in the past shouldn't have a problem with this change.)

[shawnce]

I think Mac OS has always defined its UI as using points and not pixels, so the hard part will be dealing with developers that didn't understand the difference. Yup that will be the biggest issue and one reason Apple has been talking to developers about this since before 10.4.

Those using Cocoa should be in the best spot since it much clearer about the difference between points and pixels (was based on the user space / device space model that Quartz and DPS before it had). Those using Carbon will have more issues to tackle given Carbon's history of being QuickDraw based which is generally pixel based.

[shamino]

I'd assume to do this as an automated process the monitor would have to report what it's native pitch is (i.e. dpi). Either that, or when you choose your resolution from the system preferences, you'd have a seperate pop-up menu to select the diagonal screen size, so that the software could calculate your dpi for you.
I think monitors already report this. At least I remember reading about something about the DDC channel (used for plug-n-play) reporting the screen's physical dimensions.

Update: Yes. Plug-n-play monitors report this. The Extended Display Identification Data (http://en.wikipedia.org/wiki/EDID) provided by monitors includes the physical dimensions, in cm. So system software can combine this with the pixel-dimensions of the screen to determine the DPI.
But whatever the dimensions, I hope there will always be a way to override the settings. I happen to like high resolutions (I use a 20" monitor at 1600x1200 at work - 100dpi. Most people I know prefer lower DPIs on their screens.)

There are times where it is useful to know that 1" on screen matches 1" on paper, but there are also plenty of times where you may want to see more content than this, or less content with more detail. Maybe this could be best done with a global zoom control on the menu bar

Leopard is going to be fun. Hopefully, it will be finished soon.

[shawnce]

So after reviewing the patent application (http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=20060284878&OS=20060284878&RS=20060284878) it looks like the primary focus of the patent is about how to generate and record "recipes" for things like buttons (with reflective lighting effects, complex color gradients, etc.) so that a rendering system can generate a bitmap appropriate for display given the current UI scaling factor.

Example image from patent application (http://aiw2.uspto.gov:80/.DImg?Docid=us20060284878ki&PageNum=9&IDKey=D3950027C8D0&ImgFormat=tif)

SVG like but more tuned for the specific purpose of UI widgets instead of a generic vector based system.

[0006] In one embodiment, the invention provides a method to represent a graphical user interface object's material map in a procedural and, therefore, resolution independent manner. The method includes receiving values for each of a plurality of attributes associated with a material map object, associating a value for each of the plurality of attributes, and storing the plurality of attributes and their associated values in a file. The file may be a "flat" file or a hierarchically-ordered file. The collection of attribute-value pairs comprise a complete description of the graphical user interface object's material map and may be used by a rendering module to create a visual representation of the material map at any number of resolutions. In addition, because material maps in accordance with the invention are represented procedurally, they may be encrypted to prevent unauthorized inspection or use.

I hope Apple rolls this tool out to developers... I know I would love to use it for building recipes for some custom UI widgets that I need to implement with support for resolution independence (not trivial to generate some of the nicer looking lighting and shading effects without falling back on bitmap textures, etc.).

[shawnce]

They can't patent this!

Vectorized interfaces have been around a while, and Windows 95 had the same exact implementation of UI scaling that they're talking about now (without the bitmapping). WTF Apple?

Yes, it even had it with the slider that you'd move to scale the DPI of the GUI up. IIRC, I remembered that and suggested it as a simple way to make aqua scalable (yay!)

That isn't what the patent application is about... :)

[macintel4me]

I get the resolution independence thing for most things. I don't know how icons would work in this world, however, with "procedural" descriptions of images. Maybe instead of one icon, five or six must be included with the app and the OS up-samples or down-samples depending on the resolution and icon size. But then again, that doesn't sound very "procedural" to me.

[shawnce]

I get the resolution independence thing for most things. I don't know how icons would work in this world, however, with "procedural" descriptions of images. Maybe instead of one icon, five or six must be included with the app and the OS up-samples or down-samples depending on the resolution and icon size. But then again, that doesn't sound very "procedural" to me.

Use of PDF for simple icons and multi-resolution icon resource files for complex ones should cover this aspect of things well... of course SVG is another option (in theory).

[Peel]

I think monitors already report this. At least I remember reading about something about the DDC channel (used for plug-n-play) reporting the screen's physical dimensions.

Great. I like it when you find out that your older hardware is compatible with new software. Thankfully whatever committee came up with the Plug-N-Play protocol, added some data on this so that it can be leveraged in the new UI.

[bretm]

LOL, funny to see how few people know what resolution independence does. LCD-Display all have a certain number of dpi (dots per inch). Usually it's 72, 75 or 100 for laptop displays. When you look closely at your screen, you can see those little squares, each of those squares has 3 rectangles (red, green and blue, mixed together you get a square in every color you want). As it is now, cranking up the resolution makes everything smaller, to a point where it's hard to read, click buttons etc. And changing the resolution of a LCD-Display to something different than its native resolution (the one that corresponds to the number of pixels the display actually has), looks terrible.

Resolution independence makes it possible to scale things without changing the resolution. If you have to squint sometimes, make it bigger, if you wish to have more space for windows, make it all smaller. Resolution independence lets you zoom in without getting blurry and loosing details. Try that zoom function (CTRL+scroll up) and you'll see how bad it looks. With resolution independence, it would still be super sharp (until a certain degree I guess). You can't convert everything into vectors or have pictures with infinite resolution, but it should be enough to double or triple the resolution on LCD-Displays. You can't really buy any of those displays yet because you wouldn't be able to read anything without a magnifying glass right now, but who knows, Apple might present new high res displays after Leopard is out giving you super sharp pictures.

Your point is moot because 99% of stuff worth looking at is based on pixels on the internet, NOT vectors. And those 15 years of web pages aren't going to change to vectors any time soon. Gifs, photos, quicktimes, windows media, real players, etc. are all a static pixel size. We're talking logos, product artwork, etc. It all looks good displayed pixel for pixel. Double the res and scale up the interface and the interface will look good, but those pictures will have to be 1/2 the size or be scaled up, forcing them to be interpolated, forcing the to look soft at the same physical size they used to.

Think regular TV on an HDTV television. You've got more than twice the res, but guess what? Regular TV looks worse on an HDTV than it does on a regular TV.

[MikeTheC]

I haven't dabbled with it as my x86 PC died a while ago and so I haven't had a box to dedicate to running Linux, but from what I understand, I think they're rigging it so that, for instance, icons can be authored in SVG format. One can only speculate whether other UI elements and objects would be handled similarly. If so, then you take an object and, as others here have said, do an on-the-fly dynamic rasterization of it, just like what a PostScript printer does, and voila! you have an object of a consistent size and clarity (meaning it doesn't shrink nor grow as the resolution changes, nor does it become pixelated and fuzzy as the resolution drops.

As far as web and other UI-design is concerned, the only thing I can think of which could be a concern to designers are static, fixed-size images (like, for instance, photos). Everything else on the site should work without issue.

My big concern is that you're going to have a situation where such a theoretical web site is going to look totally awesome on a Mac, but like complete ***t in Windows because the web site's default appearance (and how a Win32 environment web browser app displays content) just aren't as high a quality as what an Aqua (or whatever)-environment-based app would be.

Only time will tell... (I sure hope that Apple either adopts an open standard for this or promulgates one so that we don't have lots of incompatible, identically different rendering technologies out there making a complete butcher's job out of surfing the web and so forth.)

[povman]

Only time will tell... (I sure hope that Apple either adopts an open standard for this or promulgates one so that we don't have lots of incompatible, identically different rendering technologies out there making a complete butcher's job out of surfing the web and so forth.)

Actually, a website done properly using CSS should be fully scalable. By 'properly' I mean, the CSS says what the developer actually wanted, eg 50% of page, 1cm etc. So there's a standard already for websites. As for images, people would have to either start moving to SVG or they better specify real sizes for their images in the source. (unless image formats already have support for real sizes?)

[Highland]

Web scaling and UI scaling are quite different beasts (with some similarities). I'm pretty sure this patent would be regarding UI, not web.

Here's a blog post I made specifically about resolution independence and icon design: http://islayer.com/blog/?p=87

There's quite a few links at the top to other related articles.

As for websites and res ind... please have a read of what's coming for WebKit and Dave Hyatt's opinion on where it's all heading.
http://webkit.org/blog/?p=55

The simple answer seems to be that there's no one solution that covers every situation well. Vectors are brilliant for some things, but fail miserably at others. It's pretty easy to include several sizes of a bitmap in one file... OS X's file/application icons already do this (as do OS 9, Windows and most other modern OSs).

povman: using CSS "px" isn't bad... that will just be scaled up. Hyatt suggests that one CSS px may not always equal one screen px. So they can be used and scale perfectly using new techniques (ie. not the current font-sizing seen in most browsers today).

[benbow]

I have not done any graphics programming for over twenty years so most of this thread's scaling and rendering discussion is beyond me.

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps? More likely they use lines and polygons, and fill patterns such as a drawing program might use. The OS then pixilizes or rasterizes the image to the scale requested. Vectors must still exist at some level in the scaling software (deep down) to describe the orientation of the primative line elements, polygons, and their fill patterns / colors.

I also remember writing programs to display scientific data plots using a 1970s monochrome display that could display only dots or vectors, and if one worked at it axes and numbers. Royal PITA.

[Highland]

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps? More likely they use lines and polygons, and fill patterns such as a drawing program might use. The OS then pixilizes or rasterizes the image to the scale requested. Vectors must still exist at some level in the scaling software (deep down) to describe the orientation of the primative line elements, polygons, and their fill patterns / colors.
The OS X icons are bitmaps. Each icon (.icns) file can have several sizes, ranging from 16x16 pixels up to 128x128 pixels. Leopard introduces 512x512 pixel icons. These are essential, especially at the smaller (16x16 and 32x32) sizes. Icons need A LOT of tweaking to look good when that tiny.

Here's a screen shot of Icon Factory's Icon Builder with an icon at various sizes: http://iconfactory.com/graphics/software/iconbuilder/screen1.jpg

You can see the resources that are being built on the left.

[JFreak]

Was actually just thinking of this today, because my 15 inch screen is soo small. What would happen when I upgrade to the next generation of 50" screens?

You would go crazy, because you have now used to use less than a megapixel :D

No, seriously, I have a 100" screen (HDTV projector) and I surely can appreciate the size because my eyes have been lasered. For most tasks, the 1280x720 screen size is good enough, but I would surely appreciate higher pixel counts when I do a Protools mix. Currently I get few dozen visible tracks, but it wouldn't hurt to see more. I like the direction things are going...

Everything new just costs a lot of money :(

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence?

Yes, a rough example. That's how it goes if you scale a bitmap that is "reasonably big" to begin with. But if the UI elements are vectorized, the end result is even smoother. You know, a vector is kind of an instruction set "what happens when you go from A to B" so assuming all things are vectorized, things look smooth from screen to photo print and a really big advertisement...

The problem is, not all things will be vectorized at once. I bet when Apple releases Leopard, the most common things should be vectorized, but there would be a lot of bitmaps still being used with 3rd party applications and less common items. It's a good start, but don't expect everything to be perfect to begin with ;)

My big concern is that you're going to have a situation where such a theoretical web site is going to look totally awesome on a Mac, but like complete ***t in Windows because the web site's default appearance (and how a Win32 environment web browser app displays content) just aren't as high a quality as what an Aqua (or whatever)-environment-based app would be.

That's always going to be the situation, unless the CSS or other standard begins to specify the display settings (gamma and such). To begin with, Mac and PC gamma is so different that web sites designed with Mac look different when viewed at PC and vice versa. Granted, it's not "complete #%*t", but it can be a disaster.

I want to see web sites including display calibration information!

Think regular TV on an HDTV television. You've got more than twice the res, but guess what? Regular TV looks worse on an HDTV than it does on a regular TV.

It's all a question about "upsampling". There are *HUGE* differences in upsampling algorithms, so some SDTV->HDTV conversions look like crap while others look smooth. The same goes to content scaling in the web pages; it needs to be done, and to be done with good quality.

The analogy is good, but it lacks in the respect of cost management; if a brand "A" television costs a hundred and a brand "B" television costs a thousand, wouldn't you thing the brand "B" would have allocated some time to think about the upsampling issue? I bet they would have, thus justifying the price difference. Brand "A" could have just assumed things are soon going to be in HDTV anyway... But the same thing does not apply when talking about the internet. Content is being made in some resolution and likely viewed in another, so there has to be a unified way to scale things. This is missing, and it has to be done soon. Somebody (Apple) has to lead and after some time (few years) general public can enjoy the results.

Hopefully :)

Any designer worth his/her salt will have to learn to work at the resolution of the device. It's not terribly difficult: print designers have been doing this for decades. Unfortunately, web browsers are only beginning to allow the necessary controls.

There have been print designers for +100 years, but the graphical web itself is only a teenager. There must be a difference! Sure, prints have become better in a hundred years, but so will web pages! We just have to give web designers a time to mature.

Today's web site is like a newspaper in 1900, if the analogy stands.

[R303blue]

As a web designer myself at a big company where I create tons of graphics daily and need to be fully optimized, don't you think this is going to cause nightmares with file size and optimization? Will there still be a *standard* res to work in that will be a good start across the board? I used to do a lot of print work, and my biggest limitation IMO was the size of files and having a computer powerful enough to work in those sizes. With the web I can practically be on a G3 these days and still have room to spare, but with higher res everything, not only will layered photoshop files reach massive sizes, computers will be crunching a lot more rendering new equipment obsolete quicker, and storage space needed will go up exponentially in time. I think this causes designers to just end up spending a lot more money (as if we don't have to already with soring costs of the products we use) in the long run. I better see a hefty raise from all this ;-) Maybe I'm just not getting it, and usually I embrace great new technology. I guess we'll see what happens.
:rolleyes:

[Highland]

Yes, a rough example. That's how it goes if you scale a bitmap that is "reasonably big" to begin with. But if the UI elements are vectorized, the end result is even smoother. You know, a vector is kind of an instruction set "what happens when you go from A to B" so assuming all things are vectorized, things look smooth from screen to photo print and a really big advertisement...

The problem is, not all things will be vectorized at once. I bet when Apple releases Leopard, the most common things should be vectorized, but there would be a lot of bitmaps still being used with 3rd party applications and less common items. It's a good start, but don't expect everything to be perfect to begin with ;)
I haven't found that to be true. If you have a large bitmap, and scale it down to whatever size is required, it looks "about as good" as a vector that's been created at size.

Not everything can be done with vectors as complex textures etc are very difficult. Complex vectors (like what would be required to do OS X style UI elements and icons) are usually much slower to render (you could cache as bitmaps, but what's the point?).

Some things DO work as vectors though, but certainly not everything. Don't expect to see Apple change everything over.

As a web designer myself at a big company where I create tons of graphics daily and need to be fully optimized, don't you think this is going to cause nightmares with file size and optimization? Will there still be a *standard* res to work in that will be a good start across the board?
If what's publicly available is anything to go by, then there will still always be a 1:1 size, but the user can scale the interface up or down.

Like I said, web zooming isn't really the same as UI res independence. Similar, but not the same. Also, we're not talking about print ready sizes... only about 3x what we use right now. Ie. 200ish DPI, not 600 DPI!

[shamino]

Your point is moot because 99% of stuff worth looking at is based on pixels on the internet, NOT vectors.
You don't need to vectorize everything to take advantage of resolution independence (although that also works.)

Bitmap-based interfaces can also be resolution independent if the bitmaps contain images rendered at a high resolution, and contain DPI information so the system will know how much to scale it for different displays.
And those 15 years of web pages aren't going to change to vectors any time soon. Gifs, photos, quicktimes, windows media, real players, etc. are all a static pixel size.
And all of these have DPI parameters as a part of their info. If the content creators set the DPI values correctly, very little will need to be changed. Cameras, scanners, rendering tools, and Photoshop all support this. Images designed for today's web browsers should already be set for 72 dpi (although there is a lot of badly produced stuff already in circulation, of course.)
We're talking logos, product artwork, etc. It all looks good displayed pixel for pixel. Double the res and scale up the interface and the interface will look good, but those pictures will have to be 1/2 the size or be scaled up, forcing them to be interpolated, forcing the to look soft at the same physical size they used to.
Assuming that the images are displayed on a 1:1 scale factor. The whole point of putting DPI in your image files is that this isn't necessarily going to be the case. If I have a 1" square image, and I make it 72dpi (72x72), of course, it will get blocky when rendered at 150dpi. But I can also make my image at 300dpi (300x300), and it will look good on any display.

There's no need to replace images with vectorized content to get good results on a resolution independent UI.

[shamino]

As for images, people would have to either start moving to SVG or they better specify real sizes for their images in the source. (unless image formats already have support for real sizes?)
They do. Load your favorite image into Preview and do a "get info" on it. You will find an "Image DPI" setting. Most good editors allow you to view/modify this. (I have personally used this on Graphic Converter and Photoshop Elements.)
The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps?
OS X's icons are a collection of bitmaps. Theoretically, the file may contain hundreds at all kinds of resolutions. In actual practice, most apps use one one - (typically 128x128 for OS X apps, or the legacy 32x32 size for Classic apps) and the Dock scales it up or down as necessary.
The OS X icons are bitmaps. Each icon (.icns) file can have several sizes, ranging from 16x16 pixels up to 128x128 pixels. Leopard introduces 512x512 pixel icons. These are essential, especially at the smaller (16x16 and 32x32) sizes. Icons need A LOT of tweaking to look good when that tiny.
The icon file format supports multiple sizes, but the Dock doesn't use them. I ran tests on this by creating an icon with some distinctive differences in all the different sizes. When applied to something in the Dock, they all get ignored. The Dock uses the largest size image in the icon and scales it to produce all the other sizes, ignoring the smaller images in the file.

Is this a bug? Maybe, but it looks fine anyway.
It's all a question about "upsampling". There are *HUGE* differences in upsampling algorithms, so some SDTV->HDTV conversions look like crap while others look smooth. ...
The analogy is good, but it lacks in the respect of cost management; if a brand "A" television costs a hundred and a brand "B" television costs a thousand, wouldn't you thing the brand "B" would have allocated some time to think about the upsampling issue? I bet they would have, thus justifying the price difference.
You'd like to think so, but experience has shown that inexpensive devices often look just as good as (or sometimes better than) more expensive models from other manufacturers.

Not always, but often enough that you should never assume high quality simply because of a high price tag. But this is really off-topic here...

[dusanv]

They can't patent this!

Vectorized interfaces have been around a while, and Windows 95 had the same exact implementation of UI scaling that they're talking about now (without the bitmapping). WTF Apple?

Yes, it even had it with the slider that you'd move to scale the DPI of the GUI up. IIRC, I remembered that and suggested it as a simple way to make aqua scalable (yay!)

Still, they can't patent that! Prior art omg.

I agree, it's ridiculous. Gee, who would have thought? Vectorized graphics!

This is one of the things that totally turns me off when it comes to Apple. They frequently patent obvious and prior art stuff and what's worse, pay for totally frivolous patents (Amazon's 1-Click, the Creative menu patent, ...).

[Highland]

The OS X icons are bitmaps. Each icon (.icns) file can have several sizes, ranging from 16x16 pixels up to 128x128 pixels. Leopard introduces 512x512 pixel icons. These are essential, especially at the smaller (16x16 and 32x32) sizes. Icons need A LOT of tweaking to look good when that tiny.
The icon file format supports multiple sizes, but the Dock doesn't use them. I ran tests on this by creating an icon with some distinctive differences in all the different sizes. When applied to something in the Dock, they all get ignored. The Dock uses the largest size image in the icon and scales it to produce all the other sizes, ignoring the smaller images in the file.
You're right. The dock is a special case though... it needs to be able to scale smoothly from large to small. I assume that's why Apple chose to only use the 128x128 resource.

In actual practice, most apps use one one - (typically 128x128 for OS X apps, or the legacy 32x32 size for Classic apps) and the Dock scales it up or down as necessary.
While I completely agree with most of what you've said, I feel I have to correct this point :) A lot of icons have 16x16 and 32x32 resources, and those are used everywhere in the OS. The Finder and open/save dialogue boxes are common places for them.

I agree, it's ridiculous. Gee, who would have thought? Vectorized graphics!

This is one of the things that totally turns me off when it comes to Apple. They frequently patent obvious and prior art stuff and what's worse, pay for totally frivolous patents (Amazon's 1-Click, the Creative menu patent, ...).
I think that's just the industry in general. If Apple didn't patent this, someone else would and Apple would have to license it. Amazon's 1 click and Creative's menu aren't Apple's fault!

It sucks and is completely out of control.

A patent on browsing music organised by Artist/Album/Song? F$#K OFF!

[Manuel Moreno]

seems that apple is going to introduce if not a new, a more polish interface. the screenshots associated to the patent reveal a graphical interface maker with a lot of parameters for each control, including the photoshop layers overlay modes for any material...

[dusanv]

I think that's just the industry in general. If Apple didn't patent this, someone else would and Apple would have to license it. Amazon's 1 click and Creative's menu aren't Apple's fault!

It sucks and is completely out of control.

A patent on browsing music organised by Artist/Album/Song? F$#K OFF!
You're right, the whole thing is out of hand. Everyone is patenting obvious stuff. Hopefully it's just a defensive patent on Apple's part.

[gnasher729]

But whatever the dimensions, I hope there will always be a way to override the settings. I happen to like high resolutions (I use a 20" monitor at 1600x1200 at work - 100dpi. Most people I know prefer lower DPIs on their screens.)

To be more precise, I prefer text and other things displayed a bit larger, and at the moment I can only get that through lower DPIs.

Then of course there are people displaying on a TV, and there you definitely want things displayed much much larger because you watch it from three or four meters distance! I just checked, just for fun, my TV displays about ten lines of text in exactly the same height as my MacBook screen.

So what the software has to take into account is several factors: Screen size in hardware pixels, screen size in centimeters, text size preference of the user, viewing distance.

[gnasher729]

The OSX dock icons appear to scale up and down smoothly. Is that an example of limited resolution independence? I bet they are not rendered as a series of carefully drawn bit maps? More likely they use lines and polygons, and fill patterns such as a drawing program might use.

You lost.

Every icon is turned into a texture, and then just drawn into a square of arbitrary size. Since this is done by your 3D graphics card, the OS could just as easily rotate the icons by any angle, transform them, and so on, at practically no extra CPU time and GPU time at all.

[OdduWon]

You lost.

Every icon is turned into a texture, and then just drawn into a square of arbitrary size. Since this is done by your 3D graphics card, the OS could just as easily rotate the icons by any angle, transform them, and so on, at practically no extra CPU time and GPU time at all.

Yeah, a Icon in OsX is just a square you can paste an image into. or it think there is a way to make paths in PS and use the icon application in the Developer kit to create transparentness in the remaining square. But i haven't got it to work yet :o

[ajprice]

I understand that resolution independence can make the interface graphics bigger, so it would emulate, say, 1024x768 on a 1280x960 monitor and have smooth graphics and fonts. Would it work the other way around, making the interface graphics smaller and emulating a bigger resolution than the monitor (ie get the interface the scale of 1280x960, but on a 1024x768 monitor??

[AppliedVisual]

I understand that resolution independence can make the interface graphics bigger, so it would emulate, say, 1024x768 on a 1280x960 monitor and have smooth graphics and fonts. Would it work the other way around, making the interface graphics smaller and emulating a bigger resolution than the monitor (ie get the interface the scale of 1280x960, but on a 1024x768 monitor??

It can work that way, but not well. The problem with trying scaling down is that you only have so much physical resolution to work with and you begin to compromise visual quality.

Where resolution independence will really shine is with upcoming displays that will have much higher pixel densities. True resolution independence will allow for individual applications and display elements to be scaled independently. And we'll be able to view web pages, documents, video windows, etc.. based on factors of intended resolution and specified size rather than the current less defined approach.

In addition to my animation and video work, I also install display systems (kiosks, AV installs in places like sports bars, public venues, government buildings, etc..). I often run into situations where certain displays need to be configured for the elderly or persons with impaired vision. True resolution independence will make it easier to deliver larger displays such as the 30" or even HDTV displays such as the newer 46" 1080p panels and have them show lower resolutions with better scaling. I have found the Dell and Apple 30" displays to already be invaluable in this regard. A 30" display locked at 1280x800 gives perfect 4x enlargement and works great for elderly computer users. Now if we could take that same principal to the next level and deliver 1280x800 on a 30" display, but actually make use of all the pixels instead of just turning 4 pixels into one, with proper smooth scaling of text and other items. That would be great and make it even easier for the audience to read what's on the screen.