Apple Multi-Touch Display?

[MacRumors]

http://www.macrumors.com/images/macrumorsthreadlogo.gif

As a final note in the recent flurry of Apple Tablet and Gesture patents, one user points out some interesting possible relationships between Apple's recent patents.

One video has received a lot of attention. It is a video by Jeff Han on his research into Multi-Touch displays. The impressive display shows a user using multiple fingers/hands in manipulating objects on the screen. A closer look at technology behind the demonstration reveals that it is a very intricate setup.

The system shown uses a technique called Frustrated Total Internal Reflection (FTIR) to accomplish the tasks shown. It involves the use of a transparent screen with the images rear-projected onto it. For touch-sensing, a camera is placed behind the screen to detect the user interaction. A diagram of the setup is shown here.

The use of both a projector and rear-camera, of course, is not feasible in any potential laptop or tablet device from Apple... but conveniently enough, Apple has recently applied for a patent on an Integrated Sensing Display (diagram) providing a screen that can be used both as a display and as a camera:

The integrated sensing display includes both display elements and image sensing elements. As a result, the integrated sensing device can not only output images but also input images.

At the time, many users dismissed the need for such an elaborate display when the built-in iSights offer a reasonable solution for video-conferencing needs. Of course, the use of this "integrated sensing" display would not likely use the FTIR method as described above, but could offer similar multi-touch functionality.

[arn]

sorry for the purely speculative article, but I found it too interesting not to post.

arn

[BornAgainMac]

Toy Sight comes to mind.

[narco]

I'll take one. Lets hope there is an interactive iTunes visualizer, because that's what that video reminded me of. It's also only a matter of time before someone uses parts other than their fingers to operate this device.

Fishes,
narco.

[Chef Medeski]

Hopefully this will provide, finally, for a true innovation in computer tech. I think for about 4-5 years other than open source, there has not been any leaps in technology. I would love to see such a thing and its implications. Just a touch screen would be nice, but something like this would be great. Well, of course if the software holds up to the hardware.

[themacman]

to put it in the words of neo, "woah"

[obeygiant]

and this point of this article is......

[EricNau]

It would be very cool if Apple utilized some of the technologies mentioned in the article. It might also give Apple a head start against the others.

[VisualForces]

A screen that can input images?

Hello "1984"

[quigleybc]

Good god that video is nuts....

[noaccess]

That is just... amazing *drools*

Now that is something the 30th Anniversary Mac should really have

[Montserrat]

Damn that video is cool - makes it look like such an intuitive way to interact with a computer. I'm on holiday in Bangkok and all the malls have got these 3D displays up - couple this multi-input touch screen technology with a 3D screen and I'll buy them all... OK - bit carried away there, but it'd be cool, no?

[Laser47]

That video is amazing. Too much for words.

[Michael Edwards]

Originally Posted by narco

It's also only a matter of time before someone uses parts other than their fingers to operate this device.

Fishes,
narco.

Hahahahaha.

That's sick.

[nagromme]

All these tablet and multitouch hints and patents may never turn into a shipping product, but they have me drooling

What I'm wondering, though, is why not just use conventional touchscreen technology like a PDA uses? Is there something that makes it physically impossible to sense multiple points at once with that kind of technology? Integrating a camera into a display just for touch-sensing sounds a little overkill.

[Felldownthewell]

Originally Posted by VisualForces

A screen that can input images?

Hello "1984"

Exactly what I thought. Opens up a whole new world of conspiracy theories.

[rikers_mailbox]

Originally Posted by obeygiant

and this point of this article is......

that's left to your imagination

The 'Integrated Sensing Display' sounds very interesting and is an elegant simplification of the FITR setup. I hope when I'm interacting with my LCD/touch screen, I don't get the annoying distortions and discolorations that I do now.

[DTphonehome]

Originally Posted by arn

sorry for the purely speculative article, but I found it too interesting not to post.

arn

VERY speculative. This is an idea still in the lab. It's developed by NYU Media Labs, and is very far from working its way into a production product. One day, sure, but it's certainly not going to be in a video iPod or Mac Tablet anytime soon.

--DT

[snowmen]

Well~ I think I have to buy a glove to use the Multi-Touch Display...

[iMeowbot]

Already mentioned in another thread after it was long dead, but a display with integrated sensing is something they've been wanting at Apple for a very long time.

[rikers_mailbox]

Originally Posted by iMeowbot

Already mentioned in another thread after it was long dead, but a display with integrated sensing is something they've been wanting at Apple for a very long time.

Woah. Impressive to say the least.

Why have I never seen that before? And what is it from?

[DTphonehome]

Originally Posted by nagromme

What I'm wondering, though, is why not just use conventional touchscreen technology like a PDA uses? Is there something that makes it physically impossible to sense multiple points at once with that kind of technology? Integrating a camera into a display just for touch-sensing sounds a little overkill.

It's actually pretty clever. A typical multitouch display is very complex...what this one does is sense the disruption in internal reflectivity of the uppermost layer of the screen. The "camera" is not exactly the sort which you'd use as a video or still camera...it's more like the "camera" in your optical mouse. It senses when a new point is touched and detects the track of that point by the light being scattered as it moves. It can do that for many points because they are all scattering light toward the camera and they all have discrete, distinct paths. The camera can see the entire screen at once, so it sees all the tracks simultaneously. Even if they cross paths, the camera sees the intersection and is able to logically determine which track belongs to which point.

The limitations of a classic touchpanel is that it is essentially a grid. When you depress a portion of the grid, the computer sees that as a point based on the x and y coordinates. Sort of like in Battleship. Now, you can move your point in two dimensions, and the digitizer will be able to quickly report the new coordinates to create a line. The only limitation to that single line is how fast the digitizer can report back. By its very nature, the grid can't support another point. For example, to use a Battleship analogy, if you call out B6, then column B and row 6 are occupied, and another "hit" on either row/column will not register. So you have to wait until B6 is clear before you can register another hit on either column B or row 6.

Sorry I'm not being clearer, it sounds simpler in my head.

--DT

[Dr. Dastardly]

In the near future we will all have to deal with some sort of Minority Report.

[notjustjay]

"Minority Report" here we come!

[Doctor Q]

Take a good luck, because eventually we will all be doing this.

iSights aren't designed to watch people, not fingers, so they aren't the right technology to support finger-based screen actions. I think touch sensitive screens will become the practical choice over cameras.

I ordered a Toysight the day it was announced (gotta be first on my block you know), but I wasn't especially impressed with it. But we'll get there.