Apple Integrated Sensing Display?

[ejgisbert]

Of interest, althought a little bit off topic for the current thread, is that Robert X. Cringely echoes in his weekly column (http://www.pbs.org/cringely/) the rumors about devices left out of the MacWorld Keynote. He also makes references to the Plasma sets (The ones reported at O'Gradys Power Page).

I'm an avid reader of Bob's column, and I do believe he has some friends in good places around the tech industry. But I'm not sure what to make of his statements.

 

[odedia]

The future

I'm actually thinking about more future uses for this type of technology. If and when the quality of such a product permits, imagine TV presenters being able to read directly from the camera itself. No need for Jay Leno to have any more of these huge Bristol papers next to the camera. He just stares at the camera and read what's "written" on it. I know we have telepromters these days, but they are somewhat limited compared to what this technology can do. Imagine the TV anchor man talking to a reporter on field, but not having to play "make belief" by looking at the camera while the screen with the action is actually to the left of the camera. The anchor man can also have tons if text info next to the live reporter's image, so that he can include it in his show.

This could be really usuable.

Oded S.

 

[shamino]

Very interesting indeed!

I like this idea. The problem with traditional cameras, as has been mentioned, is that you look at the screen, not the camera. This isn't too bad when you've got a small screen with an integrated camera (like the MacBook and similar style cameras from other vendors), but it's a huge problem when you've got a big screen.

An iSight mounted on a 30" display, or on my 22" CRT, would be terrible. It would be aiming at the top of my head instead of my face.

Putting the camera "behind" the screen solves that problem very nicely.

More interesting, however, is the DSP work needed to make this work. If you've got an array of sensing elements among the display pixels, there clearly won't be a lens to focus your image on those pixels. Which means each sensing element will return a blur. It is possible to combine all the blurs from all the elements in order to resolve the image, but this requires quite a lot of processing horsepower.

In many respects, you're creating a "compound eye" like that of insects. I've read many articles in science magazines about research in this direction. I will be very interested if it turns out that the tech is now commercially viable and Apple is going to be among the first to ship it to consumers.

 

[FoxyKaye]

1984's Viewscreen is finally here...

Make it a little bigger and touch-sensitive, and we'll be all Star Trek: The Next Generation.

 

[nagromme]

Sounds like another Apple patent (there are many) with no actual product expected, but you never know.

It makes no sense to me as a camera unless you want bee-vision A camera has ONE lens, this has hundreds. That suggests to me that it's for sensing things that are in CONTACT with the screen, not in front of it.

So it could allow a scanner in a screen (why? what's the mass appeal of that?) or some fancy kind of optical stylus recognition perhaps?

 

[Bad Beaver]

Uhm, people in *this place* should know the intended application from the Knowledge Navigator "Parkbench" video - or did sbd. mention that already?

The Apple Collection has it, for example (don't go kill their bandwidth, Google will take you to many places having it:
http://www.google.com/search?client...e+Navigator+Video+Parkbench&ie=UTF-8&oe=UTF-8 )

 

[Jetson]

The TV That Watches You

This sounds like an interesting invention. Technology continues to race forward, beyond the ability of most people's awareness.

I wished that Apple would make a clear statement and take a clear stand on privacy rights. In this day of increasingly faster, always on internet connections, we the user must be able to insure that microphones and cameras can be turned off as we wish.

Malicious websites can easily control aspects of our Mac's operation without our knowledge or control. And Apple has done very little to provide us users with a suite of privacy protection products.

Not everyone wants a videophone experience. Bell Labs has been trying to sell videophones since 1964, and found that people don't want their pictures to be seen on bad hair days, etc.

Please Apple, consider the privacy and security implications of always-on microphones and cameras attached to the internet.

 

[nagromme]

Please Apple, consider the privacy and security implications of always-on microphones and cameras attached to the internet.

* Apple has never done or said anything to do with "always-on" cameras.

* All iSight cameras have a visible indicator to warn you when they are on.

 

[nothing]

Prediction

I saw about six years ago a device designed and built at Carnegie Mellon University in pittsburgh this was a round palm sized device that could be used as a video / stills camera and then with the flip of a switch it could then project the image through the same screen. Thus enabling a small palm based device PDA etc to have a much larger screen to work with. Imagine being able to film or take photos then show your mates the footage on the nearest wall or flat surface.

 

[Marvy]

Not quite 3d

...
And... if the software tracks your eyes, and the person you are talking to has an "iVid" too, you will be able to move your head and look behind them... or around the room on the other side, or whatever. It will be like those holograms... but in full HD color!!!

It would not look like a hologram, but probably look just..., uh weird. To provide a feeling of depth you would need a stereo-display that displays different images for each eye. I remember Apple investing in these a while ago, and I believe companies like Sony have even developed laptops with 3d-Displays, but it seems they haven't taken off yet.

 

[globalhemp]

Picture capture for iPod with PhotoBooth

Sounds to me like a picture phone where the screen is the "camera." Of course, this could also be placed into a monitor, but [duh] Apple has already done that with the iMac and new MacBook.

So, its most likely going to be used in something much smaller, such as [drumbroll] an iPod! I wouldn't see Apple using this for upcoming revamped Cinema Displays, or rumored HDTVs.

Rather, if Apple plans on integratating picture capture capabilities -- into the screen -- with iPod, they will most likely go one step further -- add PhotoBooth for on-the-go consumer crazy effects.

Eric Pollitt, President

 

[shamino]

It makes no sense to me as a camera unless you want bee-vision A camera has ONE lens, this has hundreds.

Not as useless as you might think. With sufficient digital processing and some creative optics, you can get a very clear image from such a device, and no "bee-vision".

Check out this Science News article from March 2003 on the subject. This excerpt is of particular interest to this thread:

In Japan, for instance, Jun Tanida of Osaka University and his colleagues have been experimenting with arrays of tiny conventional lenses, known as lenslets. Each lenslet focuses a small, low-resolution image onto a portion of an electronic detector behind the array. By taking advantage of all of the lenslets' different perspectives, a computer can then calculate a single large scene at roughly twice the resolution than would be possible if one conventional lens had been used.

A particular advantage of the Japanese approach is that the thin lenslet array can focus light onto a detector less than a paper's thickness away. In collaboration with Minolta, Tanida and his colleagues have exploited this radical abbreviation of focal length to develop a prototype of a credit-card-thin camera. Normal camera focal lengths range around a few centimeters.

That was almost three years ago. If Minolta had a prototype then, it would not surprise me in the least if companies are now investigating the possibility of integrating lenslet arrays into LCD panels.

 

[macintel4me]

Sounds to me like a picture phone where the screen is the "camera." Of course, this could also be placed into a monitor, but [duh] Apple has already done that with the iMac and new MacBook.

So, its most likely going to be used in something much smaller, such as [drumbroll] an iPod! I wouldn't see Apple using this for upcoming revamped Cinema Displays, or rumored HDTVs.

Rather, if Apple plans on integratating picture capture capabilities -- into the screen -- with iPod, they will most likely go one step further -- add PhotoBooth for on-the-go consumer crazy effects.

Eric Pollitt, President
Global Hemp, Inc.
Global Hemp Store

I 100%, totally, fully agree!! iPhotoBooth here we come!!

 

[mogo]

New Front Row....????

This seems to be an image of the new front row. Check it out! The movies icon is now like the one used in hollywood instead of the one from iMovie.
I can't wait to get the new moovie store! C'mon apple hurry up.
Ok, about the screen...now you could also use your mac as a mirror...
btw, this is the link from the apple store http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore?family=iMac

 

[Yebot]

How 'bout this.

A tablet-esque Mac sits on your lap. Mail bounces to your attention with a new email. You hit the dock with your finger to switch to mail. Of course the dock is minimized but just when the little sensor-pixels at the bottom of the screen can 'see' your finger a half-inch away, the dock pops up. You read the new message and hit reply. After the new message window pops open, an image of a transparent qwerty keyboard materializes (via a killer coreimage animation of course); hovering over your screen. You drag the keyboard image to an optimal position. The screen senses your typing.

Send.


Apple is a GUI leader. This is a neXt step.

 

[nagromme]

That was almost three years ago. If Minolta had a prototype then, it would not surprise me in the least if companies are now investigating the possibility of integrating lenslet arrays into LCD panels.

Interesting. I suppose for lots of little "cameras" to assemble one big image in a clean way, they'd all have to be converged on a single focal distance though? Sounds like it would lack flexibility, but still intriguing!

 

[jeffy.dee-lux]

cloaking!

okay i tried to scan through this thread real quick, cause i would think that somebody woulda mentioned this already, but i didn't see it.

I think that apple is now gonna get crazy funding from the US military cause it seems like this new technology could lend itself well to a predator-style cloaking suit! Just make a suit covered with these pixel-cameras, each one displaying what the camera on the other side is seeing. Of course you could have crazy geometric arrangements of pixels to get the different angles and stuff like that to work. If anything, you'd at least have some sweet adaptive camouflage. But i don't like war, so lets just use this tech for like, sweet games of capture the flag or something.

 

[blackstone]

How 'bout this.

A tablet-esque Mac sits on your lap. Mail bounces to your attention with a new email. You hit the dock with your finger to switch to mail. Of course the dock is minimized but just when the little sensor-pixels at the bottom of the screen can 'see' your finger a half-inch away, the dock pops up. You read the new message and hit reply. After the new message window pops open, an image of a transparent qwerty keyboard materializes (via a killer coreimage animation of course); hovering over your screen. You drag the keyboard image to an optimal position. The screen senses your typing.

Send.


Apple is a GUI leader. This is a neXt step.

Yeah, I think this could be more of an interface technology than a videoconferencing thing. I'm thinking along the lines of the gesture-based interface in Minority Report.... imagine Photoshopping with that kind of thing!

 

[frome]

This seems to be an image of the new front row. Check it out! The movies icon is now like the one used in hollywood instead of the one from iMovie.
I can't wait to get the new moovie store! C'mon apple hurry up.
Ok, about the screen...now you could also use your mac as a mirror...
btw, this is the link from the apple store http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore?family=iMac

I noticed that the icons don't have any reflections, could this mean that they might create a front row for all macs (without the reflections it would run faster on slower macs.)

 

[shamino]

Interesting. I suppose for lots of little "cameras" to assemble one big image in a clean way, they'd all have to be converged on a single focal distance though? Sounds like it would lack flexibility, but still intriguing!

A trivial implementation, where each records a segment of the overall image, to be combined mosaic-style, would require this.

But if they all aim in the same direction (perpendicular to the surface), you'll get lots of overlapping images, all capturing the object from different perspectives. The amount of DSP and software work to put this together would be massive, but it should work.

If you've ever used Photoshop to stitch images together into a panorama, you've done this on a small scale. While you'd never want to manually stitch together thousands of images, there's no problem with an automated procedure doing this, especially if all the images are taken by an array of sensors with precisely-known dimensions and spacings.

What's more, since there are different perspectives involved, it might even be possible to construct a 3D image of what it's aiming at. After all, two lenses (your eyes) spaced a few inches apart give you depth perception, so it would stand to reason that a few thousand lenses spread out over the area of a 12-30" LCD panel should be able to do the same.

As for focussing, I don't think it is a concern. As the aperture decreases, depth-of-field increases. This is why pinhole cameras can focus on just about anything. If the lenslets are small enough to squeeze between the pixels of an LCD panel, that should allow it to focus on just about anything.

Of course, what Apple is actually referring to is anybody's guess, but I don't see any tehcnological reason why they couldn't do what I'm describing.

 

[shamino]

I think that apple is now gonna get crazy funding from the US military cause it seems like this new technology could lend itself well to a predator-style cloaking suit! Just make a suit covered with these pixel-cameras, each one displaying what the camera on the other side is seeing. Of course you could have crazy geometric arrangements of pixels to get the different angles and stuff like that to work. If anything, you'd at least have some sweet adaptive camouflage.

All true. And research is already going on in that direction. Note the following article about the Boeing Bird of Prey. In particular:

Sources suggest they may include active camouflage systems to reduce visibility by using panels or coatings that change colour or luminosity. This could allow safe combat missions in daylight, rather than being restricted to night flying.

Of coruse, active camoflage is not the same as showing "what's on the other side". But that's really not possible without truly becoming transparent, since multiple viewers may see the object from multiple angles. A single point on the surface can only be one color.

But simply applying patterns to mimic the background (say, starscape or sky) is much easier and probably just as effective.

The big deal about some kind of flat imaging tech is that unmanned spy planes won't need to carry discrete cameras. The skin of the plane can be the imaging surface.

 

[bigandy]

in the words of peter griffin,

"Sweeet!"

 

[faintember]

How 'bout this.

A tablet-esque Mac sits on your lap. Mail bounces to your attention with a new email. You hit the dock with your finger to switch to mail. Of course the dock is minimized but just when the little sensor-pixels at the bottom of the screen can 'see' your finger a half-inch away, the dock pops up. You read the new message and hit reply. After the new message window pops open, an image of a transparent qwerty keyboard materializes (via a killer coreimage animation of course); hovering over your screen. You drag the keyboard image to an optimal position. The screen senses your typing.

Send.


Apple is a GUI leader. This is a neXt step.

This would make me drool. I can imagine using this as an interface for controlling realtime audio manipulation....mmmmmmmmmmmm (drools....wipes mouth then drools again)

 

[MrCrowbar]

what scares me (actually I'd be very happy if apple did such a thing), this kind of interface could really do work.

That would solve the problems of tablet-Computers: handwriting recognition. And really, even if handwriting recognition would work, you can't write as fast as you can type, right? There is already a device for cell phones out there that projects a keyboard on a flat surface via laser and "sees" what you're typing on that surface. Unfortunately, it doesn't work that great. But that Minority Report Interface would really be nice. Imagine: Punching towards the screen means Expose, pointing at the bottom of the screen brings up the doc, showing the middle finger closes tha application... okay, geeks would soon have serious arm aches.

I'd prefer an interface requiring less physical action. Like an eye tracking pointer or direct brain connection (welcome to the matrix).

 

[Jotham]

How I'd make it

But if they all aim in the same direction (perpendicular to the surface), you'll get lots of overlapping images, all capturing the object from different perspectives. The amount of DSP and software work to put this together would be massive, but it should work.

Close, but then you mention overlapping images and the need to stitch them back together. Alot of people are also mentioning lens, etc. I think the actual implementation would be alot simplier, especially considering how small it would need to be. Simply stick a single pixel sensor (CCD) at the end of a narrow tube so only light parallel to the tube (perpendicular to the surface as you said) can reach it. Now this doesn't record an image but just the color at that pixel co-ordinate - now do this for every pixel and you have a screen-resolution copy of what is infront of the screen -- no zoom ability and focus would be only good for close range (this think isn't going to do landscape photography), but no stitching, processing, etc to worry about either). Sounds ideal for video-conferencing.

Side view:
______________________________
______________________________<--------------------------------light

This little tube sensor would be packed in the LCD array along with the standard Red, Green and Blue sub-pixels which make a single pixel, thus creating a display that can capture as well as display.

PS. If this isn't how they do it, maybe I should get to the patent office quick - though this is kinda the obvious way to do it.