Lenticular displays and other tidbits from Mike Rockwell - WWDC Talk Show

[tomtad]

If you've not seen it yet I would give the Mike Rockwell interview on WWDC Talk Show with John Gruber a watch. It's very interesting to hear more about the development of Vision Pro and also just to hear the passion and detail that has gone into this product. The Rockwell interview is about 38 minutes in.

A few things stood out to me which I've not yet seen mentioned:

• Lenticular display for EyeSight - The EyeSight feature uses a lenticular display, and the first ever curved one, to display your eyes and face three dimensionally set back so it appears you are wearing the headset in front of them. This display technology employs vertical prism lenses to achieve the effect and then they render multiple different versions of your eyes at different angles so it looks convincing wherever you are viewed from. So if I'm viewing someone from the right I will see a different image to if I viewed someone front on etc. This sounds very cool.
• More EyeSight -: When you are immersed in a different environment a blurred Siri like effect appears on the external display. As we have seen if someone approaches you they break through into your environment in the headset but at the same time your eyes then appear on the external display so they can see you.
• Point controls - As well as eye tracking you can also just point and touch items as you would on an iPhone
• Personas are volumetric - Rockwell hinted that what we've seen from personas so far isn't everything. They will also be able to break out of flat windows and I assume be physically in your space in three dimensions as they are volumetric models. He said there was a brief glimpse of this at the end of a demo but I haven't seen this anywhere yet.
• Hand identifying - Your hands are identified and isolated so they appear on top of any content/interface you are seeing. There are also some circumstances where you can push through that content and your hands appear behind it.

 

[matrix07]

Lenticular display for EyeSight - The EyeSight feature uses a lenticular display, and the first ever curved one, to display your eyes and face three dimensionally set back so it appears you are wearing the headset in front of them.

This makes me wonder: will it work with many people around you at once? In the event Apple only show you with only one human being nearby.

 

[subjonas]

An interesting tidbit- Rockwell said EyeSight kicks in when someone looks at you. I think he said that’s also when they break into your view. That means if you want to scare someone wearing the headset, you just sneak up on them without making eye contact haha.

 

[subjonas]

This makes me wonder: will it work with many people around you at once? In the event Apple only show you with only one human being nearby.

I believe Rockwell said it does work with multiple people around you—that the display simultaneously shows multiple angles of your eyes so that each person around you sees the appropriate angle of your eyes. But I was lost on how that’s accomplished because it sounds impossible.

 

[matrix07]

But I was lost on how that’s accomplished because it sounds impossible.

it indeed sounds impossible.

 

[zakarhino]

MR writers or some forum user should compile all the info we currently have about the device into one place. Nobody's talking about all of the tidbits of information in the WWDC dev videos for visionOS. e.g, someone on the forum thought the device wouldn't be capable of producing 3D objects that interact with the real world, like someone dropping a virtual ball in space and watching it bounce off of your couch via Passthrough, but a dev video explicitly mentioned this kind of use case being built into RealityKit (I mean, to be fair, the iPhone could already do this).

 

[zakarhino]

I believe Rockwell said it does work with multiple people around you—that the display simultaneously shows multiple angles of your eyes so that each person around you sees the appropriate angle of your eyes. But I was lost on how that’s accomplished because it sounds impossible.

He said 'up to 7 people' which I think translates to seven different angles of your eyes being rendered at once rather than some kind of software dynamically tracking the people around you to show them a special image, at least that's how I think it works. Think about how a Nintendo 3DS works, the display uses different pixels to show your eyes two different images and that's how you get the 3D effect. It's not doing any sort of eye tracking and advanced rendering, it's simply showing two different images for two different angles that happen to align with each eyeball. I think he's saying the display will do that but instead of showing 2 different images per eye, it will show 7 different renders of your eyes depending on where you're standing.

Kind of a modification of this:

 

[subjonas]

MR writers or some forum user should compile all the info we currently have about the device into one place. Nobody's talking about all of the tidbits of information in the WWDC dev videos for visionOS. e.g, someone on the forum thought the device wouldn't be capable of producing 3D objects that interact with the real world, like someone dropping a virtual ball in space and watching it bounce off of your couch via Passthrough, but a dev video explicitly mentioned this kind of use case being built into RealityKit (I mean, to be fair, the iPhone could already do this).

I think there was a recent “everything we know so far about Vision Pro” MR article, but it is missing details like those mentioned here. I’m sure they’ll update it over time, hopefully.

 

[subjonas]

He said 'up to 7 people' which I think translates to seven different angles of your eyes being rendered at once rather than some kind of software dynamically tracking the people around you to show them a special image, at least that's how I think it works. Think about how a Nintendo 3DS works, the display uses different pixels to show your eyes two different images and that's how you get the 3D effect. It's not doing any sort of eye tracking and advanced rendering, it's simply showing two different images for two different angles that happen to align with each eyeball. I think he's saying the display will do that but instead of showing 2 different images per eye, it will show 7 different renders of your eyes depending on where you're standing.

Kind of a modification of this:

That’s insane. I barely understand this for two eyes, much less scaling up to 7 people. But it’s ok, I’ll just use it and not question how lol.

 

[zakarhino]

That’s insane. I barely understand this for two eyes, much less scaling up to 7 people. But it’s ok, I’ll just use it and not question how lol.

This is what lenticular art looks like. You might have seen it in person before. In this case the artist is using physical triangles to paint a different painting on each side of the triangle. Depending on where you stand you see a different image. This principle can be applied to displays using lenses instead.

A very crude example: Recreating the below image using a display would basically mean each vertical line of pixels in the display would be dedicated to showing one of the two images, alternating between each one on a line by line basis (look at the top 'head on' image in the below example). But remember displays are flat so if we moved our heads around we would only see the top image (the Vision Pro's front display being curved is an unimportant detail for the sake of this explanation). So what we do is place a separate long lens in front of every single line of pixels to direct the emitted light to either a left or right angle. First line of pixels directs light to the left. Second line directs to the right. Third line directs to the left. And so on and so on. Now if someone stands on the left all of the lines of pixels meant for someone on the left will be directed in that direction but the pixel lines meant for the right are not visible because no light is being directed in your direction. Scaling that up to seven angles is easy because instead of swapping between two images every two lines of pixels, you do that seven times instead with seven different lens angle tunings.

Basically lenses are being used to replicate the effect of a physical lenticular art piece like this lol sorry I'm bad at explaining this. There's probably more going on behind the scenes but there's the oversimplified explanation I guess. I don't think Vision Pro is sending two different images to each of the 7 persons' 2 eyeballs to simulate a 3D image, that would be overkill, it's just the below art but using a display with seven different image angles rather than 2 like below.

 

[Jensend]

He said 'up to 7 people' which I think translates to seven different angles of your eyes being rendered at once rather than some kind of software dynamically tracking the people around you to show them a special image, at least that's how I think it works. Think about how a Nintendo 3DS works, the display uses different pixels to show your eyes two different images and that's how you get the 3D effect. It's not doing any sort of eye tracking and advanced rendering, it's simply showing two different images for two different angles that happen to align with each eyeball. I think he's saying the display will do that but instead of showing 2 different images per eye, it will show 7 different renders of your eyes depending on where you're standing.

Kind of a modification of this:

Did he say seven? I just hear “we actually render separate views of your eyes for every person who is looking at you”

 

[zakarhino]

Did he say seven? I just hear “we actually render separate views of your eyes for every person who is looking at you”

In the keynote I believe he said up to seven people

 

[sunapple]

I keep being amazed by how complicated this thing really is. Yes, they called it the most advanced piece of consumer electronics on stage, but still it feels like they undersold it somehow.

 

[Slartibart]

I believe Rockwell said it does work with multiple people around you—that the display simultaneously shows multiple angles of your eyes so that each person around you sees the appropriate angle of your eyes. But I was lost on how that’s accomplished because it sounds impossible.

This is how lenticular lense sheets work. You might have seen postcards using them? Here is a video tutorial which explains how you can create one.

This is done similar - you have a lenticular lensing sheet with a certain lpi and you use the 3D scan to generate views at different angles (very simple: right lateral, frontal, left lateral) and "cut" these "in stripes" and place them alternating under the lenticular lensing.

 

[sevoneone]

In the keynote I believe he said up to seven people

I took that to mean that the headset can track up to 7 people in your surroundings and detect when they look/interact with you.

I am pretty sure the idea with the display is that a 3D model of your actual eyes is rendered and then somewhere between 5 and 8 angles of that model are sliced up on the screen behind a lenticular lens. An observer would then see the correct angle for where they are standing. It’s not quite as high tech as they want it to sound. Ever have a ruler when you were in grade school that had a 3D image or scene that looked like it moved? It’s the same concept.

In fact, 5 years ago, Insta 360 made a surprisingly good camera that shot 180-degree video and photos in 3D. It even came with a lenticular lens you could snap to the front of your iPhone as an option for watching your captured content: https://www.insta360.com/mobile/product/insta360-evo/ It was a bummer it failed when it did. I have a feeling lots of people are going to want a way to make 3D VR content soon and the only solutions you can get for VR 180 right now cost thousands of dollars.

 

[Unregistered 4U]

This is done similar - you have a lenticular lensing sheet with a certain lpi and you use the 3D scan to generate views at different angles (very simple: right lateral, frontal, left lateral) and "cut" these "in stripes" and place them alternating under the lenticular lensing.

Those postcards have quite powerful GPU’s as they have to render the image for everyone that sees the card going through the post office, from whatever angle they look at it from!

 

[alanvitek]

The lenticular tech is similar to this screen Delta has in some airports in which uses the same concept to show individual flight information to people that is “invisible” to others. I read an interview with the developer that said their screens can track up to 100(?) different guests and uses the lenticular display to show your flight information based on your current position. Essentially someone would have to put their head on your shoulder to get the right angle to see your flight info on the screen. Otherwise they would see theirs own flight info, or nothing on the screen if they didn’t scan their ticket.

Very cool tech

 

[Slartibart]

Those postcards have quite powerful GPU’s as they have to render the image for everyone that sees the card going through the post office, from whatever angle they look at it from!

it’s what to expect for Apple’s Vision SE model… allegedly 🤣😂