Augmented vs virtual reality

[Chuckeee]

I don’t have an answer. I’m just asking the question.

Most of the use cases and applications people seem to be discussing for the AVP are based on virtual reality, with the device provides a total synthetic image to the user. But the AVP is an augmented reality device. It provides an overlay of synthetic images on top of a real scene.

For example, Viewing movies is fine and good but for an AR device Wouldn’t the real world seen behind the movie image actually distract from the movie viewing experience?

Seems like the advantages and disadvantages of augmenting a real world scene Is not really part of the discussion.

Pokémon Go (which I actually detest) Is one of the few actual augmented reality applications I can think of.

So what are the augmented image (not virtual reality) Applications available for the AVP?

 

[CodeSpyder]

I don’t have an answer. I’m just asking the question.

Most of the use cases and applications people seem to be discussing for the AVP are based on virtual reality, with the device provides a total synthetic image to the user. But the AVP is an augmented reality device. It provides an overlay of synthetic images on top of a real scene.

For example, Viewing movies is fine and good but for an AR device Wouldn’t the real world seen behind the movie image actually distract from the movie viewing experience?

Seems like the advantages and disadvantages of augmenting a real world scene Is not really part of the discussion.

Pokémon Go (which I actually detest) Is one of the few actual augmented reality applications I can think of.

So what are the augmented image (not virtual reality) Applications available for the AVP?

One can set an environment to immerse one’s self in a virtual scene to watch movies.

 

[gerald.d]

For example, Viewing movies is fine and good but for an AR device Wouldn’t the real world seen behind the movie image actually distract from the movie viewing experience?

There is no need to have any of the "real world" visible behind any screen that is projected in an augmented reality. The "reality" is fed through from the external cameras, and displayed on the same physical displays that the movie screen is shown on.

In theory, it would be possible to set the virtual screen that the movie is being shown on to be semi-translucent, but I'm not sure why anyone would want to do that.

Bottom line, what you describe is not what the experience will be when watching a movie, even if you did choose to watch it in your own room environment.

 

[Chuckeee]

So the AVP provides no augmentation to the actual scene, it only supplies a totally virtual image to the operator? Seems a bit counterintuitive.

 

[gerald.d]

So the AVP provides no augmentation to the actual scene, it only supplies a totally virtual image to the operator? Seems a bit counterintuitive.

I guess it depends what you mean by "actual scene". It is not an optical pass through to the room itself. It uses multiple cameras to capture the room, and then displays the room on the screens with close to zero latency.

Comments from those who have tried out the AVP would seem to imply that it does this really, really, well.

 

[Chuckeee]

Some of the recently announced apps do address my concerns but let me try explaining it another way.

One of the unique AVP features is to a view a dynamic image as the user move around. The concerns I had is that much (but not all) of the AVP discussions and usages was based on using it while remaining stationary. This seem to ignore the new capabilities the AVP was trying to bring to market.

 

[gerald.d]

Some of the recently announced apps do address my concerns but let me try explaining it another way.

One of the unique AVP features is to a view a dynamic image as the user move around. The concerns I had is that much (but not all) of the AVP discussions and usages was based on using it while remaining stationary. This seem to ignore the new capabilities the AVP was trying to bring to market.

There will be plenty of use cases for the AVP that will be identical (except for a huge leap in visual quality) to those experienced in any other headsets. Where it looks to excel, is the way that you can "dial in" the amount of reality (i.e. your actual room environment) you want to experience together with the virtual content. So, you can sit in your chair and have a virtual 120 inch screen just feet from you and enjoy a movie in your own surroundings, or you can spin the digital crown and transport yourself to another location completely, and watch it there.

But as soon as you want to start walking around, trust me - you are going to want to be able to see your actual environment!

I've not sure if you've seen the demo video of the F1 car that basically just sits lifesize in front of you in your room. You can get out of your chair and just walk around the thing.

People are going to be absolutely blown away by experiences like this.

 

[Jensend]

At this point in time, I mostly see the video passthrough as more of a user comfort feature rather than something that fundamentally changes what kind of applications can/will be developed for the device.
It's convenient to be able to see your couch and coffee table, even if that isn't really incorporated into the virtual experience. You can fade the room view while watching a movie, and fade it back in if you need to interact with the real world in some way.

Developers have fairly limited access to the user's surroundings. They can access at least some basic geometry of the user's room, and can supply visual anchors to find in the room, but they don't have an kind of direct access to the cameras. iPhone and iPad developers actually can have access to the cameras.

 

[throAU]

So what are the augmented image (not virtual reality) Applications available for the AVP?

Given the device hasn't even shipped yet, very little.

But conceptually expect the following types of things (some of which already exist on HoloLens):

• remote assistance, where an expert can see what a remote novice is seeing and guide them
• overlaid product manuals/docs/3d models to assist with repairs
• viewing 3d models of furniture, etc. in your house pre-purchase
• games like Pokemon go
• ability to get product info by just looking at bar codes / QR codes
• ability to inventory products by look looking at bar codes, etc. and adding to a database

The big thing is having information available without needing to use one or both hands to manipulate a device.

Because it is on your head, your hands are free to ... work on real world things. How many times have you had to work on something and stop work, get up, go back to your device to keep referring back to a book/manual/iPad/etc.

That's wasted time you could have been working on the real world thing you're trying to fix. For certain activities, an AR device would be a massive productivity boost. Not all activities.

 

[throAU]

At this point in time, I mostly see the video passthrough as more of a user comfort feature rather than something that fundamentally changes what kind of applications can/will be developed for the device.
It's convenient to be able to see your couch and coffee table, even if that isn't really incorporated into the virtual experience. You can fade the room view while watching a movie, and fade it back in if you need to interact with the real world in some way.

being able to see/mix the real world is fundamental to AR usage.

If all you're not including the real world this is almost 100% an entertainment only device and has extremely limited real world productivity application.

A huge advantage of AR is being able to have information with you without needing to hold/carry/manipulate a device with your hands - which are then free to do other things.

 

[Jensend]

Given the device hasn't even shipped yet, very little.

But conceptually expect the following types of things (some of which already exist on HoloLens):

• remote assistance, where an expert can see what a remote novice is seeing and guide them
• overlaid product manuals/docs/3d models to assist with repairs
• viewing 3d models of furniture, etc. in your house pre-purchase
• games like Pokemon go
• ability to get product info by just looking at bar codes / QR codes
• ability to inventory products by look looking at bar codes, etc. and adding to a database

The big thing is having information available without needing to use one or both hands to manipulate a device.

Because it is on your head, your hands are free to ... work on real world things. How many times have you had to work on something and stop work, get up, go back to your device to keep referring back to a book/manual/iPad/etc.

That's wasted time you could have been working on the real world thing you're trying to fix. For certain activities, an AR device would be a massive productivity boost. Not all activities.

Floating windows that can be interacted with hands-free could be convenient, yes.
The current API doesn't give developers access to the cameras, so some of your suggestions may not be possible at this time.

 

[gerald.d]

<snip>
It's convenient to be able to see your couch and coffee table, even if that isn't really incorporated into the virtual experience. <snip>

Not entirely sure what you are getting at here. It's clear from the demos that the real world environment is incorporated into the virtual experience because virtual screens cast shadows in the real world. Developers absolutely can incorporate real world objects into the virtual experience, as demonstrated by the Lego game screenshots.

https://twitter.com/x/status/1749142833712279904

/edit
(I guess adults play with Lego these days? VR/AR etc are explicitly recommended only for those over the age of 13.)

 

[Jensend]

being able to see/mix the real world is fundamental to AR usage.

If all you're not including the real world this is almost 100% an entertainment only device and has extremely limited real world productivity application.

A huge advantage of AR is being able to have information with you without needing to hold/carry/manipulate a device with your hands - which are then free to do other things.

If your primary focus is on the real world, and the headset is just for floating instructions, you may be better off with see-through AR.

Not entirely sure what you are getting at here. It's clear from the demos that the real world environment is incorporated into the virtual experience because virtual screens cast shadows in the real world. Developers absolutely can incorporate real world objects into the virtual experience, as demonstrated by the Lego game screenshots.

https://twitter.com/x/status/1749142833712279904

/edit
(I guess adults play with Lego these days? VR/AR etc are explicitly recommended only for those over the age of 13.)

The location of the table has no importance to the game. If you moved the table, the game wouldn't move with the table. You could float the game in exactly the same place without the table and there would be no difference in how the game is played.
With a VR headset with no passthrough mode, the game could just have a configuration step where you touch the controller to the tabletop to set the location of the game. It's nice being able to see the table so you don't bang your legs against it.

What could move it beyond VR is if you had physical game tokens with AR codes on them, and you could move the physical tokens to also move or activate something in the digital game world.

 

[throAU]

If your primary focus is on the real world, and the headset is just for floating instructions, you may be better off with see-through AR.

Point me at the available see-through AR devices on the market, HoloLens (which I've used, we have at work) has been discontinued. It was also 5k USD.

The cameras on the Vision Pro can do both, which is better.

 

[gerald.d]

The location of the table has no importance to the game. If you moved the table, the game wouldn't move with the table.

Ahh ok, I understand what you were getting at. Although I'm not entirely sure that is correct.

The Vision Pro doesn't just have optical cameras, it has LIDAR as well, no?, and has a complete 3D mapping of the room that you are in. Would it not be possible for it to actually recognise the surface that the game was placed on, understand where it was in 3D space (it seems to do this with the dropped virtual screen shadows by the way), and move the game with it?

In fact it has to understand where that surface is in 3D space, because the game will stay affixed to it if you change your position relative to the table. If you move relative to the table, and the game stays affixed to the surface (which I can absolutely guarantee you that it will), then it doesn't matter whether it is you or the table that is moving. It's the exact same thing.

 

[Jensend]

Point me at the available see-through AR devices on the market, HoloLens (which I've used, we have at work) has been discontinued. It was also 5k USD.

The cameras on the Vision Pro can do both, which is better.

I can buy HoloLens 2 right now from the Microsoft website for $3,500.00 and have it by the end of the week. I'd personally rather have Vision Pro. It has a broader set of use cases. I just think if most of your use is hands free instructions/documentation while focusing on a real object, the HoloLens (2) may be a better choice.
But I haven't tried either.

 

[Jensend]

Ahh ok, I understand what you were getting at. Although I'm not entirely sure that is correct.

The Vision Pro doesn't just have optical cameras, it has LIDAR as well, no?, and has a complete 3D mapping of the room that you are in. Would it not be possible for it to actually recognise the surface that the game was placed on, understand where it was in 3D space (it seems to do this with the dropped virtual screen shadows by the way), and move the game with it?

In fact it has to understand where that surface is in 3D space, because the game will stay affixed to it if you change your position relative to the table. If you move relative to the table, and the game stays affixed to the surface (which I can absolutely guarantee you that it will), then it doesn't matter whether it is you or the table that is moving. It's the exact same thing.

The headset knows its own position relative to the entire room. If I had to bet, I'd guess that once you tell the Lego game where you want it to be, it becomes fixed in world space, not in the coordinate space of a single object. Coffee tables are largely stationary, so that will usually be the same thing. But if you move the coffee table, I don't think the game would move. Imagine if the coffee table had drawers and you pulled them open. Would the Vision Pro know the difference between opening a drawer and moving the table? I doubt it.

If anything, AR constrains the app design, because you have to limit the design of the 3D objects to fit in the user's physical space.

 

[gerald.d]

The headset knows its own position relative to the entire room. If I had to bet, I'd guess that once you tell the Lego game where you want it to be, it becomes fixed in world space, not in the coordinate space of a single object.

Well of course it could be programmed in exactly that way, but there is absolutely no reason why it would have to be restricted in that manner, which was the point I was making in response to your "The location of the table has no importance to the game. If you moved the table, the game wouldn't move with the table." comment.

Physical objects in rooms absolutely could have importance in apps and games (simple example - "show me how that new chair will look if it replaces this one"). You've just decided that in this instance, you would bet that the programmers wouldn't bother to account for the fact that things can move around.

 

[Jensend]

Well of course it could be programmed in exactly that way, but there is absolutely no reason why it would have to be restricted in that manner, which was the point I was making in response to your "The location of the table has no importance to the game. If you moved the table, the game wouldn't move with the table." comment.

I'm not necessarily saying it would have to be restricted in that way, but I believe it currently is. I don't think the Vision Pro has any kind of concept of objects moving, except for the user's hands and fiducial markers. If you move the table, the Vision Pro will know that the geometry of the scene has changed, but I don't think there is a continuity of understanding between the old layout and the new. What you could do is put a fiducial marker on the table, and track that as a proxy for the table.

Here's a confirmation I found after writing the above:
There is no API for object detection or object tracking on visionOS. The closest feature available is image tracking.

Physical objects in rooms absolutely could have importance in apps and games (simple example - "show me how that new chair will look if it replaces this one"). You've just decided that in this instance, you would bet that the programmers wouldn't bother to account for the fact that things can move around.

Yes, seeing how a piece of furniture would look in your real space is a good use for AR. But there isn't really a way to remove the real chair from the scene, or retexture it. Developers are very limited in what they know about the scene, even more so than AR developers for iOS. I don't believe you could, for example, implement a snapchat-style filter in a Vision Pro app.

 

[Jensend]

Here's an example of the image tracking feature.
Basically, the Disney app could ask the Vision Pro to look for that specific poster, and return its precise location if found. Then the digital Mickey Mouse animation could be aligned with the poster, and an image of the poster without Mickey could be overlayed on the original poster (though I doubt it would be as seamless as in the mockup). Mickey could interact with the geometry of the room, but in a fairly limited way.

 

[gerald.d]

I'm not necessarily saying it would have to be restricted in that way, but I believe it currently is. I don't think the Vision Pro has any kind of concept of objects moving, except for the user's hands and fiducial markers. If you move the table, the Vision Pro will know that the geometry of the scene has changed, but I don't think there is a continuity of understanding between the old layout and the new. What you could do is put a fiducial marker on the table, and track that as a proxy for the table.

Here's a confirmation I found after writing the above:
There is no API for object detection or object tracking on visionOS. The closest feature available is image tracking.


Yes, seeing how a piece of furniture would look in your real space is a good use for AR. But there isn't really a way to remove the real chair from the scene, or retexture it. Developers are very limited in what they know about the scene, even more so than AR developers for iOS. I don't believe you could, for example, implement a snapchat-style filter in a Vision Pro app.

Thanks for digging that up - rather disappointing!

So it seems like the limitations are on the API side then, and hopefully something that they can add in the future. Furniture swapping is something that has been pretty common in other VR use-cases (such as virtual tours) for quite some time now.

 

[klasma]

One can set an environment to immerse one’s self in a virtual scene to watch movies.

That's still VR, not AR.

 

[contacos]

I personally have never been a fan of AR (first thing anyone turned off on Pokemon Go for example) and I barely ever use it on my Quest 3 either. Obviously people have different tastes and that is also fine but the whole point for me to use one of these headsets would be to "escape" from my current location (e. g. pretend to be working from Hawaii) and not my living room.

I use YouTube VR all the time and I love how it makes you feel like you are sitting in a huge IMAX theatre. I don't just want a floating screen in my actual bed room.

 

[Unami]

There are tons of potential usages, from navigation to surgery to repair manuals that actually recognize and label parts. Imagine never again to have to keep track where a screw goes) Curiously, apple has shown none of this interactivity with the real world but only apps positioned in a room, stale MR demos (like the dinosaur), a VR enviroment for those apps and 2D/stereoscopic ”3D” media consumption. Not even VR-games so far, alhough some of them could be easy to port and so be the low hanging fruit. That last one is probably very deliberate - and I wonder if they’ll even allow those in VisionOS1 (anyway, without a dedicated controller, most are only half as good - afaik, even support for traditional controllers like the ps5 controller doesn’t extend to the features that would make them interesting vor VR - like rumble, adaptive triggers and motion sensors)

 

[Rychiar]

Given the device hasn't even shipped yet, very little.

But conceptually expect the following types of things (some of which already exist on HoloLens):

• remote assistance, where an expert can see what a remote novice is seeing and guide them
• overlaid product manuals/docs/3d models to assist with repairs
• viewing 3d models of furniture, etc. in your house pre-purchase
• games like Pokemon go
• ability to get product info by just looking at bar codes / QR codes
• ability to inventory products by look looking at bar codes, etc. and adding to a database

The big thing is having information available without needing to use one or both hands to manipulate a device.

Because it is on your head, your hands are free to ... work on real world things. How many times have you had to work on something and stop work, get up, go back to your device to keep referring back to a book/manual/iPad/etc.

That's wasted time you could have been working on the real world thing you're trying to fix. For certain activities, an AR device would be a massive productivity boost. Not all activities.

Yup even simple things. Like i currently use an iPad to display recipes when i cook but with this i could just have it in front of me and not have to worry about getting hands covered in ingredients all over a screen. And i could also message friends back, listen to music, see my outdoor cameras etc all at the same time