Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

DogDogDogFood

macrumors newbie
Original poster
Jun 28, 2022
4
6
The usual Apple rumor community has had nothing to say about Neural Radiance Fields (NeRFs), with the exception of Twitter where an increasing number of people seem to believe that NeRFs will be a key feature of the upcoming Apple headset and eventually the rest of the Apple ecosystem.

Most people don't even know what NeRFs are - there isn't even a wikipedia article for it yet! This is a new concept in AI and computer vision that uses machine learning to represent a scene without 3D models. It went from a resource-intensive research project at Nvidia in 2020, to an efficient near-realtime technology just within the last year. People are thinking up all sorts of use cases, but the most popular is view synthesis. This would be a step up from photogrammetry that you currently see in Object Capture - the technology can take a relatively small number of input images, and generate a compact ML model that encodes the scene. Unlike photogrammetry, it can accurately capture reflections, transparency, volumetric effects like fog or smoke, and even lens flare.

The speculation about NeRF features in the Apple headset partly stem from the unique capabilities of Apple Silicon, which have GPUs and a Neural Engine powerful enough to make NeRF features practical. We know that Apple is interested, since they have published several papers about Radiance Fields on their Machine Learning site. The potential here is huge - Users of a headset will be able to easily capture scenes or objects and share them with others, who can explore them in MR with photorealistic quality. Apple could stream live sports events or concerts in VR, where the user could position themselves wherever they want in 3D space, leveraging spacial audio to complete the experience. A new standard datatype to encapsulate this information would allow NeRFs to be integrated with headset apps from day one, and open up all sorts of applications that no one has even thought of yet.
 
So Nerfs will be the secret weapon for AR/VR success? Very nice.

Nerfmeme2.jpg
 
This is a new concept in AI and computer vision that uses machine learning to represent a scene without 3D models.
Given the focus on ML in the last years, this is something I can see. Thrilling what Apple Glass will truly be once it's officially introduced to the public.
 
Given the focus on ML in the last years, this is something I can see. Thrilling what Apple Glass will truly be once it's officially introduced to the public.
There is an opportunity for RealityOS to be a Machine Learning platform. With a headset you have visual, audio, and spacial inputs - those can all be enhanced with ML to provide a novel AR/MR experience.

It doesn't have to be all consumption, either. Creating and sharing NeRFs (or whatever Apple ends up calling them) is the simplest case. Apple also has their no/low-code systems like SwiftUI and Trinity AI. It should be possible to create AI tools right within Reality OS - either by training object recognition through Object Capture + NeRFs, or with audio or spacial data. Someone working in this system could create a ML-driven solution and then share it with colleagues and friends to help with them with various tasks.

Imagine being a researcher counting migratory birds in the field. You could take some recordings of your subject and train a model to recognize them. Then carry the Apple headset with you, and the system would alert you on your Apple Watch when it hears the audio you are interested in. You then put on the headset, and it uses the mic array to display on-screen indicators showing the direction of the sound. Using optical zoom, you could get a better look; maybe even drop in a pre-trained image enhancement model. You could then log the results using the data service on your iPhone and put the headset away.

There are tons of uses around the home for this, too.
- My kid makes a sculpture for a school project, and I use the headset to create a virtual model I can share with the grandparents. This can even use Apple's subject-isolation technology to remove the background.
- I want to watch a movie on Apple TV+ with some friends - I can use Share Play to create a shared viewing experience, while offloading the streaming and decoding to my Apple TV to save on power.
- We get done painting a room, and want to share what it looks like before and after.
- I want to do something on my Mac, but it's a different part of the house and I'm outside on the porch. I can bring up a virtual screen on the headset, and do desktop-style work anywhere.

If Apple does this right, the eternal question of "What is AR really useful for?" will quickly become "What is AR not useful for?". Apple is the only company in the world who has competence in every field needed to bring a platform like this together. They can do great hardware, and have custom silicon to power it. They have many other devices and services that can be tied into the headset (Mac, iPhone, AirPods, Apply TV, Apple Music, App Store, etc). They have all the pieces needed for a breakthrough Machine Learning environment. They have mature developer tools and APIs, and a developer community eager to adopt new form factors. They have a world class ecosystem not just for apps, but for sharing and communication. Neither Microsoft, Meta, Google, or NVIDIA are on this level.
 
  • Like
Reactions: Kierkegaarden
Interesting thoughts. From what you are saying, there seem to be many productivity applications for even the average consumer, but also for professionals or researchers. The comments I read the most are around not thinking past the entertainment or consumption applications of such a device. This is probably where Meta and others have put their efforts into. This could certainly be part of it, but that alone probably wouldn’t make it compelling for the vast majority of the public.

I wonder how such a device could transform live concerts and education. For education, what if the teacher was speaking directly to you, and there weren’t distractions around like in a normal classroom — would this improve the learning experience? There could be interactive elements of the class, though I don’t know how this would work with the teaching if it wasn’t live. There would have to be some kind of platform to deliver all of this efficiently and professionally — otherwise it may be too similar to other online learning platforms.

For live concerts, I think more people would attend if they could get a good seat without the hassle of going to a concert, paying for parking, sitting in uncomfortable seats, etc. The cost of setting up cameras in a venue would be less than putting on a concert in front of people, security wouldn’t be an issue, etc. How many people would attend a live concert if they could enjoy it from home and have the best seat?

What if Robert Plant and Jimmy Page decided to jam together in a living room — how many people would pay to see this around the world? The ticket would only work for the person that paid for it and attendance would be verified when the user puts on the device and has their ID verified (I would imagine through an iris scan). Basically, you couldn’t pay for something and then give your headset to someone else to view for free. Just like a live event, 1 ticket per person. Ticket costs could be low. You could charge $1 for a concert and have 100M attend = $100M for a single concert.

The productivity use cases you mention sound great — definitely not what most people are thinking about. I also think the idea of live concerts to be not something that the technology was meant for, but something that could transform live concerts.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.