MacRumors

macrumors bot
Original poster
Apr 12, 2001
54,126
15,932


A number of Apple patent applications, published earlier today, appear to be directly related to its long-rumored mixed reality headset, covering a range of aspects including design elements, lens adjustment, eye-tracking technology, and even software.

headset-patent-document-software.jpg


The patents, filed with the U.S. Patent and Trademark Office by Apple, were made public earlier today and seemingly relate to specific features of its mixed-reality headset product.

Firstly, Apple applied for a patent related to several of the design elements of a head-mounted display unit in a filing titled "Head-Mounted Display Unit With Adaptable Facial Interface." This filing attempts to explain how a number of individual design elements can prevent a headset from being moved around by facial movements, increase the ability of a user to move their face when wearing a headset, and improve general comfort.

headset-patent-design.jpg


In supporting a headset on both the upper and lower facial regions separately, Apple seeks to reduce tension and facial compression for a more comfortable fit. This is achieved with "a light seal that conforms to the face of the user," which also blocks out environmental light, various facial supports of different stiffnesses, and even a "sprung" lower section. Among the recent deluge of reports surrounding Apple's mixed reality headset was The Information's remark that Apple is using a supportive "mesh material" around the headset for comfort.

In another application published today titled "Electronic Device With A Tunable Lens," Apple describes a lens-adjustment system for a head-mounted display unit. To present content optimally to a specific wearer, the optical lenses inside a VR/AR headset usually have to be adjusted.

headset-patent-lens-adjustment.jpg


Apple's system for adjusting lenses involves using a first and second lens element "separated by a liquid-filled gap with an adjustable thickness." In modulating how much fluid is allowed in this gap, the headset is able to move the lenses closer together or further apart to be suitable for a specific user. Unlike many other VR headsets, which require users to manually move lenses, Apple's system is entirely electronic and controlled by actuators. Lens elements may also be "semi-rigid" to be able to adjust their curvature as needed.

Earlier this month, The Information claimed that Apple's headset would feature "advanced technology for eye tracking." Now, a new patent application from Apple simply titled "Eye Tracking System," outlines a process to detect the position and movement of a user's eyes in a head-mounted display unit.

The eye tracking system includes at least one eye tracking camera, an illumination source that emits infrared light towards the user's eyes, and diffraction gratings located at the eyepieces. The diffraction gratings redirect or reflect at least a portion of infrared light reflected off the user's eyes, while allowing visible light to pass. The cameras capture images of the user's eyes from the infrared light that is redirected or reflected by the diffraction gratings.

Infrared cameras and a light source can be placed within the headset behind the lenses to detect infrared light bounced off of a user's eyes. The "diffraction grafting," placed between the camera and a user's eye, may take the form of a thin holographic film laminated to the lens, and serves to direct the light from a user's eye directly to the camera, while allowing visible light from the headset's display to pass through as normal. The IR camera would need to be placed at the edges of the display panels, near to a user's cheekbones.

headset-patent-eye-tracking.jpg


The filing goes on to describe in detail how this system allows the headset to precisely locate and track the movement of a user's "point of gaze." The system is so accurate that it is even able to detect pupil dilation. In terms of software applications for eye tracking, Apple remains vague, but it does suggest that the technology could be used for "gaze-based interactions" such as creating eye "animations used in avatars in a VR/AR environment."

The Information said that Apple's mixed reality headset will use "cameras on the device, the headset will also be able to respond to the eye movements and hand gestures of the wearer." It is also of note that in 2017, Apple acquired SensoMotoric Instruments, a German firm that made eye-tracking technology for VR headsets. One purpose suggested by The Information was as follows:

Apple has for years worked on technology that uses eye tracking to fully render only parts of the display where the user is looking. That would let the headset show lower-quality graphics in the user's peripheral vision and reduce the device's computing needs, according to people with knowledge of the efforts.

Finally, Apple has revealed a user interface concept for "a virtual reality and/or augmented reality device," in a filing called "3D Document Editing System." The patent application sets out how documents can be edited in a virtual 3D space. The system involves a keyboard being paired with the headset via Bluetooth or a wired connection to edit text.

The VR device may be configured to display a 3D text generation and editing GUI in a virtual space that includes a virtual screen for entering or editing text in documents via a keypad of the input device. Unlike conventional 2D graphical user interfaces, using embodiments of the 3D document editing system, a text area or text field of a document can be placed at or moved to various Z-depths in the 3D virtual space.

The filing adds that the headset may be able to detect a user's gestures to move selected content in a document such as highlighted text, shapes, or text boxes, and explains that users could move document elements in three dimensions on a Z-axis. It goes on to list a number of specific finger gestures to perform specific actions in a 3D text editor application.

In some embodiments, a document generated using the 3D text editing system may be displayed to content consumers in a 3D virtual space via VR devices, with portions of the document (e.g., paragraphs, text boxes, URLs, sentences, words, sections, columns, etc.) shifted backward or forward on the Z axis relative to the rest of the content in the document to highlight or differentiate those parts of the document. For example, active text fields or hot links such as URLs may be moved forward relative to other content in the document so that they are more visible and easier to access by the consumer in the 3D virtual space using a device such as a controller or hand gestures.

headset-patent-document-software-2.jpg


The patent application also proposes that the headset could pass through a view of the user's real-world environment in the document editing software.

In some embodiments, the VR device may also provide augmented reality (AR) or mixed reality (MR) by combining computer generated information with a view of the user's environment to augment, or add content to, a user's view of the world. In these embodiments, the 3D text generation and editing GUI may be displayed in an AR or MR view of the user's environment.

This is similar to The Information's claim that the "cameras on the device will be able to pass video of the real world through the visor and display it on screens to the person wearing the headset, creating a mixed-reality effect," and the general parity of these patent applications to recent rumors about Apple's mixed reality headset is striking. While patent applications cannot be taken as evidence of the exact technologies Apple is planning to bring to consumer products, it is difficult to look past the way in which these patents fit into the bigger picture surrounding Apple's AR/VR project.

Analyst Ming-Chi Kuo has said that Apple will reveal an augmented reality device this year, and according to JP Morgan, the device will launch in the first quarter of 2022. The headset is expected to be priced around $3,000, competing with the likes of Microsoft's HoloLens 2, which costs $3,500. Yesterday, designer Antonio De Rosa shared photorealistic renders of what Apple's mixed reality headset is believed to look like based on recent rumors.

Article Link: Apple's Work on Mixed Reality Headsets Revealed in New Patent Filings
 
Last edited:

AngerDanger

macrumors 603
Dec 9, 2008
5,322
27,657
In terms of software applications for eye tracking, Apple remains vague, but it does suggest that the technology could be used for "gaze-based interactions" such as creating eye "animations used in avatars in a VR/AR environment."
It'd be cool if Apple used eye-tracking to dynamically render objects the user is looking towards at a high resolution and the objects in their peripheral at a lower, blurrier res.

That way, the effective resolution of the displays could be much higher while the computing power needed to drive them could be lower.
 
Last edited:
  • Like
Reactions: jefhai
Comment

gcmexico

macrumors 6502a
Dec 22, 2007
913
104
Littleton, CO
I’m still confused on what is the purpose for this product? Apple doesn’t have a game console or any games worthy for VR...VR to use on a mac? iphone? or maybe it will have Apple TV+ pre installed to watch movies? but why? I have a PS4 VR, there are games for it and I never use it, I tried watching a movie on it once, and got bored quick...the VR just sits by my PS4 now...I buy everything Apple, this might be the first product I completely skip
 
Last edited:
Comment

newyorksole

macrumors 601
Apr 2, 2008
4,445
5,227
New York.
I’m still confused on what is the purpose for this product? Apple doesn’t have a game console or any games worthy for VR...VR to use on a mac? iphone? or maybe it will have Apple TV+ pre installed to watch movies? but why? I have a PS4 VR, there are games for it and I never use it, I tried watching a moving on it once, and got bored quick...the VR just sits by my PS4 now...I buy everything Apple, this might be the first product I completely skip
You buy everything Apple yet are dismissing this product before it’s even announced because you can’t *possibly imagine* what you would use this for?

I just don’t understand the thought processes here sometimes Lol. I would think you know Apple well enough by now to wait until they reveal the thing...
 
Comment

jefhai

macrumors regular
Feb 11, 2021
148
254
People are missing the purpose of this and think it doesn’t make sense if it doesn’t play games...

This is for creatives and productivity! Super high resolution displays? Heck even the patent has “document backward” “document forward”

This is for developers. It is what John Carmack has said he has always wanted.

Imagine seeing your desktop in VR with retina legible text and media!

Coding in VR becomes bliss... enough screen space for everything!

I’m hoping this means macOS in VR..
 
  • Like
Reactions: Duane Martin
Comment

GermanSuplex

macrumors 65816
Aug 26, 2009
1,264
29,304
I can’t see Apple introducing a niche product like this these days... if they do, I’d imagine it would be far cheaper than they are predicting. The iPad was rumored to be expensive and the low price sort of shocked people when it was revealed. I can see the same happening here.
 
Comment

Duane Martin

macrumors 6502a
Oct 15, 2004
517
1,165
Calgary, Alberta
For approximately the first 10 years after motion pictures (film) was invented almost everything that was shot were Actualities; essentially a single set up watching an event take place. There is even an apocryphal story of Edison hiring and training 200 cinematographers on a 2 year contract because after 2 years they would have shot everything there was to shoot. People had no idea what could be done with the new medium.

That is where I feel we are with AR/VR. We simply have not imagined the use cases yet and we will not be able to until powerful and useable tools get into more hands. That happens to be something Apple is good at. Whether they will succeed here we cannot yet know but I, for one, will not be betting against them.
 
  • Like
Reactions: Mescagnus
Comment

Edsel

macrumors 6502
Mar 18, 2010
425
532
Over There
I’m failing to get excited over this. Maybe I need a pill.
You might be onto something. Apple is probably working on a pill. The pill would replace the physical computer/keyboard/screen. The new APPill would utilize "programmer" Timothy Leary's "code".
 
Comment

Schizoid

macrumors 6502a
May 29, 2008
863
541
UK
Patent #427:
“Glancing at Dan’s syrup automatically overlays Craig’s luxurious barnet”
 
Comment

filmantopia

macrumors 6502a
Feb 5, 2010
751
1,974
I can’t see Apple introducing a niche product like this these days... if they do, I’d imagine it would be far cheaper than they are predicting. The iPad was rumored to be expensive and the low price sort of shocked people when it was revealed. I can see the same happening here.
Apple’s long term intent is to make AR the next mass market paradigm in computing. This one product only markets the very beginning of long term ambitions that could entirely change the way we use computers. This specific niche product is a de facto beta/development tool to build a sophisticated and robust AR/VR platform unlike anything else on the market. Eventually the ‘big one’ will drop, which will be the lightweight, lower cost AR glasses.
 
Comment

CJ Dorschel

Cancelled
Dec 14, 2019
407
806
Berlin
In all seriousness from what I’ve been told by some inside Apple these are meant to replace displays. The example some gave:

when you’re on an airplane or anywhere watching a movie on your computer or iPad the glasses would replace your computer or iPad while giving you privacy for a full immersive experience.

The 8K resolution (4K per lens) - I don’t know how that would work as they explained the glasses would be more akin to everyday eyewear than goggles.

I initially thought it would be augmented reality system one would wear every day and overlay graphics that would give you information on stores, locations, traffic information, etc. - something out of “Ready Player One” - which just makes my head hurt. I can’t imagine the sensory overload and I can’t imagine Apple would design a device that would be more of a distraction from life as opposed to working seamlessly with our daily lives. I know Cook wants to make devices less distracting so we’re not always staring into our iPhones. Apparently this is meant to also alleviate social distractions by acting as a display that would connect to iPhones and eventually replace them when technology advances enough so we’re interacting with people more - looking up instead of down - while still being able to communicate with tech.

That I find interesting.

apparently they will also have AirPods built in or connect with AirPods.
 
Comment

dumoore

macrumors member
Nov 30, 2017
66
415
Disagree with you all, this is extremely exciting! There's limitless uses but simply isn't anything out there yet because there hasn't been any meaningful advances in AR/VR yet, mostly novelty uses at high price. Imagine 50 years from now when this becomes extremely small and powerful, consider even in everyday glasses with no bulk, or eye contacts.

Your grandkids will look back and laugh that we carried around a 5" screen we constantly poked at. Or sat at a desk leaning into a monitor. They will have phone/computer/tablet/etc in front of them, all around them at all times. Expand this video to fill your entire viewing angle, shrink it down and push it to the side while you read an article.

AR is the most exciting. Technician working on a complex machine sees nuts and bolts light up that he needs to remove, step by step instructions. Google lens translating everything you see in a foreign country. GPS always lighting up the path. Doctors can rely on ML processing billions of diagnoses and surgeries. This alone is worth investing heavily into AR/VR. We will get there!
 
Last edited:
Comment

jefhai

macrumors regular
Feb 11, 2021
148
254
We have imagined the use
Disagree with you all, this is extremely exciting! There's limitless uses but simply isn't anything out there yet because there hasn't been any meaningful advances in AR/VR yet, mostly novelty uses at high price. Imagine 50 years from now when this becomes extremely small and powerful, consider even in everyday glasses with no bulk, or eye contacts.

Your grandkids will look back and laugh that we carried around a 5" screen we constantly poked at. Or sat at a desk leaning into a monitor. They will have phone/computer/tablet/etc in front of them, all around them at all times. Expand this video to fill your entire viewing angle, shrink it down and push it to the side while you read an article.

AR is the most exciting. Technician working on a complex machine sees nuts and bolts light up that he needs to remove, hoses that he needs to connect, step by step instructions. GPS always lighting up the path. Combine this with machine learning too. Doctors can rely on ML processing billions of diagnoses and surgeries. This alone is worth investing heavily into AR/VR. We will get there!

I totally agree.

I think people are either saying oh but it’s for gaming and Apple sucks at gaming or we don’t know use cases for how AR/VR will be used...

AR/VR is already in use. Used for training etc look at Vusix.. look at what hololens is used for.

I think for consumer use- we already know all the use cases... just imagine what you do on your phone but in your face and hands free.. imagine things in AR, landmark objects with interaction that you can walk up to etc..

I think this is clearly the future. And Apple is about to flip the world with a premium product just like the iPhone...

No one will use a watch or a phone.. all optical displays strapped to our heads. Watches maybe still for swimming and athletics... But why use a phone when AR is a thing?

I would love to do the dishes and see a big YouTube video in the parking lot outside my kitchen window haha
 
Comment

JosephAW

macrumors 601
May 14, 2012
4,168
5,057
Someday all iPhone devices will just have one camera for wide angle to zoom. :rolleyes:
 
Comment

jefhai

macrumors regular
Feb 11, 2021
148
254
In all seriousness from what I’ve been told by some inside Apple these are meant to replace displays. The example some gave:

when you’re on an airplane or anywhere watching a movie on your computer or iPad the glasses would replace your computer or iPad while giving you privacy for a full immersive experience.

The 8K resolution (4K per lens) - I don’t know how that would work as they explained the glasses would be more akin to everyday eyewear than goggles.

I initially thought it would be augmented reality system one would wear every day and overlay graphics that would give you information on stores, locations, traffic information, etc. - something out of “Ready Player One” - which just makes my head hurt. I can’t imagine the sensory overload and I can’t imagine Apple would design a device that would be more of a distraction from life as opposed to working seamlessly with our daily lives. I know Cook wants to make devices less distracting so we’re not always staring into our iPhones. Apparently this is meant to also alleviate social distractions by acting as a display that would connect to iPhones and eventually replace them when technology advances enough so we’re interacting with people more - looking up instead of down - while still being able to communicate with tech.

That I find interesting.

apparently they will also have AirPods built in or connect with AirPods.

I thought I read 8K per eye. Also 8k is not two 4K (math for pixel count in K units doesn’t work that way)... 8k is quad 4K.. not two times. Just mentioning that..

Ready Player One was a lot of VR. The protagonist lived in a slum and went to high school in VR.

I think it will be more than just talking to people. I think we will still be distant and on our technology. Glasses isn’t going to change that...

But everything will be bigger. And no more any other kind of screen except for people with poor eyesight... Families will only own AR/VR glasses...
They will sync so you can share the same view like watching a movie on your 200 inch AR television...

Movie theaters will be completely dead. Cause you can turn a stary night sky into a huge IMAX Omni display... things will just get more technical... Imagine Japan having anime girls dancing outside coffee shops etc etc

I really don’t think Apples goal is to sell a assistant device.. Apple is probably taking their sweet time to blow us away.. They want people buying these glasses yearly like iPhones... if they really shipped a lackluster version one of AR glasses I don’t think people would be serious.

But if they made a revolutionary step above current AR VR tech. I think that will definitely be very interesting
 
  • Like
Reactions: CJ Dorschel
Comment

gcmexico

macrumors 6502a
Dec 22, 2007
913
104
Littleton, CO
You buy everything Apple yet are dismissing this product before it’s even announced because you can’t *possibly imagine* what you would use this for?

I just don’t understand the thought processes here sometimes Lol. I would think you know Apple well enough by now to wait until they reveal the thing...
that is why I am asking what is the purpose? everything Apple has been doing lately has a purpose watch, phone, airpods, airpods max, apple tv, homepods, mini homepods, ipad, ipad pro...all have a purpose right off the bat...I bought all of them and love them...VR is not demonstrating any purpose right off the bat, so as a consumer I am asking why VR...again, they have no console, no other product that can tie in other than computers...if it’s for their computer line, then cool, but no thanks for me
 
  • Like
Reactions: newyorksole
Comment

gcmexico

macrumors 6502a
Dec 22, 2007
913
104
Littleton, CO
People are missing the purpose of this and think it doesn’t make sense if it doesn’t play games...

This is for creatives and productivity! Super high resolution displays? Heck even the patent has “document backward” “document forward”

This is for developers. It is what John Carmack has said he has always wanted.

Imagine seeing your desktop in VR with retina legible text and media!

Coding in VR becomes bliss... enough screen space for everything!

I’m hoping this means macOS in VR..
that is the only tie in I can see...which would be awesome, but not really for media content
 
Comment

gcmexico

macrumors 6502a
Dec 22, 2007
913
104
Littleton, CO
I would be super on board for Apple Glasses...that shows my iphone info on the display but I can still see and walk, like Google Glass that failed...maybe Apple makes this full VR headset first then the glasses?
 
Comment

rum0rm0nger

macrumors member
Aug 7, 2020
40
103
that is why I am asking what is the purpose? everything Apple has been doing lately has a purpose watch, phone, airpods, airpods max, apple tv, homepods, mini homepods, ipad, ipad pro...all have a purpose right off the bat...I bought all of them and love them...VR is not demonstrating any purpose right off the bat, so as a consumer I am asking why VR...again, they have no console, no other product that can tie in other than computers...if it’s for their computer line, then cool, but no thanks for me
I think one of the reason folks have a difficult time seeing this as anything other than an entertainment device is because we are looking at this through a consumer lens (pun intended). AR and VR is already used widely in many commercial settings for things like design, training, realistic job previews, etc. For example, somewhat akin to a flight simulator, these devices have been used to train people for everything from using heavy machinery to hotel customer service.
 
Comment
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.