Become a MacRumors Supporter for $25/year with no ads, private forums, and more!

MacRumors

macrumors bot
Original poster
Apr 12, 2001
56,603
19,350



Apple has been granted an augmented reality navigation patent stemming from its acquisition of AR startup Flyby Media earlier this year (via AppleInsider).

The patent was published today by the U.S. Patent and Trademark Office under the title "Visual-based inertial navigation", and describes a system that allows a consumer device to position itself in three-dimensional space using data from cameras and sensors.

17977-15998-160823-Visual-l.jpg

The system combines images from an onboard camera with measurements gleaned from a gyroscope and accelerometers as well as other sensors, to build a picture of the device's real-time position in physical space.

The patent notes that visual-based inertial navigation systems can achieve positional awareness down to the centimeter scale without the need for GPS or cellular network signals. However, the technology is unsuitable for implementation in typical mobile devices because of the processing demands involved in variable real-time location tracking.

To overcome the limitation, Apple's invention uses something called a sliding window inverse filter (SWF) that minimizes computational load by using predictive coding to map the orientation of objects relative to the device.

17977-16000-160823-Visual-4-l.jpg

The system could be used in a navigational AR device that overlays an output image with location-based information. One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles. Another describes the use of depth sensors to generate a 3D map of a given environment.

Whether or not Apple uses the patent in an upcoming product is obviously unknown at this time, but the company has been relatively open about its interest in innovating in the virtual reality and AR space. Apple is said to have a large team experimenting with headsets and other technologies and is believed to have been working in the area since at least early 2015.

The patent was filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Michigan professor Stergios Roumeliotis, as its inventors.

Article Link: Apple Patent Details Visual-Based AR Navigation Device
 

SandboxGeneral

Moderator emeritus
Sep 8, 2010
26,483
9,996
Detroit
One scenario describes how the technology could be used to pinpoint items in a retail store as a user walks among the aisles.
This is something that I would find quite useful. I'm the type of shopper that wants to get in the store and get out as quickly as I can. As long as I already know where everything I need/want is in the store, like my weekly grocery shopping task, I'm good. I don't particularly like having to hunt for something in a store that I don't normally frequent. If I had the ability to map out a specific items location in an unfamiliar store and then walk right to it when I get that, that would be quite useful to me.
 
  • Like
Reactions: Oblivious.Robot

Thunderstrike76

macrumors regular
Feb 8, 2012
104
162
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
 

2457282

Suspended
Dec 6, 2012
3,327
3,014
I think AR has far more average people application (something I was stating on another thread yesterday). VR has its place but right now AR is something that can be applied to more things and this is a perfect example. It would be awesome if I could make a shopping list and have AR map my trip through the store and point out where each item was. Having said that this is not something for the phone a la HER, where you need to have it in your shirt pocket. This would need to be more like a google glass implementation (but better) I think.
 

anthonylambert

macrumors regular
Mar 20, 2002
186
42
UK
Apple bought the company PrimeSense which developed some of the technology in Microsoft Kinect.

I believe Apple is developing AR software and glasses that like the Microsoft Hololens will have Kinect like hardware in it. Basically it will sense using infrared where all the surfaces are in a room - a little like short range radar. This then allows Apple to Augment the view you can see through the glasses with images super imposed onto the view in real-time. These glasses would work along side some other computing device like a Mac, iPad or iPhone.
 

orbital~debris

macrumors 68000
Mar 3, 2004
1,581
3,404
UK, Europe
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.
Came to the comments to make a similar point (this seems obviously developed with intention to use in the Car; side benefit may be application on personal devices too) but you've beaten me to it - and with a more detailed post :)
 
  • Like
Reactions: Thunderstrike76

anthonylambert

macrumors regular
Mar 20, 2002
186
42
UK
While this might be useful in a Car windshield.... It may prove a little dangerous if not limited to minimal relevant use!

Microsoft have gone on record saying that they will be launching Hololens for Windows 10 next year and it will be integrated into the Windows Desktop or some new 3d AR version of it.

It makes sense for Apple to be developing some Operating System level AR functionality where you can run multiple AR applications at the same time. So you can be running a virtual TV on your wall while you play a virtual game on the table in front of you and look out the window and see the names of stars and planets in the sky provided by your astronomy app for example.

For this is be viable Apple need to update all the Mac range with GPUs that are powerful enough to support AR.

If Apple aren't working on this they will miss the next big thing. This will hit before any Apple Car. This could be the next Mac, iPod, iPhone, iPad type product. iSee?
 
Last edited:
  • Like
Reactions: AlexGraphicD

ksnell

macrumors 6502a
Aug 26, 2012
696
1,202
Texas
It seriously amazes me I have not seen anyine talk about this yet. Tim Cook recently describe AR as seeing it as a core technology. I see no useful benefit of using AR on an iPhone or a device like an iPhone. The only scenario that makes any sense to me would to make AR a core technology of the rumored Apple Car. The windshield is the ultimate screen in my opion and it would also benifit from multiple overlays. If they did make a car that would be the next user interface. Controlling your autonomous car with knobs and switches doesn't seem very disruptive and Apple to me. That's how they separate themselves from Tesla and Google.

I don't think I want my windshield covered in fingerprints though.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.