iPhone AR (Augmented Reality)

Discussion in 'iPhone' started by zombitronic, Apr 6, 2009.

  1. zombitronic macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #1
    This thread is about a completely speculative yet completely possible feature to be built into a future iPhone. If you're not familiar with augmented reality, think of it as virtual images that are overlaid onto images of reality in real-time.

    A simpler explanation can be given with an example. Imagine you're standing at an empty table. You point your iPhone's camera at the table and through the display, in the center of the table you see an apple. As you walk around the table while pointing the camera at the center, you see a full, three dimensional apple. By looking through the display, you'd swear that it's really there, except that when you take away the display the table is obviously empty. This is the concept of AR.

    For more info, just Google augmented reality and watch a few videos. It's impressive in action.

    Some demos have been created to show off AR on the iPhone. This method uses markers. The camera analyzes the image on a paper marker and then compares it to an internal library of these marker images. The camera then displays a three dimensional object on top of or around this marker, relative to the position of the marker in the camera's sight.

    This is more of a fun app than a useful application. The next step would be a large scale outdoor implementation using GPS data instead of paper markers.

    The first step towards this ability would be a built in digital compass in the iPhone. Imagine this:

    You enter some directions in your iPhone. From the Maps app, there's a new button to "Show Destination Marker in Camera," or something like that. When you touch this, the Camera app opens. To the left or right on the screen are arrows to show you which way you need to aim the camera to see the destination. Through the display, you see the marker floating in the air in the distance above the destination.

    This can all be done with GPS and a compass. GPS gets your coordinates and matches them to your point of view in the camera. Using the compass and data from Maps, the camera knows which direction to display the marker.

    Then we need LIDAR and a stereo camera to judge distance to display augmented reality images in relation to their real world surroundings. LIDAR and a stereo camera wouldn't be used to judge the distance of the destination. GPS will do this. Instead, this technology would judge the distance of objects blocking the destination.

    Imagine this in a close range scenario:

    You're at one end of a small park and you add a destination marker to the other end. Let's assume that this augmented marker is just a big red ball about 10 feet tall if you were right up to it. First of all, the GPS would have to be precise enough to accurately display coordinates based on latitude and longitude but also altitude so it can display this marker at ground level, relative to where you're standing. The accelerometer would need to be sensitive enough to adjust data based on the tilt of your iPhone, as well. Assuming that it can, you would see this marker as a small globe on your display, fixed at a certain point in the distance.

    The problem would be that this marker would block out trees in the foreground, so the only way to judge distance would be as it appears larger on the screen as you get closer to it. To remedy this, a combination of LIDAR and a stereo camera could analyze objects in the foreground and allow these objects to appear closer to you than the marker, so the ball would appear to be partially blocked by trees. This would make it appear, when looking through the camera display, that there really is a big red ball on the other end of the park.


    I'm interested in hearing other people's thoughts and ideas about this topic.
     
  2. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #2
    With the news of the new iPhone to include a magnetometer (digital compass), this concept is closer to becoming a reality.
     
  3. Stormbringer macrumors regular

    Joined:
    Jul 20, 2007
    #3
    It would be great of course, but I don't think it's fully developed yet, is it?

    So it's a bit difficult for Apple implement this the iPhone in relative short time...
     
  4. Drumjim85 macrumors 68030

    Drumjim85

    Joined:
    Oct 7, 2007
    Location:
    DFW, TX
    #4
    if you put the right APIs in there (which they seem to be doing), Apple won't have to implement anything. They can let 3rd party folks figure it out.
     
  5. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #5
    I agree. AR is still relatively young, as far as mainstream development goes. If Google got into this, I could imagine some awesome results, especially tied into their Maps project.

    The GPS/magnetometer method of real world navigation is still new but at least the technology is affordable. Stereo (dual) cameras may not be expensive but the software that's needed to judge distance based on two images is probably not developed enough to be useful in consumer devices. LIDAR works, but the hardware is likely very expensive and not sized for mobile devices. I think we'll still be waiting a few more years for a true AR experience.
     
  6. dvdhsu macrumors 6502a

    dvdhsu

    Joined:
    Mar 28, 2008
    Location:
    Palo Alto, CA
    #6
    This could be quite amazing.
    However, the issue is how developed it will become.
    If Google implemented it in their Maps, then Apple would be pressured to support it, as Google Android would definitely support it.
    The other issue is obviously the stereo camera issue.
    http://en.wikipedia.org/wiki/Stereo_camera
    Manufacturing a stereo camera would reduce the sexiness of the iPhone (IMO), and wouldn't really have other uses except for Augmented Reality. If Apple thought of any other purposes for this, it may become reality.
     
  7. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #7
    I don't think so. Components inside the phone would need to be rearranged but the lenses are fairly small. I would imagine that instead of just one camera lens on the upper-left side of the back of phone there would be an additional lens on the upper-right side. Although, you'd probably want it to work in landscape mode, so maybe you'd put another lens under the upper-left lens equally spaced as the two lenses in portrait orientation. But then the camera would have to automatically switch between two lenses (the upper-right and lower-left) when you switch between portrait and landscape, not to mention making the original upper-left lens operate as a right "eye" when in portrait and a left "eye" when in landscape.

    Alright, that's getting confusing.

    Another purpose, aside from being able to judge short term distances, would be the obvious. Record and play back 3D video on your 3D monitor. Or your iPhone with a 3D display. I don't know of any consumer ready 3D displays yet, but if Apple ever got in this game you'd bet that they'd want a stereo camera to go along with it.
     
  8. nick9191 macrumors 68040

    Joined:
    Feb 17, 2008
    Location:
    Britain
  9. Pooshka macrumors 65816

    Joined:
    Jun 28, 2008
    #9
    Wait, should it not be iPhone XR (Extended Release)??????????? :mad:
     
  10. ZunePod macrumors regular

    Joined:
    Apr 1, 2009
    #10
    This is possible, with the new GPS chip being embedded.
     
  11. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #11
    I haven't read about this new GPS chip. Maybe I missed it. Hopefully it'll be more precise.

    Do you have a link?
     
  12. ZunePod macrumors regular

    Joined:
    Apr 1, 2009
    #12
  13. lanceharris macrumors newbie

    Joined:
    Apr 8, 2009
    #13
    Things start to get really interesting when you introduce near eye displays. For example that apple on the desk would simply appear to be there to your natural field of view. The possibilities of this are endless.
    I hope apple gets involved in the production of near eye display hardware, cause what's currently being done won't fly with the general public.
    http://www.vuzix.com/iwear/products_wrap920av.html
    I can't imagine a lot of people wearing these around all day. Apple has the design/fashion sense to make something iconic, like the ipod, that people will be clamoring to wear and use. The have the capital to invest in the R&D and patent licensing, for near eye displays, I hope they are looking ahead 10-15 yrs to see that this will be big as soon as the tech is miniaturized.
     

    Attached Files:

  14. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #14
    I like the rumor roundup page. Nice find on the GPS chips. I checked the data sheets, though, and there's no improvement in position accuracy. The current one and the new one are both 2m, steady state. That's good, but not good enough for very close range (within a few feet) GPS-based AR.

    I've been reading a bit about improving GPS position accuracy and it seems that there are a few ways to enhance this. Methods include augmenting external information, (maybe cellular and WiFi triangulation count, and possibly visual sensors; I'm not sure,) better integration with GPS time versus UTC time, (From Wikipedia: The difference is that GPS time is not corrected to match the rotation of the Earth, so it does not contain leap seconds or other corrections which are periodically added to UTC.) and (also from Wikipedia:) precise monitoring and measuring of the existing GPS signals in additional or alternate ways. This includes (a bit over my head) Relative Kinematic Positioning.

    Again, from Wikipedia: Relative Kinematic Positioning (RKP) is another approach for a precise GPS-based positioning system. In this approach, determination of range signal can be resolved to a precision of less than 10 centimeters (4 in). This is done by resolving the number of cycles in which the signal is transmitted and received by the receiver. This can be accomplished by using a combination of differential GPS (DGPS) correction data, transmitting GPS signal phase information and ambiguity resolution techniques via statistical tests—possibly with processing in real-time (real-time kinematic positioning, RTK).

    More work may have to be done to the satellite infrastructure, as well.

    From the Wikipedia page about GPS modernization:

    The project involves new ground stations and new satellites, with additional navigation signals for both civilian and military users, and aims to improve the accuracy and availability for all users. The target date is 2013 with incentives offered to the contractors if they can complete it by 2011.
     
  15. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #15
    Great post, lance. I completely agree, a head mounted display would provide the best experience. You'd probably be very interested in reading this:

    Apple Researching Laser-Based Head Mounted Display

    [​IMG]

    Hrmpf reports on an interesting new patent application from Apple published today. "Head Mounted Display System" describes a method for display images within a goggle-like headset that is worn by the user.

    While these headset displays already exist, Apple describes decoupling a "laser engine" from the goggles itself. Instead, it is a separate piece which generates the laser light which is piped into the headset.
    Apple notes that when the unit is not being used for playing video "no images are being created and thus the display unit acts just like glasses." In fact, they suggest that the headset could even include lenses and actually double as daily-wear glasses.


    Not exactly the iconic image of the iPod, but still a cool concept, nevertheless. Not to mention, it's just a patent image. The two "tubes" seem kind of lame, to me. They remind me of those strings to keep your glasses around your neck when you take them off. If they could be combined into one wire, I think that would look better, aesthetically.

    I really like the idea of no bulk on the glasses. Instead, you'd just presumably plug the glasses into your iPod.

    Another article that may get your hopes up:

    Apple Researching Virtual Reality Headsets

    Apple's latest patent application reveals that they're continuing to look into personal virtual reality headsets. The February 2008 application is titled "Automatically adjusting media display in a personal display system" and details a "personal display system" which can give "the impression of being in the theater."

    [​IMG]

    Apple suggests that by detecting the user's movements, the image could be adjusted accordingly:Apple even makes the point that they could take the realism so far as adding theatre surroundings, adjusting sound effects based on the user's "seat" and even adding outlines of other patrons sitting in the theatre.

    It's not unusual for Apple's patent applications explore novel concepts but Apple has published previous patents related to similar headsets.


    It's not the same as see-through goggles or glasses, but it's cool to know that Apple is interested in head-mounted displays. It's not unprecedented to think that this is just being patented for only part of the experience. Who knows what product these ideas will make their way into.
     
  16. lanceharris macrumors newbie

    Joined:
    Apr 8, 2009
    #16
    useful AR links

    Thanks for the apple patent links. I had seen those awhile back but had forgotten since. It's good to see that apple has at least some interest in creating that type of hardware. I'm sure that the images are in no way representative of a product apple would release, they just want to cover the patent ideas, not give away free design ideas to the competition, it seems something similar happened with the "iphone patent". I like what microvision is trying to develop, they suggest having a laser that enters the pupil and draws the images directly on the retina, thus preserving color and brightness in even the glaring sun.
    This is only my second post here, but I have been obsessed with AR lately that it is nice to hear others talk about it with similar enthusiasm. I just want to share a few links that are related to different aspects of AR, but as this is my second post here, I may not link it all up nicely, for instance not sure if people prefer embedded youtube videos or not or how to do it for that matter, but anyway. You've probably seen most of them but these are some of the more impressive videos and links (IMHO) that I think are starting to put all the pieces together, to provide a truly great AR experience.

    Ori Inbar's blog is an outstanding overview of everything AR in many areas, I'll try not to duplicate what he has already neatly organized.

    This video shows what AR gaming could become someday with headmounted displays and clever software. Fighting life size zombies in your own house. I believe armusement the company that created this demo is gone :(

    This video and this video show some impressive markerless tracking.

    Total immersion shows an impressive video where AR objects react with introduced physical objects on the fly (also in this video skip to about 2:45).

    Gesture controls demonstated here there are other more impressive examples but I saw this one today.

    Bruce Branit's video while more virtual reality than augmented reality is a must see. It is beautiful and demonstrates some clever gesture controls as well.

    Just thought I'd share a few things that stuck out to me. I worry that intellectual property being scattered among many different companies will hinder progress towards the best AR experience possible. Apple has definitely shown that they know how to bring all the pieces together and produce beautiful, seemless products. Unfortunately, no matter how good the AR experience is, it won't have mass adoption if you have to wear goofy looking glasses.
     
  17. reetu.raj macrumors newbie

    Joined:
    Mar 12, 2009
    #17
    Ya, I have worked a lot in AR. AR based applications will be cool as hell for iPhone !! cant believe this is happening !! this is awesome :D
     
  18. Mr. Giver '94 macrumors 68000

    Mr. Giver '94

    Joined:
    Jun 2, 2008
    Location:
    London
    #18
    I think Apple would probably partner up with one or more sunglass companies if they wanted to make this type of thing a success. It will definitely be interesting to see what happens down the road with this type of thing. The technology is pretty much there, but not for the mass market.

    I'm thinking the iOakley or iPrada. :)
     
  19. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #19
    I'd like some iAviators.
     
  20. zombitronic thread starter macrumors 65816

    zombitronic

    Joined:
    Feb 9, 2007
    #20
    These actually look a bit like Apple's style although it doesn't look like they're meant to be see-through HUD glasses.

    [​IMG]
     
  21. Mr. Giver '94 macrumors 68000

    Mr. Giver '94

    Joined:
    Jun 2, 2008
    Location:
    London
    #21
    Interesting, but they look a bit like tanning goggles. :p
     

Share This Page