Video-In and Magnetometers Could Introduce Interesting iPhone App Possibilites

[MacRumors]

http://www.macrumors.com/images/macrumorsthreadlogo.gif (http://www.macrumors.com/2009/04/08/video-in-and-magnetometers-could-introduce-interesting-iphone-app-possibilites/)

The discovery (http://www.macrumors.com/2009/04/07/iphone-video-recording-interface-digital-compass-voice-control-and-auto-focus-camera/) that Apple's next iPhone might contain both video input as well as a magnetometer (digital compass) opens up some interesting possibilities for future iPhone applications.

Over at TouchArcade (http://toucharcade.com/2009/03/26/impressive-augmented-reality-game-possibilities-on-the-iphone/), game researcher Blair MacIntyre (http://www.cc.gatech.edu/~blair/home.html) demonstrated an example of an Augmented Reality application that was in testing on the iPhone. Augmented Reality is described as a combination of real-world and computer-generated data, where computer graphics objects are blended into video footage in real time.

The example shown demonstrated the realtime overlaying graphics on top of video on the iPhone:

http://www.youtube.com/watch?v=_0bitKDKdg0

A more sophisticated example of a Zombie shooter was also shown (http://www.youtube.com/watch?v=uGNgyGU-81E) using non-iPhone hardware. Due to SDK video restrictions, these apps would not be allowed in the App Store at this time, but both of these would be possible on the new iPhone if the rumors hold true.

The inclusion of a magnetometer (Digital Compass) will also introduce a number of new possibilities. Unlike accelerometers, a 3-axis magnetometer can determine absolute position. A Nokia blog post (http://blogs.forum.nokia.com/blog/andreas-jakls-forum-nokia-blog/2009/03/24/carchallenge) details what this could mean:A traditional compass only works when it is held horizontally. As Paul Coulton recently described in his Forum Nokia Blogs posting, the Nokia 6210 includes a 3-axis magnetometer. Through some calculations, it is possible to find out the absolute direction in which the phone is facing, no matter how the phone is oriented.Examples of how this could be used include pointing your iPhone's camera at a building and the phone telling you what building it is by combining GPS, accelerometer and compass information. The iPhone could even overlay graphics and text on top of the image to provide additional information. Another possible application described is the ability to show information about stars and constellations simply pointing your iPhone towards the sky.

These sensors could also make this mockup/concept application (http://www.macrumors.com/iphone/2008/08/22/ihologram-looks-remarkable-but-is-it-real/) a reality as well:

http://vimeo.com/1496857

Such an application is not possible with the current iPhone's accelerometers alone.

Article Link: Video-In and Magnetometers Could Introduce Interesting iPhone App Possibilites (http://www.macrumors.com/2009/04/08/video-in-and-magnetometers-could-introduce-interesting-iphone-app-possibilites/)

[Harpo]

The iPhone could even overlay graphics and text on top of the image to provide additional information.

That's very futuristic, like Back to the Future II.

[roberry82]

It's exciting to think all that's in store in technology for us.

[djourd]

wow this is cool

[DeathSticks]

Soon to live in a Star Wars reality.. thanx Apple

[coolfactor]

Two words...

Unbelievably Amazing!!

again....

Unbelievably Amazing!!

The iPhone truly is living beyond what many of us could've imagined 2 years ago.

[Rychy]

Holy crap that sounds cool!!

[MikeELL]

I'm in love with this phone already.
2 more months, 2 more months...!

Can you imagine a new generation of first person shooter games: like paintballing or laser skirmish but in ANY environment you wanted to. Or going for runs every afternoon but racing a ghost of yourself from previous runs. Doing renovations of your house and being able to walk through an architect's 3d model. Surgeons being able to view a patient's CAT scans from any angle while in the middle of an operation.

Augmented reality... here we come.

[winks360]

How did these guys make these if the new iPhone is yet to be released :confused:

[Emulsion]

I really like this idea. I just hope someone makes an add on for those of us that can't upgrade this time around.

[kastenbrust]

I'm in love with this phone already.
2 more months, 2 more months...!

Can you imagine a new generation of first person shooter games: like paintballing or laser skirmish but in ANY environment you wanted to. Or going for runs every afternoon but racing a ghost of yourself from previous runs. Doing renovations of your house and being able to walk through an architect's 3d model. Surgeons being able to view a patient's CAT scans from any angle while in the middle of an operation.

Augmented reality... here we come.

Seriously man, become a programmer.

It is rather cool though, in that second vid the cat looked like it was popping out the screen :p

[DipDog3]

I'll play them all!

[dmmcintyre3]

It is rather cool though, in that second vid the cat looked like it was popping out the screen :p

It did.

I would use it to make it look like I had a 30 inch display and a Mac Pro

[MrSmith]

Would the compass work with the current 3Gs? I mean, is the required hardware already in the phone?

[ravenvii]

If this actually becomes reality in the new iPhone, I'm officially old.

When you are so mind blown by technology like that, is when you know you're no longer a member of the current generation.

[kastenbrust]

It did.

I would use it to make it look like I had a 30 inch display and a Mac Pro

Dual 30" Displays obviously.

Compass + GPS + Google Maps = Heaven.

[vvebsta]

Dual 30" Displays obviously.

Compass + GPS + Google Maps = Heaven.

Compass + GPS + Google Maps =Fountain of Youth, City of Atlantis, El Dorado :p

[arn]

How did these guys make these if the new iPhone is yet to be released :confused:

1st one is on an iPhone but Apple doesn't allow Video in officially, so it's a hack.

2nd one is a mockup.

arn

[kajitox]

This is VERY interesting stuff, thanks arn!

[SFStateStudent]

I totally love this stuff. Virtual Pet rocz...:p:p:p:p

[mikeinternet]

This would be great to use with google street view.

[talkingfuture]

Magnetometer is the coolest word ever!

[dongmin]

Google is already well on their way in creating 3D maps of major city centers. All that is required now is an efficient framework to stream 3D data to your phone.

Imagine another couple of years when people can wear special goggles with magnetometers built-in that can overlay 3D information onto the real world you're viewing through your goggles. Imagine also being able to project this information onto your car windshield. The possibilities are really endless. This could be a whole new paradigm of combining the internet's vast data with the physical world.

Not too far off really. All that's required is an iPhone Goggle™ and a really fast network connection to constantly fetch 3D overlay data.

[MrSmith]

Imagine another couple of years when people can wear special goggles with magnetometers built-in that can overlay 3D information onto the real world you're viewing through your goggles. Even wearing earphones can be dangerous along roads.

[Oh-es-Ten]

And Microsoft's response to this innovative, exciting and rapidly expanding platform is.... a big ass table.
:D

[cal6n]

Unlike accelerometers, a 3-axis magnetometer can determine absolute position.

As a militant physicist, I feel duty-bound to point out that this is utter nonsense.

In and of itself, a 3-axis magnetometer can only detect the roll, pitch and yaw angles of the device relative to the Earth's magnetic field lines. It can't even detect the phone's position relative to that field. It's only when these readings are coupled with data from the GPS and the accelerometer that the device will be able to calculate its position, velocity, acceleration, heading and attitude. Admittedly, that'll be pretty impressive and useful but, even then, these data are all relative values, and not absolutes.

To be fair, these relative values are probably of more practical use to everyone except maybe astrophysicists and cosmologists!

[neil1980]

If this actually becomes reality in the new iPhone, I'm officially old.

When you are so mind blown by technology like that, is when you know you're no longer a member of the current generation.

lol the current iPhone maps app did that with me. I was blown away that I could type the name of a store, see where it was and within 2 presses of the button id be speaking to them!!

I actually use the maps app more like the most sophisticated phone directory ever with the added bonus that you can use it as a map too lol

[hypermark]

Hard not to wax poetic when you see demos like the first one. Just incredible.

Now, the capitalist in me sees a huge marketplace for creating virtual pet overlays for real time communications, virtual reality apps and all sorts of video blue screen scenarios.

Let's hope the rumors are true, and this stuff is opened up at the SDK/API level, as the developer community will have a field day with this.

Now if I can just get Apple to port the iPhone OS platform to Apple TV, my living room would be positively electric!

Cheers.

Mark
--
Read: "Right Here Now" services (http://thenetworkgarden.com/weblog/2009/03/right-here-now-services-weaving-a-realtime-web-around-status.html): weaving a real-time web around status

[MatthewCobb]

2nd one is a mockup.
arn

So the cat's not true? It's just fancy CGI? Shame! It *looks* real!

[MacFly123]

Doesn't the G1 have a compass like that and when you move the phone in maps it moves the maps? They need that in the iPhone.

This would be really cool :) Honestly how do people deny that the iPhone started the mobile computing revolution??? It is insane what the iPhone has come to be and will be in the future! Can't wait :D

Oh, what i really really really want to accompany the next round of iPhone hardware and all this cool potential we are seeing, is a new Apple TV with an SDK and app store on the Apple TV. Can you people honestly imagine how FREAKING HUGE that could be??? Just think about it........Games linked to your TV with Apple TV and your iPhone as the controller, Tivo/DVR apps, widgets, Safari, endless possibilities........ Apple would seriously pave the way to the living room of the future. Please Apple open your eyes and do it quick before it is too late!!! :eek:

[mikeinternet]

So the cat's not true? It's just fancy CGI? Shame! It *looks* real!

That mock up has been around a long while. Even when it didn't really make much sense how it worked.

[Mac Heretic]

In the future with virtual sofas everyone gets fat while sitting on the floor.

[wonderbread57]

Omg that's freaking awesome! So MUCH awesome ZOMG!!

The great thing about the iPhone is that with its established app distribution store these things can be a reality as opposed to other platforms where the hardware may be there but not a viable option economically for companies or individuals to sink time into.

[sunfast]

The future is bright :)

[Veri]

I know this might sound odd, but could anyone tell me if they actually use their iPhone for real work? By work I don't mean making business-social arrangements, or replying to a quick e-mail, i.e. things you can do with any 'phone. I mean sit-down-and-concentrate work - the sort you might be able to do with a real laptop for hours on a train, or plane, or in the park.

The screen is just so small, and without the reassurance of a replaceable battery, the iPhone seems to be targeted as a portable gaming console and casual GPS client.

[wonderbread57]

^^- you're right. you that does sound odd.

[8CoreWhore]

As a militant physicist, I feel duty-bound to point out that this is utter nonsense.

In and of itself, a 3-axis magnetometer can only detect the roll, pitch and yaw angles of the device relative to the Earth's magnetic field lines. It can't even detect the phone's position relative to that field. It's only when these readings are coupled with data from the GPS and the accelerometer that the device will be able to calculate its position, velocity, acceleration, heading and attitude. Admittedly, that'll be pretty impressive and useful but, even then, these data are all relative values, and not absolutes.

To be fair, these relative values are probably of more practical use to everyone except maybe astrophysicists and cosmologists!
As a militant milk drinker, I feel duty-bound to point out that this is udder non-sen... oh, never mind... :p

[Schizoid]

And Microsoft's response to this innovative, exciting and rapidly expanding platform is.... a big ass table.
:D

Microsoft's solution has to be big, it has to support Steve Ballmer's weight as he bounces on it and rubs himself over it shouting, "Developers" over and over again...

[Schizoid]

In the future with virtual sofas everyone gets fat while sitting on the floor.

Don't worry...

In the future we shall have unibody aluminium sofas, that are too expensive to buy and uncomfortable, so we'll be forced to wander around with our iPhones looking after our virtual dogs and pointing it at our aluminium tables filled with random objects

[Oh-es-Ten]

I know this might sound odd, but could anyone tell me if they actually use their iPhone for real work?

The screen is just so small, and without the reassurance of a replaceable battery, the iPhone seems to be targeted as a portable gaming console and casual GPS client.

I use mine for both work AND play. I think it is quite obvious that it is not supposed to be a laptop replacement, but it is more than adequate for writing out detailed notes, reviewing PDFs - not to mention having the most accessible e-mail and calendaring of any phone I have ever had (I have I owned a bunch of smartphones over the years).

I don't think there is any device under the size of a netbook that does not have limitations for work on the road. I hated trying to work with a stylus etc.

[preservative]

Impressive!

[entropys]

Google is already well on their way in creating 3D maps of major city centers. All that is required now is an efficient framework to stream 3D data to your phone.

Imagine another couple of years when people can wear special goggles with magnetometers built-in that can overlay 3D information onto the real world you're viewing through your goggles. Imagine also being able to project this information onto your car windshield. The possibilities are really endless. This could be a whole new paradigm of combining the internet's vast data with the physical world.

Not too far off really. All that's required is an iPhone Goggle™ and a really fast network connection to constantly fetch 3D overlay data.

I think I will wait for the interface directly with your mind using a bluetooth connection to an iMplant. Wearing goggles as I wander the street would be rather limiting.

[scottness]

Very cool. I could waste another 6 months just playing with this stuff. --Not really, I do have work to do, impressive as this all is.

You would think a computer (sorry, iPhone) like this could AT LEAST have background apps... Then I could really use this thing for work.

[slowpoke]

From what I hear there are a good few Japanese mobile phone games which make pretty active use of the camera... Here's hoping western devs take note. Augmented reality could be damned cool.

[addicted44]

From what I hear there are a good few Japanese mobile phone games which make pretty active use of the camera... Here's hoping western devs take note. Augmented reality could be damned cool.

Now that BG apps are the only things left for people to complain about (which is also technically possible. Apple has to work harder to prevent it from happening, for battery life reasons), once Apple enables those when battery technology allows it, what will people start complaining about?

I am putting my money on "I can't get my work done unless Apple sells it in a flashy pink color with a picture of a giraffe engraved on the back"...

In all seriousness, though, more than background processes (which again, are a technical usability issue) Apple needs to improve inter-app communications, and notifications. The notification system with the popup dialogs is not very scalable, and will become a mess with push notifications.

[MrCubes]

43 comments so far, and nobody whining that {insert device here} has already had this for ages (ok, MacFly123 mentioned the G1, and the article itself obviously references the Nokia 6210 - but neither in a detractive way)!

For all their annoyances and tardiness in other areas (cut & paste?) the thing Apple really does well is integration. If the magnetometer rumour proves to be true (and it sounds very plausible) you can bet that the Core Location APIs will "just work" with them (either transparently, or with some extensions). So developers will be able to easily take advantage of them and that means more cool apps for the rest of us.

This is something you just don't see on other platforms (caveat: I haven't used the Android APIs).

[dasikes]

One use for video-in I'd LOVE to see would be as a monitor for video shoots. Seriously. When I'm shooting, it would be so helpful to have a small, compact monitor for my boss to look at without me even having to leave the camera, or depending on our small LCD display.

I think that's a fairly practical use. Obviously, if it could somehow be wireless, it'd be even more amazing.

[MrCubes]

Apple needs to improve inter-app communications, and notifications.

Would be nice - but there are issues there that make it a harder problem than it sounds.


The notification system with the popup dialogs is not very scalable, and will become a mess with push notifications.
How would you do it?
How would BG processes help with the scalability?

Personally, from having used, as well as developed for, mobile platforms that allow third party background process, I'm really loving the iPhone's lack of them and think the Notification Service system is the right solution. There might still be 0.01% of apps (figure plucked from the air in true forum posting fashion) that would benefit from having them, but I think it's a case of "the needs of the many".
Of course Apple can still screw it up with a MobileMe-esque rollout - but I'm hoping they were serious when they said the delay was while they redesigned it to scale enough to meet demand.

[Cousin Dirk]

Even wearing earphones can be dangerous along roads.

It's called Darwinism, and I think by looking at the state of the world today we could do with giving it a bit of a helping hand.

[audioteknika]

Wait.. A magnetometer? Just give me a nuclear powered battery and I'll be impressed..

iPhone G3 with 2 hours of battery life, comming to an apple store near you.. Yawn..

[DELLsFan]

Now that BG apps are the only things left for people to complain about (which is also technically possible. Apple has to work harder to prevent it from happening, for battery life reasons), once Apple enables those when battery technology allows it, what will people start complaining about?

I am putting my money on "I can't get my work done unless Apple sells it in a flashy pink color with a picture of a giraffe engraved on the back"...

In all seriousness, though, more than background processes (which again, are a technical usability issue) Apple needs to improve inter-app communications, and notifications. The notification system with the popup dialogs is not very scalable, and will become a mess with push notifications.

Good question, point, and assertion.

I think there will come a point where one of two things will happen:

1. Apple just won't or can't practically innovate too much further to improve upon the phone's hardware - really a computer - due to the form factor limits and battery restrictions.

2. The form factor (ie. size) of the phone will have to increase - making it more Star Trek PADD-like or small tablet-like to accommodate all the desired bells and whistles and battery demands. The iPhone will be reworked - and transform into some new product smaller than a netbook, but larger than the current iPhone (think mini-Kindle) that will still be able to make and receive phone calls (video and audio).

I think both 1 and 2 are possible ... with Apple simply releasing some sort of iPhone Deluxe model that is larger for 2 ... while keeping the original iPhone size form factor (iPhone classic) for 1.

:apple:

[NightFox]

Wow, just hope we're not setting ourselves up for a big disappointment - let's remember that potential and what we actually get on the next iPhone may not be the same.

Here's hoping though!

[X38]

Using the existing video out capabilities and the third party hardware access being added in the new SDK along with these new compass and video input functions, one could add something like this http://www.vuzix.com/iwear/products_wrap920av.html
and things might get very interesting indeed.

[miketcool]

3D magnetometer + Android + SkyMap app = amazing!

Basically the app just gets your GPS location so it loads the correct map. Then your screen shows the overlay of constellations for where you are looking. The possibilities really open up with these kind of interfaces. You can just move around and get the star information for any part of the sky, including those below horizon.

[7egend]

How did these guys make these if the new iPhone is yet to be released :confused:
the SDK doesn't support any of this, but you can code hardware to do anything.

---

I like how the first App is set to recognize the barcode to generate a certain animal and then allow it to move across the "scene" pretty slick really.

And the last one, I have seen this done before but not using a built in magnetometer or the touch controls. The pop-up affect is always great, cause everyone loves a good pop-up book.

[bokdol]

there is a japanese virtual doll that you can buy for your computer that you can interact with in the very same manner. a little more perverted but they have had it out for a bit now.

http://techblips.dailyradar.com/video/japanese_webcam_doll_offers_augmented_reality/

[iPhoneNYC]

Wow. Imagine the potential. And I understand the thing makes phone calls as well.

[diamond.g]

Would be nice - but there are issues there that make it a harder problem than it sounds.


How would you do it?
How would BG processes help with the scalability?

Personally, from having used, as well as developed for, mobile platforms that allow third party background process, I'm really loving the iPhone's lack of them and think the Notification Service system is the right solution. There might still be 0.01% of apps (figure plucked from the air in true forum posting fashion) that would benefit from having them, but I think it's a case of "the needs of the many".
Of course Apple can still screw it up with a MobileMe-esque rollout - but I'm hoping they were serious when they said the delay was while they redesigned it to scale enough to meet demand.

Honestly the only apps that would really use background processes would be internet radio type apps (think ipod) and skype like apps (think phone). Everything else could do without. The only fear I guess for the push notification feature is that everyone will want to use popups when a badge notification is probably more elegant. The only thing Apple needs to worry about is push notifications being a single point of failure (I seem to recall them not saying if they will guarantee service availability) see RIM...

[Lesser Evets]

It's quite amazing what can fit into a tiny iPhone.

Though nice programming and interesting at a glance, these games or applications shown aren't particularly useful or engaging.

I can't wait to see what Apple does with its new hardware.

[iPaf]

If this actually becomes reality in the new iPhone, I'm officially old.

When you are so mind blown by technology like that, is when you know you're no longer a member of the current generation.

Hey, I'm 20 and I'M blown away that these possibilities!

Don't worry, you're probably not that old :P

[twoodcc]

wow. these things sound great! can't wait for the new iPhone!

[Truz21]

The Dog coming out of the printed card isn't really anything new. Sony has a whole card playing game based around this with their PS3 and the EyeToy.

In fact you can produce something similar right now with a webcam and a printer, thanks to General Electric. Check it out:

http://ge.ecomagination.com/smartgrid/#/augmented_reality

[crossifixio]

damn this is good stuff. Apple is always one step ahead even when they are two steps back! :apple:

[*LTD*]

damn this is good stuff. Apple is always one step ahead even when they are two steps back! :apple:

Well consider this:

Apple's iPhone interface is about two years old now.

Other phones that are current, or that are due for release in the immediate future (aside from Pre vapourware), don't even come close. After 2 YEARS! And Apple is just widening and widening the gap.

It' sad to see hoe half-asleep the rest of the industry was at the time of the iPhone's release, and even now, two years later.

[megfilmworks]

In and of itself, a 3-axis magnetometer can only detect the roll, pitch and yaw angles of the device relative to the Earth's magnetic field lines.

It can also detect direction of the magnetic field. Hence where north is.

[Cander]

3D magnetometer + Android + SkyMap app = amazing!

Basically the app just gets your GPS location so it loads the correct map. Then your screen shows the overlay of constellations for where you are looking. The possibilities really open up with these kind of interfaces. You can just move around and get the star information for any part of the sky, including those below horizon.

Now that would be awesome.

[kspreen]

Hey all,

As the developer behind ARf I'm excited to see this discussion :) I want to clear up a few things though...

First, as a few people have mentioned, this isn't new technology. The only new thing here is putting it on the iPhone. Sony has been doing this with the EyeToy, and the new FLARToolKit has really opened up desktop augmented reality (using your computer's webcam).

However, what we're excited about here is handheld Augmented Reality (AR). The phone becomes a window to another world, which you can explore much more freely than you could with a fixed camera. Again, there are other small devices that will let you do this, but the iPhone is something of the "Holy Grail" in the mobile world at the moment. Putting these apps on the iPhone would be a great way of getting more people to think in terms of AR.

The video you see here uses an older version of the SDK. There's no reason these apps shouldn't exist now, except that the functionality required is in private rather than public frameworks. We can only hope that Apple intends to make any new video capabilities part of the public SDK, so that developers can finally explore this exciting class of applications.

Thanks for all your feedback! Hopefully you'll be seeing more apps like this later this summer ;)

[zombitronic]

A few of us have been discussing augmented reality on these forums for awhile. This is really amazing stuff and the technology of the future. Comments about yesterday's article (http://forums.macrumors.com/showthread.php?t=682224) about the new camera and magnetometer went into detail about this as well as 3-axis magnetometers. There are also a lot more links to videos showing off this technology on the iPhone and other devices. Recently, I started a thread (http://forums.macrumors.com/showthread.php?t=681753) detailing examples of GPS-based navigational AR. Check it out if you're interested. I really love this technology and love discussing it.

We won't see the true can't-tell-virtual-from-reality version of AR until we have the technology to implement augmented data in relation to all of it's real world surroundings. Think of an imaginary virtual friend. You point your camera at a location and you see a virtual person. Creepy. Anyway, as you move around the virtual person, keeping your camera pointed at them, you see the virtual person from all angles. It looks, through the display, as if there's really a person there. This would work great through a pair of goggles with see-through displays.

For this to work this well, GPS and the magnetometer would need to be precise enough to know the exact location and orientation of your device down to an inch, maybe even less. I'm not sure that GPS is there, yet.

In addition, if you put your hand up in front of your virtual friend, ideally, your hand should block them. Without technology to sense and judge distances of near range objects, however, your hand and everything else in the foreground would be blocked out by the augmented image. Technology such as a stereo camera (dual cameras to see two images and judge depth perception) and LIDAR (lasers that scan and determine range) would need to be implemented to properly display augmented data behind objects in the foreground, in relation to the augmented image's intended position. Unfortunately, LIDAR is bulky and expensive and stereo camera hardware and software isn't very consumer ready, yet.

Regardless, this gives you an exciting look at what the world may look like, or at least how we may look at the world in the future.

[zombitronic]

Hey all,

As the developer behind ARf I'm excited to see this discussion :) I want to clear up a few things though...

First, as a few people have mentioned, this isn't new technology. The only new thing here is putting it on the iPhone. Sony has been doing this with the EyeToy, and the new FLARToolKit has really opened up desktop augmented reality (using your computer's webcam).

However, what we're excited about here is handheld Augmented Reality (AR). The phone becomes a window to another world, which you can explore much more freely than you could with a fixed camera. Again, there are other small devices that will let you do this, but the iPhone is something of the "Holy Grail" in the mobile world at the moment. Putting these apps on the iPhone would be a great way of getting more people to think in terms of AR.

The video you see here uses an older version of the SDK. There's no reason these apps shouldn't exist now, except that the functionality required is in private rather than public frameworks. We can only hope that Apple intends to make any new video capabilities part of the public SDK, so that developers can finally explore this exciting class of applications.

Thanks for all your feedback! Hopefully you'll be seeing more apps like this later this summer ;)

Hey kspreen. I love your work! I'd love to get into the development of software like this. Imagine the possibilities... augmented advertising with virtual billboards, augmented navigation and virtual tour guides, augmented social networking real-world overlays, ... and I'm sure the adult content industry would get in on this.

[Stately]

Can't wait for WWDC ! It's gonna be phenominal. :cool:

[miketcool]

Now that would be awesome.

IS awesome.

SkyMap (http://www.starreservoir.com/)

[kas23]

This would be great to use with google street view.

Another application/function that would be totally possible is the ability to search Google Streetview.I know this would likely be entirely redundant, but it would be cool if we could take a photo(s) of a street and it would search the Streetview database and show exactly where you are via Streetview. Or, maybe Google could construct Streetview for areas that are not currently available via people's uploaded photos (a la photosynth). Or, allow people to upload photos that Google could incorporate into Streetview to show streets during different seasons or at different time periods (besides the static time snapshot we get now). The magnetometer would make these images much more precise and accurate. Some of these capabilities would revolutionize the ability to search for real estate.

[qubex]

There's a serious mistake in the post

A triple-axis magnetometer does not allow the determination of position, it only allows the determination of orientation.

Determining position is a much more complicated affair that requires (for example) a known point of origin and then continued integration of an inertial frame, as is the case with inertial guidance systems.

Or GPS.

Which raises the issue: if determning position were all that a triple-axis magnetometer is useful for, why would they bother to include redundant capability already covered by the GPS?

[guzhogi]

The dog video looks kinda cool. It would've been funny (in a perverted way) if there was a way to make the dog lift his leg (if you know what I mean). Hey, Apple allowed iFart.

[Cander]

IS awesome.

SkyMap (http://www.starreservoir.com/)

No. Would be. Not on the iPhone yet. I have no interest in the G1

[marv08]

It can also detect direction of the magnetic field. Hence where north is.

Not exactly, if I remember correctly...

It will know where the North Magnetic Pole is, it would still need to know your position to figure out where the Geographic North Pole is to establish the heading. Depending on where you are, this can be quite a deviation.

[mrklaw]

check this out if you have a webcam
http://ge.ecomagination.com/smartgrid/#/landing_page

youtube demo http://www.youtube.com/watch?v=00FGtH5nkxM


Sony uses similar technology in their 'eye of judgment' game, and also I had some little cards with an old sony vaio about 10 years ago that did similar things. Still impressive though, and as mentioned, its likely to be the way Apple expose this to developers that will bring the really interesting things out.

Even if they don't have the magnetometer, it is possible to use the camera to detect movement - there have been the odd game on older nokia phones that did similar things without a magnetometer.

[megfilmworks]

The magnetometer can produce a voltage proportional to the magnetic field it senses and it also senses polarity(Magnetic field flows South to North).

The other issue is the difference between Magnetic north and
True north (used on maps and charts).
This is called magnetic variation (around 13 degrees here in SoCal)
and it is dealt with by software post the magnetometer which only senses magnetic north.

[iMacoo7]

Gets better as the days go by

[schneb]

The most important information you can have when lost is...

Where are you.
Which way you are pointing.
Where you need to go from here.

A couple weeks ago, a friend of mine's grandson and his girlfriend lost his backpack over a cliff during a hike in Alaska with a blizzard coming in. All their shelter and food was in the pack. Luckily, his iPhone was in his pocket. He was able to make a call, but the GPS could not pickup his location. Rescuers eventually found them about midnight, thankfully.

[str1f3]

Would be nice - but there are issues there that make it a harder problem than it sounds.


How would you do it?
How would BG processes help with the scalability?

Personally, from having used, as well as developed for, mobile platforms that allow third party background process, I'm really loving the iPhone's lack of them and think the Notification Service system is the right solution. There might still be 0.01% of apps (figure plucked from the air in true forum posting fashion) that would benefit from having them, but I think it's a case of "the needs of the many".
Of course Apple can still screw it up with a MobileMe-esque rollout - but I'm hoping they were serious when they said the delay was while they redesigned it to scale enough to meet demand.


I disagree. With IM alone, there would be popups all over the place. This is not even considering developers like Erica Sadun are trying to find ways to add Twitter to the PNS. I can just imagine a sports junkie getting and update of all the college basketball games through the PNS. I'm sure there are a ton of apps that will use this feature.

These popups take over the whatever you may be doing. Only iCal appointments should have that priority. The pre and android offer a far better solution. You still get the notification but it does not overwhelm their screen.There would be a lot of applications that would make use of the PNS.
The Pre is also copying Apple and coming out with their own PNS and will still have background apps so it will further limit use of the battery. The Pre will have the best of both worlds from that stadpoint. Whether you like it or not Apple will have to eventually do background apps. There are some things that just can't be done through PNS. I would love the ability to use GPS while listening to Pandora in my car while the iPhone is in the charger.

My guess is that you will see an API that will allow apps to interoperate in OS 4.0 next year. For now the seemingly never ending list of features that Apple is adding to the iPhone should keep customers more than happy for another year.

[cdinca]

GPS with video input and augmented reality would make for some fun treasure hunts! "find the largest branch of the largest tree in Pleasure Point". You point your phone at that branch, and you get a clue to the next location.

Fun fun.

[alphaod]

Even wearing earphones can be dangerous along roads.

It's just Apple's way of population control. :p

[stagi]

cant wait to see what this brings!

[zombitronic]

There's a serious mistake in the post

A triple-axis magnetometer does not allow the determination of position, it only allows the determination of orientation.

Determining position is a much more complicated affair that requires (for example) a known point of origin and then continued integration of an inertial frame, as is the case with inertial guidance systems.

Or GPS.

Which raises the issue: if determning position were all that a triple-axis magnetometer is useful for, why would they bother to include redundant capability already covered by the GPS?

Q, you answered your own question. Determining position is not what a triple-axis magnetometer is used for. It allows the determination of orientation. Therefore, it's not redundant, since GPS does not determine orientation.

[BeSweeet]

It looks really cool, but it (sadly) looks kinda fake.

[jhsfosho]

This stuff is really cool. In fact, its amazing that people come up with it.

[Niteace]

This all sounds really cool and I hope becomes a reality! Imagine this with The Sims game, seeing how your little people are living etc!

[arn]

It looks really cool, but it (sadly) looks kinda fake.

it's not fake. here it is demonstrated "in person"

edit: oops forgot the link http://toucharcade.com/2009/03/26/impressive-augmented-reality-game-possibilities-on-the-iphone/

arn

[Dagless]

Thats outstanding. As a developer myself (though a stoic one working on a single traditional project than something as advanced as this) I'm blown away by the potential the upcoming iPhone could have.
I dismissed the current iPhones gaming potential, and even if the iPhone doesn't get games as great as Mario, Pokemon, LocoRoco and Monster Hunter at least there will be some stellar tech demos over the next few years.

[months]

update from source...

Update: It's not real:

"The iHologram app was not real. It was an illustration of an idea I had which I believe could work with the technology (combining anamorphosis and motion sensing). Unfortunately I’m just an ideas person, I can show how things should look, but I’m no hardcore programmer."

http://www.macrumors.com/iphone/2008/08/22/ihologram-looks-remarkable-but-is-it-real/


That sucks... it seemed so cool if it was real :(

[Stately]

cant wait to see what this brings!

I agree. I love apps like Color Expert and Shazam, that can readily translate real time info whether it be visual or sound and provide an immediate translation. I would love to see an app developed that when one takes a picture or keeps the video camera on a particular subject for a certain amount of time could do something like scan and analyze the features of a subject i.e. a bird and the size, shape and color pattern of it's feathers and claws or a type of boat etc . . and give a full history and definition of the thing in question. I think that would be next level. iphone 10 maybe? :p

[NightFox]

...In addition, if you put your hand up in front of your virtual friend, ideally, your hand should block them. Without technology to sense and judge distances of near range objects, however, your hand and everything else in the foreground would be blocked out by the augmented image. Technology such as a stereo camera (dual cameras to see two images and judge depth perception) and LIDAR (lasers that scan and determine range) would need to be implemented to properly display augmented data behind objects in the foreground, in relation to the augmented image's intended position. Unfortunately, LIDAR is bulky and expensive and stereo camera hardware and software isn't very consumer ready, yet.

Gee, thank goodness for that. It was seriously looking as if there'd be nothing left to moan about on the next iPhone. At least I can now look forward to all of the "What... No LIDAR??? I'm through with Apple" postings come the Summer

[kdarling]

How did these guys make these if the new iPhone is yet to be released :confused:

Augmented reality has been done on other devices already. It's not iPhone specific. A lot of code sections are out on the net for programmers to play with.

Reminds me of how everyone thought that using cell towers or hotspots was some new locating technology... when public domain projects had been around for years... and are often the basis from which commercial implementations come.

Ditto for a lot of stuff we'll be seeing soon. For instance, there are public projects out there that follow your hands to manipulate Google Maps or other things. Microsoft has hinted that their future mobile phones won't need touch, since they'll watch your actions via tiny cameras instead. (For that matter, HP's Touchsmart uses cameras for their touchscreen, so that nothing hinders the video display clarity.)

So much cool stuff. So little time!

Honestly the only apps that would really use background processes would be internet radio type apps (think ipod) and skype like apps (think phone).

Also location-aware apps, which on other phones automatically do things for you, such as sending your kid a text to let them know you're about to pick them up... or will be late.

And anything that you'd like to happen without your having to manually relaunch the app. Other mobile OS's allow apps to register to auto-start upon a notification.

The only fear I guess for the push notification feature is that everyone will want to use popups when a badge notification is probably more elegant. Agreed.

The only thing Apple needs to worry about is push notifications being a single point of failure (I seem to recall them not saying if they will guarantee service availability) see RIM...They don't even guarantee notification delivery. Then of course there's the problem if you're in areas (such as basements or airplanes) where a background notification might not, or can't, come... which would otherwise be done via a simple alarm task.

[mambodancer]

The most important information you can have when lost is...

Where are you.
Which way you are pointing.
Where you need to go from here.

A couple weeks ago, a friend of mine's grandson and his girlfriend lost his backpack over a cliff during a hike in Alaska with a blizzard coming in. All their shelter and food was in the pack. Luckily, his iPhone was in his pocket. He was able to make a call, but the GPS could not pickup his location. Rescuers eventually found them about midnight, thankfully.

Now think about this feature as it might be implemented within google maps. As you use the GPS feature, the map is actually oriented in the direction you are traveling and automatically rotates with each turn you make. Additionally, the built-in speech then tells you at each junction point what turn to make and how far to go. All of this should be possible with the 3.0 update.

[mambodancer]

GPS with video input and augmented reality would make for some fun treasure hunts! "find the largest branch of the largest tree in Pleasure Point". You point your phone at that branch, and you get a clue to the next location.

Fun fun.

I'm thinking of the Rowdy Piper movie "They Live!" Hold the iPhone up and see what hideous monsters lurk among us. Kewl!

[Mattjeff]

Microsoft's solution has to be big, it has to support Steve Ballmer's weight as he bounces on it and rubs himself over it shouting, "Developers" over and over again...

>_<

I was eating... Thanks.

[qubex]

Q, you answered your own question. Determining position is not what a triple-axis magnetometer is used for. It allows the determination of orientation. Therefore, it's not redundant, since GPS does not determine orientation.

That is precisely my point: mine is not a question. Rather I'm rhetorically asking the poster why they'd believe that Apple has incorporated two redundant systems into the same device.

[masse]

of course Georgia Tech came up with it.

[michael.lauden]

that is freaking ridiculous

[jerkjosh]

Magnetometers? I'm taking a MEMS course. This'll be an awesome question to ask the prof (wasting time in class :-P)

[iMacoo7]

Not to sound like a "fan boy" , but since I have used Apple products, implementing new software,firmware is done on a different level than others.
That being said, I really think that this new feature set will not just be of the norm. but tote some really neat or unseen things or if the process has been seen it will be on a much more "wow" factor than others have used.
For only being in the smartphone market almost 3 years now, Apple has made allot of strides with the iPhone and continue to make more.

[zali0n]

I didn't watch all the video but can someone explain to me why the video "jumps" at 0:15 ? :confused:

[mikeinternet]

I didn't watch all the video but can someone explain to me why the video "jumps" at 0:15 ? :confused:

Not fake.

[kspreen]

I didn't watch all the video but can someone explain to me why the video "jumps" at 0:15 ? :confused:

The game takes a long time to load and I didn't want to lose everyone in the first 30 seconds. But you're right, from a convincing-of-truth standpoint that doesn't help.

I assure you this isn't fake though, we showed the demo at GDC and you can see a live video demo on TouchArcade:

http://toucharcade.com/2009/03/26/impressive-augmented-reality-game-possibilities-on-the-iphone/

[t0mat0]

http://www.macrumors.com/images/macrumorsthreadlogo.gif (http://www.macrumors.com/2009/04/08/video-in-and-magnetometers-could-introduce-interesting-iphone-app-possibilites/)

The discovery (http://www.macrumors.com/2009/04/07/iphone-video-recording-interface-digital-compass-voice-control-and-auto-focus-camera/) that Apple's next iPhone might contain both video input as well as a magnetometer (digital compass) opens up some interesting possibilities for future iPhone applications.

Over at TouchArcade (http://toucharcade.com/2009/03/26/impressive-augmented-reality-game-possibilities-on-the-iphone/), game researcher Blair MacIntyre (http://www.cc.gatech.edu/~blair/home.html) demonstrated an example of an Augmented Reality application that was in testing on the iPhone. Augmented Reality is described as a combination of real-world and computer-generated data, where computer graphics objects are blended into video footage in real time.

The example shown demonstrated the realtime overlaying graphics on top of video on the iPhone:

http://www.youtube.com/watch?v=_0bitKDKdg0

A more sophisticated example of a Zombie shooter was also shown (http://www.youtube.com/watch?v=uGNgyGU-81E) using non-iPhone hardware. Due to SDK video restrictions, these apps would not be allowed in the App Store at this time, but both of these would be possible on the new iPhone if the rumors hold true.

The inclusion of a magnetometer (Digital Compass) will also introduce a number of new possibilities. Unlike accelerometers, a 3-axis magnetometer can determine absolute position. A Nokia blog post (http://blogs.forum.nokia.com/blog/andreas-jakls-forum-nokia-blog/2009/03/24/carchallenge) details what this could mean:Examples of how this could be used include pointing your iPhone's camera at a building and the phone telling you what building it is by combining GPS, accelerometer and compass information. The iPhone could even overlay graphics and text on top of the image to provide additional information. Another possible application described is the ability to show information about stars and constellations simply pointing your iPhone towards the sky.

These sensors could also make this mockup/concept application (http://www.macrumors.com/iphone/2008/08/22/ihologram-looks-remarkable-but-is-it-real/) a reality as well:

http://vimeo.com/1496857

Such an application is not possible with the current iPhone's accelerometers alone.

Article Link: Video-In and Magnetometers Could Introduce Interesting iPhone App Possibilites (http://www.macrumors.com/2009/04/08/video-in-and-magnetometers-could-introduce-interesting-iphone-app-possibilites/)

Nokia had this last year, Android did several months later- but will the iPhone masses take this up? What would be the compelling app or apps to get people to see it as useful, not frivolous? Is it one that wins through the cool factor (e.g. like showing Street View to many people the first time)?