Apple Releases ARKit 3.5 for Developers With Support for iPad Pro's LiDAR Scanner

[MacRumors]

Apple today informed developers that ARKit 3.5 is now available, with the update adding support for the LiDAR Scanner and depth-sensing system included in the new 11 and 12.9-inch iPad Pro models.

According to Apple, the new LiDAR Scanner will allow for a "new generation of AR apps" that use Scene Geometry for enhanced scene understanding and object occlusion.

Existing AR experiences on iPad Pro can also be improved with instant AR placement and improved Motion Capture and People Occlusion.

The LiDAR Scanner uses reflected light to measure the distance from the sensor to surrounding objects up to five meters away, indoors and outdoors. Depth frameworks in iPadOS combine depth points measured by the LiDAR Scanner, data from the two cameras, and data from motion sensors with computer vision algorithms handled by the A12Z Bionic to create a detailed and complete understanding of a scene.

At the current time, the new LiDAR Scanner with its enhanced augmented reality capabilities is limited to the new iPad Pro models, but Apple is also expected to include the feature in the 2020 iPhone models set to be released this fall.

Article Link: Apple Releases ARKit 3.5 for Developers With Support for iPad Pro's LiDAR Scanner

 

[farewelwilliams]

people that say "but the 12Z chip is awful!" have no clue what it means to include an expensive lidar feature.

 

[[AUT] Thomas]

At the current time, the new LiDAR Scanner with its enhanced augmented reality capabilities is limited to the new iPad Pro models

But I guess the LiDAR should still help in normal AR applications as well, e.g. when it comes to detecting a surface. Currently this is done using the cameras and AI only. So, the LiDAR should be able to transparently assist that AI in current AR applications as well, or?
The question there would be: will it remove lag in AR applications? Last time I tried an AR app the stabilization of AR objects was imho too slow. For objects in space it doesn't really matter, but for objects supposed to be attached to or sitting on a surface it didn't really met my expectations...

 

[chucker23n1]

But I guess the LiDAR should still help in normal AR applications as well, e.g. when it comes to detecting a surface. Currently this is done using the cameras and AI only. So, the LiDAR should be able to transparently assist that AI in current AR applications as well, or?
The question there would be: will it remove lag in AR applications? Last time I tried an AR app the stabilization of AR objects was imho too slow. For objects in space it doesn't really matter, but for objects supposed to be attached to or sitting on a surface it didn't really met my expectations...

Yes. The calibration step goes away.

Instant AR
The LiDAR Scanner on iPad Pro enables incredibly quick plane detection, allowing for the instant placement of AR objects in the real world without scanning. Instant AR placement is automatically enabled on iPad Pro for all apps built with ARKit, without any code changes.

 

[calstanford]

I take Gimmicks the world doesn't need for $100, Alex

 

[Defender2010]

This is all in preparation for their glasses. To gather real life data. iPhone next when devs have caught up.

 

[TheShadowKnows!]

Forget about AR worlds.
Instead, iPad mounted on auto with LIDAR look-ahead obscuring laser-fitted, speed guns.
"I have an app for that!"/s

 

[az431]

I take Gimmicks the world doesn't need for $100, Alex

Always fun when people pass judgment based on their own internal subjective beliefs. But at least you didn't say that you polled all 8 of your friends and none were interested in it.

 

[vmistery]

Presumably if it does come to the new iPhone it will be only on the non pro models? Have to wait and see I guess. It will be interesting to find out if the A13 can make any additional use of it as well over the A12Z

 

[MauiPa]

people that say "but the 12Z chip is awful!" have no clue what it means to include an expensive lidar feature.

No one says the a12z is awful

 

[Pman17]

Presumably if it does come to the new iPhone it will be only on the non pro models? Have to wait and see I guess. It will be interesting to find out if the A13 can make any additional use of it as well over the A12Z

I'm thinking we'll see a LIDAR on the iPhone 12 Pro models then it'll be on all of them next year. This A12Z is so odd that I think Apple decided to not develop a A13X to have more time to work on an A14X. They must have decided to release this iPad Pro anyways since it's due for a update and to get the LIDAR out in the wild for developers so some apps will be LIDAR ready when the iPhone 12 comes out and they can market the hell out of it. It's also a way to get the content rolling for the AR glasses so once those are announced, the apps are already here. We could see another iPad Pro in the Fall with a huge bump in features.

 

[citysnaps]

We could see another iPad Pro in the Fall with a huge bump in features.

What features would you like to see that would offer a significant and easily noticeable benefit over the just released iPad?

 

[SurferPup]

Another use for LiDAR could be to help jam Police fixed and mobile speed cameras which are used for revenue generation in Australia. Could also track Police speeding and report these Officers' abuse of power to the CJC. This is a business opportunity for an App Developer. An interesting but unknown fact, except among Police, is the Nokia 8110 (the Matrix mobile phone) would trigger the Police radar detectors - them thinking you had a radar detector in your car.

 

[farewelwilliams]

No one says the a12z is awful

literally? no. in a sense of disappointment? yes.

 

[citysnaps]

I take Gimmicks the world doesn't need for $100, Alex

Nineteen years ago that would have been: I'll take Who Needs 1,000 Songs in Your Pocket for $100, Alex.

 

[chucker23n1]

Nineteen years ago that would have been: I'll take Who Needs 1,000 Songs in Your Pocket for $100, Alex.

True, but I don’t think ARKit is the runaway success just yet. There’s some other shoe yet to drop.

 

[iPadified]

Can the LiDAR be used as a 3D scanner? That would be very useful. Scan your foot and get the correct pair of shoes. Scan your head and try glasses in 3D.

Will photographs be improved by getting distance measurements to objects?

It far from a gimmick, but it is for other professionals than developers. 3D representations of objects on 2D screens are definitely useful right now. Think engineers, medical doctors, real estate agents, architects, teachers and probably many more that I cannot think of at the moment, but please fill in.

 

[chucker23n1]

Can the LiDAR be used as a 3D scanner? That would be very useful. Scan your foot and get the correct pair of shoes. Scan your head and try glasses in 3D.

Will photographs be improved by getting distance measurements to objects?

It far from a gimmick, but it is for other professionals than developers. 3D representations of objects on 2D screens are definitely useful right now. Think engineers, medical doctors, real estate agents, architects, teachers and probably many more that I cannot think of at the moment, but please fill in.

My understanding is this LIDAR has good z (depth), but not very good x and y. So the point cloud you get is rather limited.

 

[MandiMac]

My understanding is this LIDAR has good z (depth), but not very good x and y. So the point cloud you get is rather limited.

Do you have any source? I mean if it helps the Measure app x and y should be reliable as well.

 

[chucker23n1]

Do you have any source? I mean if it helps the Measure app x and y should be reliable as well.

I don't have a source.

But as far as the Measure app is concerned: that's helped by the z. The camera lenses are primes, so previously, you were prompted to move your device in a three-dimensional space, which creates a parallax, which helps extrapolate the z. With the LIDAR, it gets real actual z. Much faster and more accurate. The x and y are then good enough for measurements if you move the device around slightly.

LIDARs that produce decent point clouds are much bigger (think several AirPods stacked on top of each other). LIDARs that produce excellent point clouds are massive (about the form factor of a wastebasket), heavy, and expensive (five figures). [ I'm involved in a research project on using LIDAR in a maritime context. ]

But the one they put in here already helps a ton. Just, no, I don't think it's very suitable as a 3D scanner. Maybe if you combine that with machine learning, you can extrapolate something OK.

 

[Vjosullivan]

Can the LiDAR be used as a 3D scanner? ...

Possibly, but so can any camera. You "only" need sufficient pictures from different angles and suitable software to extract the data.

 

[chucker23n1]

Possibly, but so can any camera. You "only" need sufficient pictures from different angles and suitable software to extract the data.

Yes, but the LIDAR should significantly reduce the necessary angles.

 

[MandiMac]

Isn't LIDAR in theory the same like the Face ID combination of sensors on the front?

 

[otternonsense]

Nineteen years ago that would have been: I'll take Who Needs 1,000 Songs in Your Pocket for $100, Alex.

Wrong equivalence is wrong. Almost all of us had Walkmans, Discmans, ill-designed early MP3 players etc. in hand, and could use something better indeed. The opportunity was there for the taking. Lidar on the iPad also counts as first-time innovation, sure, but it's still fantabulously niche no matter how hard you try to put it in context. To go back to your example, that would be like Apple releasing the iPod.. within the iMac. It doesn't scale. Not like that.

A great deal of people already walk around with depth sensing equipment: it's called Portrait mode and that's more than they care about. Other than Pokemon go, AR is yet to be proven as a thing. It's been the cringiest part of Apple's keynotes watching those wonky AR Lego demos with people clamouring around an empty table or forced "what's a computer" tone-deaf ads.

AR has potential but a better iPad camera (whose rear camera is used maybe once a year or invoked by accident) is kinda off for fully democratising this tech.

And yea, for a 2-year update of a flagship device from the richest company in the world, that was kinda weak.

 

[citysnaps]

Wrong equivalence is wrong. Almost all of us had Walkmans, Discmans, ill-designed early MP3 players etc. in hand, and could use something better indeed. The opportunity was there for the taking. Lidar on the iPad also counts as first-time innovation, sure, but it's still fantabulously niche no matter how hard you try to put it in context. To go back to your example, that would be like Apple releasing the iPod.. within the iMac. It doesn't scale. Not like that.

A great deal of people already walk around with depth sensing equipment: it's called Portrait mode and that's more than they care about. Other than Pokemon go, AR is yet to be proven as a thing. It's been the cringiest part of Apple's keynotes watching those wonky AR Lego demos with people clamouring around an empty table or forced "what's a computer" tone-deaf ads.

AR has potential but a better iPad camera (whose rear camera is used maybe once a year or invoked by accident) is kinda off for fully democratising this tech.

And yea, for a 2-year update of a flagship device from the richest company in the world, that was kinda weak.

You missed my point. Back in 2001 when Jobs announced the iPod with his "1,000 songs in your pocket" catch-phrase, the cynical and imagination-less were crying, "Who asked for a 1,000 songs in your pocket?" Much like people here do today, and your post above (along with others), whenever some new and game-changing tech is brought out.

The new iPad with 3D depth-sensing camera is a great example of that.
[automerge]1585138630[/automerge]

True, but I don’t think ARKit is the runaway success just yet. There’s some other shoe yet to drop.

Of course not. ARKit needs a rear-facing 3D depth-sensing camera in order for that to happen.