Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

MacRumors

macrumors bot
Original poster
Apr 12, 2001
48,667
10,094


Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.


This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.


Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.


Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Article Link: Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display
 

AngerDanger

macrumors 601
Dec 9, 2008
4,689
21,245


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
 
Last edited:

konqerror

macrumors 68020
Dec 31, 2013
2,177
3,490
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
The Kinect required expensive 3D scanning hardware, which ultimately Microsoft couldn't afford and discontinued. (Kinect games even attracted an additional royalty which I recall was rumored at $10) This is all done with computer vision.
 
  • Like
Reactions: ZZ9pluralZalpha

adamdport

macrumors member
Mar 19, 2011
40
195
Does this confirm the depth mapping stuff and therefore FaceId on Big Sur? Or does this just use the regular camera?
 

Pakaku

macrumors 68020
Aug 29, 2009
2,246
2,319
Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
You didn't, but you could still accurately interact with the phone screen and get feedback. I don't think motions in the air will feel very accurate for an interface, but it might be okay for tracking your motions or body posture, and the phone is sort of thinking for itself instead of relying on the user's input
 

macguru212

macrumors 6502
Apr 17, 2009
379
738
NYC/Tokyo
Kind of cool if we can use this data for motion capture and apply it to 3d models in C4D or Unity. Adobe has a pretty broad collection of MoCap models ready to go, but to be able to do this yourself would be fun.
 

Joniz

macrumors 6502
Sep 21, 2017
349
917
Great! Now Siri will be able to critique your ability in bed as it happens.
 

Relentless Power

macrumors Nehalem
Jul 12, 2016
33,203
34,206
Apple really is doing their best to pull abandoned Microsoft features into their ecosystem. Tiles from Windows Phone. now Kinect.
And yet, Apple always seems to refine the technology that other companies couldn’t execute properly in the first place. Isn’t that what makes Apple so unique in the first place?

[I’d say let’s give this technology a shot, I think it has potential if it’s implemented properly. I for one, want to see where this leads, especially with AR.]
 
  • Like
Reactions: Mobster1983

Baumi

macrumors regular
Mar 31, 2005
202
240
I don't think the wizarding community is going to be too happy about this...
From what I’ve read, wizards and modern technology don’t seem to go well together anyway, so Apple probably figured they could do without that particular demographic.

It doesn’t bode well, though, for my idea of an app that would judge the posture of Catholic priests during mass.
 
  • Haha
Reactions: SDJim

amartinez1660

macrumors regular
Sep 22, 2014
163
124
For motion graphics applications this would be great if expanded to it. Maybe we will start to see builtin tracking functions for puppeteering a person or object in a picture, or even better, translate performance capture from one to another pretty much like Animojis do with the front face cam or motion capture as we know it in films.
 

falainber

macrumors 68020
Mar 16, 2016
2,079
2,619
Wild West
Except they didn't do it right. You notice how they don't pursue Kinect anymore?
And you think Apple will do it right? This might be useful in very limited set of circumstances. With the regular interface, one just needs to move a finger to interact. With these new features one needs to move a hand or the entire body. Why would anyone want to do that? Apple introduce AR support three years ago. The demos were cool but it is hardly used.
 
  • Like
Reactions: PC_tech

jimbobb24

macrumors 65816
Jun 6, 2005
1,169
1,542
The Kinect required expensive 3D scanning hardware, which ultimately Microsoft couldn't afford and discontinued. (Kinect games even attracted an additional royalty which I recall was rumored at $10) This is all done with computer vision.
This may be why the iPad has LIDAR.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.