Members of Apple's PrimeSense Team Patent Method of Interacting With Mac Using Hand Gestures

[MacRumors]

Apple today was granted a patent originally filed in August 2016, describing a method in which users would be able to control a Mac computer -- and potentially other devices -- using a "non-tactile three dimensional (3D) user interface" (via Patently Apple). The patent's inventor credits go in part to Amir Hoffnung and Jonathan Pokrass, two current Apple employees who joined the company from PrimeSense after Apple acquired it November 2013.

Some of PrimeSense's tech, which was originally used in Microsoft's Kinect devices on Xbox platforms, now resides in the front-facing TrueDepth camera of the iPhone X, and the new patent hints at a potential future where this technology expands in function to Macs as well. Instead of recognizing faces, Apple's patent describes a Mac that recognizes a "gesture by a hand," allowing users to interact with the computer without tactile inputs like a keyboard, mouse, or trackpad.

Images via United States Patent and Trademark Office​

The patent includes a variety of gestures that users would use to control the 3D user interface, including what are called "push," (figure 2) "wave," (figure 3) and "up" (figure 5) interactions, which are all grouped into a category of "focus gestures." According to the patent, some of these could be used to perform basic app interactions, like scrolling through a menu, as well as change the state of the system from locked to unlocked.

Gestures described herein include focus gestures and unlock gestures. A focus gesture enables the user to engage (i.e., take control of) an inactive non-tactile 3D user interface. An unlock gesture enables the user to engage a locked non-tactile 3D user interface, as pressing a specific sequence of keys unlocks a locked cellular phone. In some embodiments, the non-tactile 3D user interface conveys visual feedback to the user performing the focus and the unlock gestures.

Examples of unlock gestures include an "up" gesture (e.g., raising hand 30 a specified distance), a sequence of two sequential wave gestures, and a sequence of two sequential push gestures, as described in detail hereinbelow.

Apple has previously been granted patents related to 3D sensing and gesture controls, but the company has yet to release a product that takes advantage of these features. Some of these previous patents included gesture controls on iPads and iPhones as well as on Magic Keyboards, which would allow users to gain access to virtual buttons and potentially streamline certain elements of the user interface.

Of course, with Face ID already launched on iPhone X, it's predicted that Apple will look into adding the facial recognition software into future iMacs and MacBooks before it focuses on a new unlocking system. It's also unclear whether the new 3D user interface patent would include both pieces of technology -- 3D gestures and Face ID -- to further augment a Mac's unlocking process.

As with any patent, the technology in question might not make it to an Apple product in the near future, if at all, but it is an interesting glimpse into what Apple might be planning to do with PrimeSense's technology down the line.

Article Link: Members of Apple's PrimeSense Team Patent Method of Interacting With Mac Using Hand Gestures

 

[bbeagle]

I would definitely use this for switching spaces on my desktop, or scrolling. That's pretty much the only things I use my trackpad for. I use the mouse for everything else.

It would seem odd at first. Like I'm an orchestra conductor in front of my computer. Sorta cool. Sorta odd.

 

[Gorms]

Tough break, Leap Motion.

 

[PaddyTB]

How can this ever be novel or non-obvious? US patent system is soooo broken :-(

 

[Mindcrime]

microsoft and intel were pushing this a while back and it failed horribly.

 

[PastaPrimav]

I don't see these things coming to fruition. The iterations of watchOS and all the failed touchscreen desktops/laptops on the market have already taught us that raising your hands is not a workable UI. Apple has shaped their entire watchOS experience around minimizing the amount of time you spend with your wrist raised, because it is critical to the UX. Every second is a lifetime.

If this research leads anywhere, it will be quick, casual gestures that you perform once in a while (turning something on or off). There is simply NO application for extended hand gestures in the air. None whatsoever.

 

[meanmusic]

It could resemble BMW's Gesture Control

https://www.youtube.com/watch?v=ttTBJkE-6fs

 

[wlossw]

I like the idea of Face ID. I don’t care about air gestures at all.

 

[OldSchoolMacGuy]

Remember, Apple has thousands of patents and most never ever see real-life use.

This doesn't mean they're likely to put it into an upcoming product. In fact, most of the time Apple doesn't give away upcoming changes by filing a patent far in advance.

 

[PaddyTB]

I don't see these things coming to fruition. The iterations of watchOS and all the failed touchscreen desktops/laptops on the market have already taught us that raising your hands is not a workable UI. Apple has shaped their entire watchOS experience around minimizing the amount of time you spend with your wrist raised, because it is critical to the UX. Every second is a lifetime.

If this research leads anywhere, it will be quick, casual gestures that you perform once in a while (turning something on or off). There is simply NO application for extended hand gestures in the air. None whatsoever.

Tell that to Marcel Marceau

 

[Luap]

They say talking to yourself is the first sign of madness. I recon that will soon be displaced by waving and gesticulating wildly at a computer.

 

[zarmanto]

Remind me again, whether or not fictional representations can be used as prior art?

 

[JM]

I want my unlock to be a middle finger. That'll be fun.

Wait, no... I want the middle finger to force-close everything.

 

[Neepman]

You know what you can do with these gestures....
God. Things like this make me giad I will be dead soon enough .

 

[Po Dameron]

#26 in those diagrams. That's the new Mac Pro. Called it.

 

[jcmeyer5]

I got a hand gesture for this "technology"....

 

[keysofanxiety]

Why is everybody thinking this would be a new UI like MS's touchscreen/mouse & keyboard bastardisation?

It could just be for accessibility features; many would find this useful and there are already existing gestures such as these, but require external hardware to work.

 

[Lepton]

It should recognize American Sign Language, which would give you a gestural keyboard that is already well worked out, and some people already know it.

 

[840quadra]

I see great value in this as accessibility as well as new ways to be able to operate these devices without touch.

Loving the iPhone X for allowing me to operate it without touch, been a great kitchen companion in this regard. Offering more options for control outside of SIRI would be awesome!

It should recognize American Sign Language, which would give you a gestural keyboard that is already well worked out, and some people already know it.

I would love to see this, it would make adoption of sign language much larger. I can imagine that it would be a great learning opportunity as well!

 

[ChrisCW11]

Oh yeah, this is WAY more usable then touching the screen.

Also I am pretty sure Microsoft patented the **** out of gesture controls with Kinect and their newer Hololens product, so good look there Apple.

There is absolutely no innovation here, just copying someone else's idea with some minor change in intent. Sure maybe Apple can might patent a specific "gesture" that wasn't covered by Microsoft or many others, but the original innovation to use body movements to manipulate on screen objects was already innovated by Microsoft, and Nintendo, and Sony and a slew of other smaller companies more then a decade ago.

Apple is the penultimate patent troll crawling through patents to find some loophole or gap not covered so they can jump on it. Apple just wades into a market and rips everyone off years later and then creates a revisionist history duping fanbois in believing that they invented all this in the first place. It would be nice if Apple took their 250 billion in profits sitting in a foreign bank and actually came to market first with ANY kind of innovation not already pioneered by someone else first. Apple is a follower, period.

Apples ONLY innovation is their logo.

 

[bernuli]

That is great, but all I really want is a vertical scroll bar on the side of my MacBook Pro screen to easily scroll documents.

 

[asleep]

Because we want a bunch of lasers in our eyes and tracking our every move?

Think what Google analytics and Amazon can do with that...

We see you're reaching for a pen... how about a uni-ball Vision Rollerball Pens, Fine Point (0.7mm), Black, 12 Count?

Would you like some SweetLeaf Natural Stevia Sweetener to go in you tea?

Oops, looks like you spilled your coffee. How about some Bounty Select-a-Size Paper Towels, the quicker picker upper!

We're noticing a few unruly nose-hairs, we've ordered you a Fancii Professional Nose & Ear Hair Trimmer with LED Light, Water Resistant, Stainless Steel Blades, and Battery Power!​

 

[Lepton]

I don't see these things coming to fruition. The iterations of watchOS and all the failed touchscreen desktops/laptops on the market have already taught us that raising your hands is not a workable UI. Apple has shaped their entire watchOS experience around minimizing the amount of time you spend with your wrist raised, because it is critical to the UX. Every second is a lifetime.

If this research leads anywhere, it will be quick, casual gestures that you perform once in a while (turning something on or off). There is simply NO application for extended hand gestures in the air. None whatsoever.

Yes, and Apple knows this. They said many years ago that having a touch screen on an iMac or laptop is no good because your arm gets tired reaching for it all the time. So with this knowledge, I feel it would be as you say, a way to give quick commands on occasion.

 

[artfossil]

I think this is fabulous.

Especially for all of us who have different learning preferences (kinestheic learner here) and for all who are differently abled.

 

[Tec972]

LOL the picture. Looks like an Xbox 1 connect. "Can't innovate my ass" my ass.