Beyond Multi-Touch: Voice, Gaze, Facial Expression Recognition

[MacRumors]

Apple's patent applications provide a glimpse into what might lie ahead in our future. For example, we saw an explosion of multi-touch patents in the year before the iPhone's release. And while many of these concepts may never come to market, they provide an interesting look into the direction of Apple's research.

One of the most recent patent applications from Apple is entitled Multitouch Data Fusion and is authored by Wayne Westerman and John Elias (formerly of Fingerworks). Westerman and Elias have been prolific publishers of multi-touch patent applications and likely helped establish the multi-touch technology behind Apple's iPhone. While many have since hoped to see some sort of advanced multi-touch interface for Apple's Macs, only limited multi-touch support has been included into Apple's notebooks.

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)

The patent application gives examples of how each of these could be used in conjunction with multi-touch to provide a better user experience. A few highlights are provided here:

Voice - Some tasks are described as being better suited for either voice recognition vs multi-touch. For example, if a user's task is to resize, rotate and change the color of an object on the screen, multi-touch would be best suited for the resize/rotation tasks but changing of color (or inserting text) may be better suited to a voice command.

Finger identification - the article suggests that using built in cameras (such as the iSight) with a swing mirror could provide an over the keyboard view of multi-touch gestures. This video information could be used to better distinguish which fingers are being used in which position. In conjunction with the touch input, this could be used to create more specific and accurate gestures.

Gaze - tracking where the user is looking could help pick windows or objects on the screen. Rather than resorting to moving a mouse pointer to the proper window, a user could simply direct his gaze at the particular window and then invoke a touch gesture.

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior. The example given is if a user is incorrectly trying to scroll a window using 3 fingers, instead of 2, the computer may be able to detect the frustration and either accept the faulty input or prompt the user.

Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.


Patent: http://appft1.uspto.gov/netacgi/nph-...RS=20080211766


Article Link

[punkybadhip]

Why can't Apple just concentrate on fixing its OS, and providing quality made hardware before doing something new.

なぜなら！

Apple is now Micopple

I want money (that's what I want)

[Wayfarer]

Originally Posted by punkybadhip

Why can't Apple just concentrate on fixing its OS, and providing quality made hardware before doing something new.

なぜなら！

Apple is now Micopple

I want money (that's what I want)

Snow Leopard, fool!

[themoonisdown09]

Originally Posted by punkybadhip

Why can't Apple just concentrate on fixing its OS, and providing quality made hardware before doing something new.

なぜなら！

Apple is now Micopple

I want money (that's what I want)

I think that Apple has more than a couple of employees, so everyone doesn't work on just one thing.

[TheSpaz]

This is ridiculous! What a waste of time.

[longofest]

I agree that judging input based on facial expression sounds pretty out there...

[My wife off screen tells me that my dog just chewed up my iPhone]



[Computer opens 50 windows, or whatever signifies]

[tk421]

Originally Posted by GorillaPaws

Couldn't agree more.


How annoying is it going to be when you tell your computer to do something with your mouse, but the gaze detection software fails to register you looking at it. The machine would ignore your command to open a document or whatever and just hang out in standby mode. That's when the anger detection software better kick in to brace the system's hardware for impact.

Why would it use gaze detection when the mouse is giving input? The mouse is a pointer device. Gaze detection would work for things like multi-touch rotation where there is no pointer.

Personally, I think gaze detection is pretty cool.

In fact, all these are cool and worth exploring. Some of them seem iffy, but why not research them? Apple won't use some concepts, and the best ones will be incorporated.

Originally Posted by TheSpaz

This is ridiculous! What a waste of time.

Care to elaborate why?

Edit: Never mind. I didn't realize which post you were replying to.

[sjo]

Originally Posted by MacRumors

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)


Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.

none of those sound like years away from the market, there already is lots of implemented user interfaces using those technologies; slr's had auto-focusing based on where the user was looking in the 90's, samsung (i believe) has already cell phones that have the "finger identification" through build-in camera, voice recognition has been around since os/2. controlling handheld by movement is available in sonye and nokia cell phones. etc etc.

actually, what's left to patent?? doing all this stuff in iphone?

[The Tall One]

Very cool. I like the idea of the finger printing scan that come on those lenovo notebooks. Perhaps some James Bond style retina scan is in the works for access to your MacBook?

[Clayne]

Originally Posted by TheSpaz

This is ridiculous! What a waste of time.

Agreed. When I'm on the computer, I want to type. I want to do everything manual. Are we too lazy for that now?

Originally Posted by Rojo

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

Amongst everything else. I don't like it. Stick with touch-screens.

[IJ Reilly]

Originally Posted by Clayne

Agreed. When I'm on the computer, I want to type. I want to do everything manual. Are we too lazy for that now?

Absolutely. Many years ago I got too lazy to type at a command prompt. Fortunately Apple came out a with a computer that made that unnecessary. I am looking forward to becoming even lazier in the future.

[bobborries]

Microsoft has been incorporating these technologies into the Zune phone and coffee table. A fist print mashed on the control surface indicates the user is frustrated with the Vista operating system. A helpful cartoon character would appear and convince the user that the frustration is not Microsoft's fault. Facial expression recognition senses the user is now even more angry, soothing music automatically plays and once the user is in a blissful state, it asks the user if they would like to purchase the music they're listening to.

[8CoreWhore]

Facial recognition? It's bad enough they know I'm looking at porn - now they want to know what kind of porn?

[Chandler Adaway]

Wow. That is WAY cool.

It seems like all of the James Bond toys are becoming real!

Do any of you believe that the creator of the James Bond gadgets was inspiration for some of these things? Or any other fiction device from the past?

I just think all of this is really neat. I can already see, with this, how computing might be in the next 3-5 years, and I'm SO hoping that I get to experience all of that.

This is going to change the way people use computers forever. I can't imagine when I'm 60! (I'm 15 now).

Go Apple! I love you!

[ishopukisfake]

Im thinking also voice recognition is something they might be able to add to existing and older handsets, via firmware updates, which would be cool.

[iNassar]

wow this stuff is great. i love how apple inovates things

[ChrisA]

This has been my answer to those who say "We don't need 8 (or 16) core computers." Yes we will if the computer is to look at your face and fingers with a webcam and listen to voice commands all at once. Just doing that is far more compute intensive then rendering HD video.

[nagromme]

"The application 'BejeweledClone' has unexpectedly crashed.

Facial expression: -104 (End of Tether)

It appears that this crash may not be entirely unexpected after all. Would you like to send a report to Apple?"

Sweet!

[twoodcc]

i really would love to see some of this stuff, but i don't think it's coming anytime soon

[nagromme]

I think iSight-based finger detection might be very effective in detecting user frustration

[seashellz]

...and if you sneezed...?

[gnikcjack]

I can see millions of possible uses for this one. For example, the software can detect vasocongestion in a certain extremity, and subsequent tissue swelling. It can then minimize all immediate windows, and open new ones of the user's favorite porn sites. This could usher in a new era of adult-themed convenience! Although they would need to work out some kinks, as I'm sure this would lead to more than a few complications in the workplace.

Let's hope this doesn't fall through!!!!!

[ppc_michael]

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior.

Cue ClipIt. "It looks like you're angry. Would you like to..."

[MrCrowbar]

Originally Posted by ppc_michael

Cue ClipIt. "It looks like you're angry. Would you like to..."

God, I hated that thing. Made me wanna bend it out into a traight line and then bend parts up and down until the metal weakends and it just breaks off.

[t0mat0]

The What's the story morning glory app?

Whilst some isn't ready for prime time, a few of these are already inthe labs - Microsoft's work on voice, gaze recognition is established, but not so much via a webcam.

Finger recognition as in fingerprints has already been around, but actually working out which finger via a multitouch pad has been mentioned in previous patents. Working out which finger via a webcam is more difficult. Same with facial recognition. However, handheld device movement is already nearly to market in some guises.

I think the tech demos of webcams being able to identify objects, then use them as user interfaces (e.g. noticing a colored plastic wheel, and then being able to use it as a steering wheel for a racing game etc) already show there is a lot more out there, but it's not had OS integration really.

Interestingly, this patent is all about how to fuse multi-touch data - and then how secondary data can be combined with touch data - either refine the touch data, be interpreted in accordance to the touch data, or combine together to be a new command. So it's moving on from just multi-touch. Would Apple release some of the gestures prior to next summer's OS update, to give time to get acquainted with them?

Interesting to see they're patenting the future possiiblities and what they are.