Beyond Multi-Touch: Voice, Gaze, Facial Expression Recognition

[MacRumors]

http://www.macrumors.com/images/macrumorsthreadlogo.gif (http://www.macrumors.com)

Apple's patent applications provide a glimpse into what might lie ahead in our future. For example, we saw an explosion of multi-touch patents in the year before the iPhone's release. And while many of these concepts may never come to market, they provide an interesting look into the direction of Apple's research.

One of the most recent patent applications from Apple is entitled Multitouch Data Fusion and is authored by Wayne Westerman and John Elias (formerly of Fingerworks (http://www.fingerworks.com/)). Westerman and Elias have been prolific publishers of multi-touch patent applications and likely helped establish the multi-touch technology behind Apple's iPhone. While many have since hoped to see some sort of advanced multi-touch interface for Apple's Macs, only limited multi-touch support has been included into Apple's notebooks.

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)

The patent application gives examples of how each of these could be used in conjunction with multi-touch to provide a better user experience. A few highlights are provided here:

Voice - Some tasks are described as being better suited for either voice recognition vs multi-touch. For example, if a user's task is to resize, rotate and change the color of an object on the screen, multi-touch would be best suited for the resize/rotation tasks but changing of color (or inserting text) may be better suited to a voice command.

Finger identification - the article suggests that using built in cameras (such as the iSight) with a swing mirror could provide an over the keyboard view of multi-touch gestures. This video information could be used to better distinguish which fingers are being used in which position. In conjunction with the touch input, this could be used to create more specific and accurate gestures.

Gaze - tracking where the user is looking could help pick windows or objects on the screen. Rather than resorting to moving a mouse pointer to the proper window, a user could simply direct his gaze at the particular window and then invoke a touch gesture.

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior. The example given is if a user is incorrectly trying to scroll a window using 3 fingers, instead of 2, the computer may be able to detect the frustration and either accept the faulty input or prompt the user.

Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.


Patent: http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=20080211766&OS=20080211766&RS=20080211766


Article Link (http://www.macrumors.com/2008/09/04/beyond-multi-touch-voice-gaze-facial-expression-recognition/)

[nick9191]

I certainly think Finger identification for security could arrive soon.

[gnikcjack]

I can see millions of possible uses for this one. For example, the software can detect vasocongestion in a certain extremity, and subsequent tissue swelling. It can then minimize all immediate windows, and open new ones of the user's favorite porn sites. This could usher in a new era of adult-themed convenience! Although they would need to work out some kinks, as I'm sure this would lead to more than a few complications in the workplace.

:apple::apple:Let's hope this doesn't fall through!!!!!:apple::apple:

[ppc_michael]

Facial expression - detecting frustration on the face of a user could provide help prompts or even alter input behavior.

Cue ClipIt. "It looks like you're angry. Would you like to..."

[MrCrowbar]

Cue ClipIt. "It looks like you're angry. Would you like to..."

God, I hated that thing. Made me wanna bend it out into a traight line and then bend parts up and down until the metal weakends and it just breaks off.

[t0mat0]

The What's the story morning glory app?

Whilst some isn't ready for prime time, a few of these are already inthe labs - Microsoft's work on voice, gaze recognition is established, but not so much via a webcam.

Finger recognition as in fingerprints has already been around, but actually working out which finger via a multitouch pad has been mentioned in previous patents. Working out which finger via a webcam is more difficult. Same with facial recognition. However, handheld device movement is already nearly to market in some guises.

I think the tech demos of webcams being able to identify objects, then use them as user interfaces (e.g. noticing a colored plastic wheel, and then being able to use it as a steering wheel for a racing game etc) already show there is a lot more out there, but it's not had OS integration really.

Interestingly, this patent is all about how to fuse multi-touch data - and then how secondary data can be combined with touch data - either refine the touch data, be interpreted in accordance to the touch data, or combine together to be a new command. So it's moving on from just multi-touch. Would Apple release some of the gestures prior to next summer's OS update, to give time to get acquainted with them?

Interesting to see they're patenting the future possiiblities and what they are.

[punkybadhip]

Why can't Apple just concentrate on fixing its OS, and providing quality made hardware before doing something new. :(

なぜなら！

Apple is now Micopple

I want money (that's what I want)

[Wayfarer]

Why can't Apple just concentrate on fixing its OS, and providing quality made hardware before doing something new. :(

なぜなら！

Apple is now Micopple

I want money (that's what I want)

Snow Leopard, fool!

[myrsky]

This is again a good excamle of the flaws in the patentability of inventions in US. Where's the invention in the notion that voice, facial expressions, gestures, gaze, body movement, flip-flops can or could be used to prompt actions from a computer? The invention lies in figuring out a technological solution to make it happen and that is also what should be patentable.

And no, I didn't read the actual patent application so I may be somewhat of here :D

[themoonisdown09]

Why can't Apple just concentrate on fixing its OS, and providing quality made hardware before doing something new. :(

なぜなら！

Apple is now Micopple

I want money (that's what I want)

I think that Apple has more than a couple of employees, so everyone doesn't work on just one thing.

[137489]

Everything sounds cool except for facial recognition. That idea is flawed. Why, how many people smile, frown, or squint when they are reading something... or even have a fustrated look when reading what someone wrote in an email that went to an unintended audience.

I can just see a lot of mis-readings on this.

I think shock detection could be good. I mean, people tend to slam their desks, shove their keyboard, flick the screen, throw their mouse when ticked off at the computer. Heck MS needs that for the fustration. Everyday my virtual PC's on my Windows work computers lock up or run slow.

Voice recognition, I am not so sure about. It may have some a long way since I tried it - but it had a hard time with some words and if the software crashed and you had to re-install it; you had to spend hours retraining it. My nephew (while he was in high school) had a comuter project he had to do to graduate. His demonstration was to have his XP machine (yes XP has been around that long - he is now 22) being totally operated by vioce command, using Dragon Naturally speaking. He said it was a fun project, but was fustrating as it took hours to train the thing. We are a long way from the old star trek episodes ("computer, 2 to beam up". "Computer, .....").

Right now I would like to see more work done on touch. On a laptop/tablet I can see not needing a mouse/trackpad anymore. A keyboard is still needed for long typing.

[mainstreetmark]

While I think there is definitely a world beyond multitouch, I think multitouch will be the norm for decades - outlasting the mouse.

What I don't want:
- Facial expression watcher? My facial expression is usually "Dammit, i'm still at work", so does that mean it would constantly be doing the computer equivalent of the famous girlfriend question: "what are you thinking... baby, what's wrong... why won't you talk to me... we never talk anymore...". And then it has a hardware failure.

- Voice recognition: Never ever ever. While it was kind of cute in the quadra 850av days with the knock knock joke, i would hate to work in a cubicle farm with a bunch of people talking to their computer all day. "change color... rotate... change color ... rotate... Corporate accounts payable, Nina speaking. Just a moment. ..."

No, the real trick is going to be multitouch with tactile feedback, so you can have functional keyboards that are task-specific.

[sjo]

In Multitouch Data Fusion, however, Westerman and Elias are already exploring the use other inputs to help improve and augment multi-touch interfaces. These include:

- Voice recognition
- Finger identification
- Gaze vector
- Facial expression
- Handheld device movement
- Biometrics (body temp, heart rate, skin impedance, pupil size)


Many of these technologies are likely years away from the market, but continue to provide an interesting peek into Apple's future.


none of those sound like years away from the market, there already is lots of implemented user interfaces using those technologies; slr's had auto-focusing based on where the user was looking in the 90's, samsung (i believe) has already cell phones that have the "finger identification" through build-in camera, voice recognition has been around since os/2. controlling handheld by movement is available in sonye and nokia cell phones. etc etc.

actually, what's left to patent?? doing all this stuff in iphone?

[bytethese]

Makes me think of a Minority Report meets Star Trek type computer. Sounds pretty damn powerful. :)

I think I like where Apple is going with this stuff, but please keep the option of typing for me whether on a keyboard or on a multitouch surface. Nothing more relaxing than being able to do my work while my music is playing and IM'ing my coworkers rather than talking on the phone. :)

[supercooled]

I certainly think Finger identification for security could arrive soon.

It behooves Apple to do more than just secured logins. It's too pedestrian even. No, I think Apple will unveil a remarkable product which incorporates all those things. The Gaze Vector appeals to me most of all.

I can't wait.

[Rojo]

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

[BigD58]

While I think there is definitely a world beyond multitouch, I think multitouch will be the norm for decades - outlasting the mouse.

What I don't want:
- Facial expression watcher? My facial expression is usually "Dammit, i'm still at work", so does that mean it would constantly be doing the computer equivalent of the famous girlfriend question: "what are you thinking... baby, what's wrong... why won't you talk to me... we never talk anymore...". And then it has a hardware failure.

- Voice recognition: Never ever ever. While it was kind of cute in the quadra 850av days with the knock knock joke, i would hate to work in a cubicle farm with a bunch of people talking to their computer all day. "change color... rotate... change color ... rotate... Corporate accounts payable, Nina speaking. Just a moment. ..."

No, the real trick is going to be multitouch with tactile feedback, so you can have functional keyboards that are task-specific.

Wow! I never thought of it like that before, that would get pretty annoying. But i think the coolest feature is the possibility of Gaze. If Gaze is created properly (which its apple so it would be amazing) then the interface on a Mac would change and become a whole new experience. I would love to see apple come through with this :D.

i love :apple:pple

[TheSpaz]

This is ridiculous! What a waste of time.

[longofest]

I agree that judging input based on facial expression sounds pretty out there...

[My wife off screen tells me that my dog just chewed up my iPhone]

:eek:

[Computer opens 50 windows, or whatever :eek: signifies]

[Brentnal]

Everything sounds cool except for facial recognition. That idea is flawed. Why, how many people smile, frown, or squint when they are reading something... or even have a fustrated look when reading what someone wrote in an email that went to an unintended audience.

I can just see a lot of mis-readings on this.



Apple would be the only company capable of identifying whether this type of technology will "work" through limitation such as this. If it doesn't work they won't use it.

Other companies would just use it for the "cool" factor, without thinking through the practicality.

[Bubba Satori]

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

http://bp3.blogger.com/_ih44QfUfNns/R1jmOGmBTFI/AAAAAAAACLw/-w6ZUmC6Xq4/s320/HAL+9000.JPG

And if it's as buggy as Leopard has been, it won't recognize your bleary eyed face in the morning and will instruct the toaster to murder you with a burnt slice of raisin cinnamon bread.

[RogueWarrior65]

Gaze tracking would be really cool. Voice recognition was always gimicky and goofy (and dopey and sneezy). The only way it would be cool is if you could say things like "Move this piece of ***** over here and delete that piece of *****." And Undo would have to be connected to the phrase "Oh, *****."

[jaybee32]

Rise of the Machines :eek:

Finally, somebody is taking notice of the success of nintendo's innovative interfacing concepts. I had my doubts before using a Wii - horrible memories of the duck hunt era - but it is surprisingly accurate and very receptive and it will be interesting what developers will be able to push forward now that Nintendo have proved that the world is ready to embrace more innovative and interactive interfaces. In igloos in Illanois it is icy. i. :o

[Xavier]

http://bp3.blogger.com/_ih44QfUfNns/R1jmOGmBTFI/AAAAAAAACLw/-w6ZUmC6Xq4/s320/HAL+9000.JPG

And if it's as buggy as Leopard has been, it won't recognize your bleary eyed face in the morning and will instruct the toaster to murder you with a burnt slice of raisin cinnamon bread.

I was about to say..

"..hello Dave.."

[The Tall One]

Very cool. I like the idea of the finger printing scan that come on those lenovo notebooks. Perhaps some James Bond style retina scan is in the works for access to your MacBook? :)

[Lesser Evets]

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

The basic technology has already been developed for year, just never implemented in common product or possession.

They are gearing up for sexy robots in 2040, and you won't be complaining THEN.

[deannnnn]

That's some cool stuff.
Makes you wonder if it will be standard in like 10 years from now.

[Wondercow]

Voice recognition, I am not so sure about. It may have some a long way since I tried it - but it had a hard time with some words and if the software crashed and you had to re-install it; you had to spend hours retraining it. My nephew (while he was in high school) had a comuter project he had to do to graduate. His demonstration was to have his XP machine (yes XP has been around that long - he is now 22) being totally operated by vioce command, using Dragon Naturally speaking. He said it was a fun project, but was fustrating as it took hours to train the thing. We are a long way from the old star trek episodes ("computer, 2 to beam up". "Computer, .....")

You do realize that Apple's had voice recognition built-in for about fifteen years, right? It won't do dictation but it will allow the user to interact with many aspects of the system and doesn't require any training. As to its effectivness, well, it could be better but if one's in a quiet environment it works fairly well (and always has IMO).

[PlaceofDis]

interesting advancements. yes creepy. but computers are becoming ingrained into life everywhere. and i do mean everywhere.

something like minority report where there are tailored ads etc could certainly happen. a smaller form of those computers used could happen too. but would still need to require less wide and flourishing gestures. but i could see it.

long way off all of this is though.

[Stately]

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

Haha, exactly . A fine line my friend, a fine line . :cool:

[ChrisA]

This has been my answer to those who say "We don't need 8 (or 16) core computers." Yes we will if the computer is to look at your face and fingers with a webcam and listen to voice commands all at once. Just doing that is far more compute intensive then rendering HD video.

[t0mat0]

interesting advancements. yes creepy. but computers are becoming ingrained into life everywhere. and i do mean everywhere.

something like minority report where there are tailored ads etc could certainly happen. a smaller form of those computers used could happen too. but would still need to require less wide and flourishing gestures. but i could see it.

long way off all of this is though.

Tailored ads already happens...

[Stately]

The basic technology has already been developed for year, just never implemented in common product or possession.

They are gearing up for sexy robots in 2040, and you won't be complaining THEN.

Haha, oh boy . I pray the Lord comes back before then . Goodness . :cool:

[IJ Reilly]

You do realize that Apple's had voice recognition built-in for about fifteen years, right? It won't do dictation but it will allow the user to interact with many aspects of the system and doesn't require any training. As to its effectivness, well, it could be better but if one's in a quiet environment it works fairly well (and always has IMO).

Right, but it's been even more than 15 years -- IIRC voice commands first appeared circa 1991. It worked pretty well, but who really found any use for it?

This is all very interesting stuff, reinventing the user interface, and I'd hope and expect Apple to be on the forefront of these developments. But I suspect we're a long way from any real revolutions on this front. It's got to work 100% of the time or it won't replace the antique keyboard and mouse.

[Doctor Q]

There are two parts to this technology:

1. Making it work at allApple or other companies will likely succeed in making most or all of these input techniques work. The question is how far they can go in what time frame.
2. Making it work for usI'd expect Apple, more than other software development firms, to be able to find practical, useful, and elegant ways to implement and adapt this technology into an operating system and applications on our desktop computers, notebook computers, tablets, perhipherals, media players, phones, and whatever else we buy from them.

[GorillaPaws]

It's got to work 100% of the time or it won't replace the antique keyboard and mouse.

Couldn't agree more.
http://images.macrumors.com/article/2008/09/04/053148-gazerec_400.png

How annoying is it going to be when you tell your computer to do something with your mouse, but the gaze detection software fails to register you looking at it. The machine would ignore your command to open a document or whatever and just hang out in standby mode. That's when the anger detection software better kick in to brace the system's hardware for impact.

How much system resources will these technologies be consuming to essentially do what we can fairly easily accomplish with a mouse and keyboard? Much of this strikes me as a solution looking for a problem; that said, I do think it's good that Apple is working hard at trying to dream up new ideas.

[Rojo]

Tailored ads already happens...

Yep, I get "tailored" ads all the time on places like Facebook, and it freaks me out! Often I'm like "how the hell does Facebook know I'd be interested in that?!?" -- and then my "oh, duh" moment of realization kicks in, and I realize it's due to what info I've written about myself. I'm often amazed on how smart it works too. For example, I get gay-related ads sometimes -- not because there's anything in my profile about being gay, but simply because I'm linked in a relationship with another same-sex Facebook user. I wonder how deep it goes sometimes. It's definitely creepy...

[tk421]

Couldn't agree more.
http://images.macrumors.com/article/2008/09/04/053148-gazerec_400.png

How annoying is it going to be when you tell your computer to do something with your mouse, but the gaze detection software fails to register you looking at it. The machine would ignore your command to open a document or whatever and just hang out in standby mode. That's when the anger detection software better kick in to brace the system's hardware for impact.

Why would it use gaze detection when the mouse is giving input? The mouse is a pointer device. Gaze detection would work for things like multi-touch rotation where there is no pointer.

Personally, I think gaze detection is pretty cool.

In fact, all these are cool and worth exploring. Some of them seem iffy, but why not research them? Apple won't use some concepts, and the best ones will be incorporated.

This is ridiculous! What a waste of time.

Care to elaborate why?

Edit: Never mind. I didn't realize which post you were replying to.

[illegallydead]

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

ha ha agreed. Also, what happens when you flip it off because it's REALLY pissing you off? "Goodbye! Come back when you calm down..." :rolleyes:


- Voice recognition: Never ever ever. While it was kind of cute in the quadra 850av days with the knock knock joke, i would hate to work in a cubicle farm with a bunch of people talking to their computer all day. "change color... rotate... change color ... rotate... Corporate accounts payable, Nina speaking. Just a moment. ..."

Also agreed with the speaking to the computer. It gets annoying, kinda awkward, and overall, not really worth it. It could have its uses, but personally, whilst the computer is still an inanimate machine and not some robot hellbent on ripping out my spleen, I would rather input through touch and typing, not carrying on a conversation with the fool thing. :D

[Clayne]

This is ridiculous! What a waste of time.

Agreed. When I'm on the computer, I want to type. I want to do everything manual. Are we too lazy for that now?

I think we're finally heading to the point where advancements in technology are starting to change from "cool!" to "creepy."

I find a computer watching where my gaze is or trying to discern my mood a really unsettling thing....

Amongst everything else. I don't like it. Stick with touch-screens.

[bobborries]

Microsoft has been incorporating these technologies into the Zune phone and coffee table. A fist print mashed on the control surface indicates the user is frustrated with the Vista operating system. A helpful cartoon character would appear and convince the user that the frustration is not Microsoft's fault. Facial expression recognition senses the user is now even more angry, soothing music automatically plays and once the user is in a blissful state, it asks the user if they would like to purchase the music they're listening to.

http://idisk.mac.com/bobborries/Public/MSClippy.jpg

[8CoreWhore]

Facial recognition? It's bad enough they know I'm looking at porn - now they want to know what kind of porn? :D

[nagromme]

"The application 'BejeweledClone' has unexpectedly crashed.

Facial expression: -104 (End of Tether)

It appears that this crash may not be entirely unexpected after all. Would you like to send a report to Apple?"

Sweet!

[twoodcc]

i really would love to see some of this stuff, but i don't think it's coming anytime soon

[nagromme]

I think iSight-based finger detection might be very effective in detecting user frustration :o

[seashellz]

...and if you sneezed...?

[IJ Reilly]

Agreed. When I'm on the computer, I want to type. I want to do everything manual. Are we too lazy for that now?

Absolutely. Many years ago I got too lazy to type at a command prompt. Fortunately Apple came out a with a computer that made that unnecessary. I am looking forward to becoming even lazier in the future.

[Chandler Adaway]

Wow. That is WAY cool.

It seems like all of the James Bond toys are becoming real!

Do any of you believe that the creator of the James Bond gadgets was inspiration for some of these things? Or any other fiction device from the past?

I just think all of this is really neat. I can already see, with this, how computing might be in the next 3-5 years, and I'm SO hoping that I get to experience all of that.

This is going to change the way people use computers forever. I can't imagine when I'm 60! (I'm 15 now).

Go Apple! I love you!

[ishopukisfake]

Im thinking also voice recognition is something they might be able to add to existing and older handsets, via firmware updates, which would be cool.

[MikeELL]

Yep, I get "tailored" ads all the time on places like Facebook, and it freaks me out! Often I'm like "how the hell does Facebook know I'd be interested in that?!?" -- and then my "oh, duh" moment of realization kicks in, and I realize it's due to what info I've written about myself. I'm often amazed on how smart it works too. For example, I get gay-related ads sometimes -- not because there's anything in my profile about being gay, but simply because I'm linked in a relationship with another same-sex Facebook user. I wonder how deep it goes sometimes. It's definitely creepy...

lol mate - that's a pretty big tell-tale. You might as well have it in neon.

I ended up removing my "single" status because all the "tailored" adds were started to p!ss me off. I mean can you imagine how little work you'd get done on the net if every banner add was actually interesting? I'm close to ADHD as it is.

[iNassar]

wow this stuff is great. i love how apple inovates things