iSight and Microphone Patent

[MacRumors]

In March 2003 a couple of new patents (http://www.macrumors.com/pages/2003/03/20030302015701.shtml) were awarded to Apple. The most descriptive patent filed in July 2002 described a method for to filtering out background noise from two independent microphones -- allowing for improved Speech recognition -- spawning speculation of renewed speech efforts by Apple as well as assistive devices built in to future displays or Macs.

Apple's iSight (http://www.apple.com/isight/) appears to incorporate at least some of this technology to improve audio capture:

iSight includes a built-in, dual element noise-suppressing microphone that delivers crystal clear audio. The microphone takes in the sound from both microphone elements, then determines which sounds are essential to the conversation with an algorithm that filters out extraneous noise.

[Kwyjibo]

cool my first submission that went thru

[strider42]

it would be cool if apple could find some companies to license such technology, or other technologies they have patents on. I wonder if apple could add additional revenue streams by leveraging their intellicetual properties. I'm sure this and a lot of other stuff they have have loads of applications, even outside of the computing realm. It would be a way to expand the business without having to go outside what they know. I don't know if this technology in particular would work for such a thing, I'm just trying to make a point. instead of simply preventing others from using their ideas, they could sell them the right to do so.

[rvernout]

These are only patent applications yet (i.e. not g r a n t e d patents).

Since 2000 the USPTO is publishing US patent applications in order to comply with the rest of the world. Patent applications are published 18 months after the earliest filing date anywhere in the world, unless the applicant requests earlier publication (which was apparently the case with both of these). Only if no corresponding patent applications are filed outside the US the applicant can request that the application is not to be published (until grant).

[iJon]

well ive been impressed with it so far. to an extent. sound quality is great. video is ok, be better if everyone had t3 lines. but ive realized even if video isnt great, i having such a great time talking to a friend, that i forget about it.

iJon

[rvernout]

I wonder whether the two mics in the iSight really contribute anything to suppress noise. According to the specification of the speech recognition patent application the mics should be a substantial distance apart (f.i. on both sides of the screen) which sounds logical.

Any sound engineers around? :confused:

[Vonnie]

Originally posted by strider42
it would be cool if apple could find some companies to license such technology, or other technologies they have patents on. I wonder if apple could add additional revenue streams by leveraging their intellicetual properties. I'm sure this and a lot of other stuff they have have loads of applications, even outside of the computing realm. It would be a way to expand the business without having to go outside what they know. I don't know if this technology in particular would work for such a thing, I'm just trying to make a point. instead of simply preventing others from using their ideas, they could sell them the right to do so.

Patents don't work that way for big corporations. Microsoft infringes on a bunch of patents from Apple, but Apple also infringes on a bunch of patents from Microsoft. So, if Apple starts asking money for it, Microsoft will just start asking money too. I mean, IBM has a patent on when you press enter in a spreadsheet, you go to the next cell. These are the sort of patents I'm talking about..

Patents are just meant to keep little competitors little. The companies that make money from patents, are usually very focused on one thing. (a specific algorithm for compression of video for example)

[rvernout]

Patents don't work that way for big corporations. Microsoft infringes on a bunch of patents from Apple, but Apple also infringes on a bunch of patents from Microsoft. So, if Apple starts asking money for it, Microsoft will just start asking money too. I mean, IBM has a patent on when you press enter in a spreadsheet, you go to the next cell. These are the sort of patents I'm talking about..

Not very likely that any infringement takes place. More likely that the patented features are licensed for a reasonable fee, or that they exchange licenses at no costs (cross-licensing).

[Wardofsky]

I remember now.

They came out and we all thought this was going to be new displays with 2 mics at each end.

Now the iSight has it.

[Freg3000]

It all comes together now. Even the rotational aspects of the original rumor now makes sense. :)

[strider42]

Originally posted by Vonnie
Patents don't work that way for big corporations. Microsoft infringes on a bunch of patents from Apple, but Apple also infringes on a bunch of patents from Microsoft. So, if Apple starts asking money for it, Microsoft will just start asking money too. I mean, IBM has a patent on when you press enter in a spreadsheet, you go to the next cell. These are the sort of patents I'm talking about..

Patents are just meant to keep little competitors little. The companies that make money from patents, are usually very focused on one thing. (a specific algorithm for compression of video for example)


well, apple licenses 1 click shopping from amazon. thats a stupid patent, but amazon was lucky enough that apple wanted to use it without getting into a whole thing about it. companies license patents from each other all the time. And apple has plenty of technology they could probably license if they tried.

So I totally disagree with your statement about patents just being meant to keep little competitors in line. patents protect intellictual property, no matter what the size of the people using it. every technology that has ever been licensed pretty much is patented. microsoft licenses stuff, IBM does, oracle, apple, everyone licenses patented technology when its in their interests not to develop something on their own or when someone has exactly what they want (one click shopping)

[MarksEvilTwin]

Removed

[mymemory]

Originally posted by rvernout
I wonder whether the two mics in the iSight really contribute anything to suppress noise. According to the specification of the speech recognition patent application the mics should be a substantial distance apart (f.i. on both sides of the screen) which sounds logical.

Any sound engineers around? :confused:

I'm one.

There are several ways to cancel background noise, I would be surprise if a computer company creates a new way while the audio industry have been working on that for at list 60 years or more.

There are several microphones out there with all kind of technologies to cancel ambient noise, that very new thing Apple is looking for is called "Polar Patterns" and it have been used successfully since WW2 when airpilots had to communicate from their planes to other pilots or to the ground, How do you think they cancelled the ambient noise in their big propeller planes?

Any way, the problem that Apple faces is not canceling any noise to isolate a specific audio signal, the problem is that the users can not place that microphone any where in the room or even in front of the computer because to isolate a signal (in this case human voice) you need to "caustically" take voice and noise and change the phase of the noise to cancelled itself leaving the voice intact, that is something that many microphones do but each microphone have their own specifications the users have to fallow to achieve that, other wise it won't work. There is a microphone for an specific environment and use. The problem Apple is facing is that the same mic have to work in my room and in a noisy office or who knows where else placed at many different distances from the subject and work. Good luck.

The only way I see is to have a pure reference of the user voice inside the computer to campared later, but still computers are not good isolating signals, for them everything is just one big wave form, so, you will need to isolated acustically from the microphone. No wonder that technology is not in full use today. It is a big pain to develop.

Capish?

[rvernout]

I suppose that the mics in the iSight are located behind each other (seen from the user), and that signals that show no or minimal phase shift are filtered out, as they are coming from aside, above or beneath. It would however pass noise coming from the back, opposite the user.

In the speech recognition system the mics are located next to each other (seen from the speaker), and I suppose signals that do show a phase difference are filtered out, as they are coming from aside. It would however also pass noise coming from above, beneath or behind.

Or am I on the wrong track? :confused:

[mrdeep]

Woah, I think they really might have speech recognition stuff planned.

They have microphones that lend themselves to speech recognition, and they its on a camera that they could use for lip reading.

[e-coli]

I remeber when everyone thought this was going to be an Apple branded digital camera. And someone even jokingly called it the iSight.

Buy that man a beer. ;)

[Thom_Edwards]

Originally posted by strider42
it would be cool if apple could find some companies to license such technology, or other technologies they have patents on. I wonder if apple could add additional revenue streams by leveraging their intellicetual properties. I'm sure this and a lot of other stuff they have have loads of applications, even outside of the computing realm. It would be a way to expand the business without having to go outside what they know. I don't know if this technology in particular would work for such a thing, I'm just trying to make a point. instead of simply preventing others from using their ideas, they could sell them the right to do so.

a friend of mine was working for a startup doing similar tech for cellphones a year or so ago. (the economy ate up the company...) they were trying to filter out all frequencies except those that the human voice used. of course it was *way* more in depth than just that, but that is it in a nutshell. they weren't using two mics--only algorithms and a dsp chip. i got to tour the office and such there in san fran. really fascinating to see and hear them talk about it.

too bad it never worked out for them and my friend. they might could have licensed this to a lot of people, including apple (even though that is mainly just speculative and wishful thinking). so maybe apple branching out and working with their cell phone buddies could pay off in more than just iSync...

[e-coli]

Originally posted by mrdeep
Woah, I think they really might have speech recognition stuff planned.

They have microphones that lend themselves to speech recognition, and they its on a camera that they could use for lip reading.

That would be super nice if they could actually get it to work. I've yet to see a computer voice recognition program that works well. Or at least works well enough to ditch the mouse.

[zuggerat]

hah...sorry mr. gates not this time

[BaghdadBob]

I'm not sure why they're investing so much into trying to get the recognition to work from a mic that is 18-24 inches away. Wouldn't a bluetooth headset be easier?

I really hope they're onto something, but it just seems like a waste.

Anyway, as far as good speech recognition goes, it's out there, my fianceé's friend has been using it for years, and she finds it highly effective. It's used as a total substitute for typing, an accomodation her employer had to make due to her developing carpel (sp?) tunnel syndrome.

Still, it requires training, and I don't know if anyone is going to develop really effective speech recognition software anytime soon that doesn't.

[neutrino23]

How might this be different from a "shotgun" microphone?

I would guess that both microphones are somewhat directional and that one points roughly towards the speaker and one doesn't. I've done background suppression work in other areas. If I were trying to do it here I think I would have the software monitor both microphones for a long time, find the quiet times when both microphones pretty much track each other and use that to calibrate how much of the background signal to subract from the voice signal.

A head set would be OK, but it is not as casual to use as just talking towards the screen. Actually, you could use a headset with iChat if you so desired.

[druggedonions]

Originally posted by rvernout
I suppose that the mics in the iSight are located behind each other (seen from the user), and that signals that show no or minimal phase shift are filtered out, as they are coming from aside, above or beneath. It would however pass noise coming from the back, opposite the user.

In the speech recognition system the mics are located next to each other (seen from the speaker), and I suppose signals that do show a phase difference are filtered out, as they are coming from aside. It would however also pass noise coming from above, beneath or behind.

Or am I on the wrong track? :confused:

Not on the wrong track at all.
If you are sitting comfortably I will begin.

If you have a stereo signal and reverse the phase of one side then sum them (play them back through one speaker or them both through two speakers) one will cancel the other out. So what appears in both the original signals will have gone. (This is how some Kareokee machines work)

So working on this principle having two matched mics, and reverseing the phase of one of them, suming them together you are left the surrounding (ambient noise). If you then reverse the phase of this and sum it with the sum of the original signals you will cancel out the ambient noise. Et voila. You're left with a signal that is free from background noise.
edited: to try and make this post shorter

[pianojoe]

Originally posted by druggedonions

So working on this principle having two matched mics, and reverseing the phase of one of them, suming them together you are left the surrounding (ambient noise). If you then reverse the phase of this and sum it with the sum of the original signals you will cancel out the ambient noise. Et voila. You're left with a signal that is free from background noise.[/audiophile]
edited: to try and make this post shorter

Brilliant. But I feel it's much easier than that:

Take two mics pointing in slightly different positions, find criteria (and an algorithm) to analyze the signals for background noise, and take the signal from the mike that has less background noise.

[tizza]

Originally posted by pianojoe
Brilliant. But I feel it's much easier than that:

Take two mics pointing in slightly different positions, find criteria (and an algorithm) to analyze the signals for background noise, and take the signal from the mike that has less background noise.

This approach assumes you have highly directional microphones and that indeed your background noise is highly localized. Normally background noise, such as that of computer fans or airconditioning etc, is pretty omni-directional, so simply taking the mic with the least noise won't achieve much. That is why, the best approaches attempt to charcaterize the noise component and attempt to subtract it from the original signal.

[rainman::|:|]

Originally posted by pianojoe
Brilliant. But I feel it's much easier than that:

Take two mics pointing in slightly different positions, find criteria (and an algorithm) to analyze the signals for background noise, and take the signal from the mike that has less background noise.

yes but you're not sampling and analyzing both inputs so you won't get the benefit of phase coupling at all... which is presumably how this mic works... by accepting sounds only generated from a specific direction...

then the software probably uses a lossy voice compressor to take out a lot of unnecessary sounds... leaving you with vocal bands directly in front of the microphone. Not audiophile quality background cancellation, but a lot better than normal (and they're not selling this feature too hard either, so it's kind of a perk)

pnw

[scan300]

If you read the patent application, noise suppression is only part of the picture. In earlier versions of speech recognition the computer would isolate the spectral pattern of the voice and use that pattern for analyses. The problem was that when a person moved their head their spectral pattern would change.

The approach in this patent uses a phoneme database at the front-end of a speech recognition engine connected to the monitor (Acoustic Model Selector ACM). The ACM selects sounds (after noise suppression is performed ) which best match regular speech sounds (vowels and consonants). They are moved to a back-end process which match the phoneme to a language and then works out the command.

The combination of multiple mics and software algorithms help in 'beam formation' of the voice ie selecting the best signal to noise ratio, which is then used to eliminate noise. The characteristics of the set-up (mic placement, type, axis etc) are stored in software and can be used by the speech recognition software to account for spectral characteristics. There also seem to be a few combinations of best mic placement as well as a 4 mic option depending on the display characteristics.

Beam formation is different to phase inversion as a noise reduction technique. My version on how this works is that the best signal from both highly directional mics are matched to form a beam, while other signals, which are weaker are suppressed.

A phase inversion technique requires an omni directional mic to monitor background noise and another directional mic for the voice (Or a polar set-up). The background noise would be phase inverted and added to the voice signal to remove the noise. This is really only a good technique if the noise is ambient ie, all around you. Eg reflected factory noise. I'm not sure this patent makes use of this technique.

[Steak]

This technology appears to be phase cancellation, as many have pointed to. Problem is, it rarely works, unless inspecialised environments. When you fly on a Boeing 747, engine noise is pumped into the cabin, reversed phasse, to lower noise. Same with a pilots headphones. here is the problem: If you are cancelling the ambient noise, you are also cancelling the voice itself. I don't understand how this would IMPROVE recognition.. The Voice would be just as skewed and cancelled as the noise. If they found a way to only pick out sounds between the noise floor, and about -30dB, it might work, but that is very difficult.
I have tried to build a noise cancelling system for my computer, using this same idea, along with surface-mount mics on computewr hardware and fan. It did help, and cut down noise, but if I spoke, my voice would also be phasey.
Unless apple somehow passed up the study of great audio engineers for the last 100 years (this idea has been around since the discovery of electricity) I don't think it would work.
In a specially designed room, with $2000 minimum altered frequency mics, you will get about a 50% improvement in ambient noise. Not really worth it....

[iLilana]

I was going to say something brilliant but someone else alread did.

[frinky23]

Originally posted by BaghdadBob
Wouldn't a bluetooth headset be easier?

I don't think Apple wants to require you to buy additional hardware to use this...

Anyway, Bluetooth really is limited as far as voice quality is concerned. Bluetooth headsets for cell phones are just fine, but if you want higher quality sound Bluetooth just isn't going to have the bandwidth for it.

[ouketii]

hehe apple makes something, then patents the crap out of it... good stuff.

[Pete_Hoover]

I hope they take this technology and develope better speech recognition software. Something that actaully works, and is practical.

[ClimbingTheLog]

Originally posted by e-coli
I've yet to see a computer voice recognition program that works well. Or at least works well enough to ditch the mouse.

That's why the guy above mentioned lip reading. Far more accurate than sound wave decomposition.