Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Researchers Demonstrate Subliminal Smart Device Commands That Have Potential for Malicious Attacks
- Thread starter MacRumors
- Start date
- Sort by reaction score
Wait, so if I get a dog whistle, Siri will finally understand me?
Balancing my bike at the traffic lights, looking cool AF triple tap AirPod "Set volume to 80%"
..."William Oliver Stone (born September 15, 1946) is an American writer and filmmaker"
Sometimes I think Siri just wants me to die single 🙁
Balancing my bike at the traffic lights, looking cool AF triple tap AirPod "Set volume to 80%"
..."William Oliver Stone (born September 15, 1946) is an American writer and filmmaker"
Sometimes I think Siri just wants me to die single 🙁
Superhai
macrumors 6502a
There are references to different research studies, one look at inaudible commands, and the other to embedded into other sounds or noise. I think the inaudible is easier to avoid, unless it uses effects of lower harmonics reacting to the actual body where the mic sits, to be able to produce local vibrations in the human voice band.
There is an interesting research paper: https://www.usenix.org/node/191969
M2M
macrumors 6502
One of the most useful comments I’ve read in a long time, if not ever on MacRumors! Thanks for this!
groovyd
Suspended
... an interesting consideration here is that this 'hack' most likely could never occur from any of the sound making devices in your home currently as most if not all are already bandwidth limited by either a low sample rate or the speaker's capable frequency response to under 20khz (within normal hearing range). In other words for this attack to actually occur requires a very special sound reproduction system you don't just find in your average consumer device. Now perhaps some of the most esoteric audiophiles with the $10k stereo systems they might need to worry 😉
Sure, but human voice frequencies don't necessarily match human hearing frequencies. Meaning the human voice could quite possibly produce frequencies that human ears cannot hear. Even if evolution will have tended to get rid of those non-useful frequencies (though who knows, some of those frequencies might have useful when interacting with animals), some might have been impractical to get rid of (ie, human vocal cords optimised for human hearing might still produce other frequencies as a by-product). Plus there is the whole ageing aspect in that older people cannot hear higher frequencies.
[doublepost=1526122003][/doublepost]
Sure, because all those people with voices outside the human hearing range can communicate so well with other people already.
[doublepost=1526122312][/doublepost]
Speech recognition is one step, but I got a Siri response with:
OK, I found this on the web for 'Set a four minute timer':
fairuz
macrumors 68020
ML is soft logic, very hard to alter for security purposes. I'll bet they didn't have any ultrasonic training samples either!
[doublepost=1526147787][/doublepost]
[doublepost=1526147787][/doublepost]
That was a thing? Oh man, maybe my ears were always old. Also, that sounds like torture.
groovyd
Suspended
It's just considered good design in any sampling system, a fundamental requirement actually. I believe they are limiting the frequency range with regards to the Nyquist Theorem perhaps just not strong enough by using a low order filter typically good enough for general purpose audio processing just not good enough for 'security' purposes. As a privacy and security centric company we can rest assured they will step up their game in the next design. I'd bet a good bit the next round of phones won't have this problem. No doubt a team there is assigned to fix this moving forward as we speak.
Researchers in the United States and China have been performing tests in an effort to demonstrate that "hidden" commands, or those undetectable to human ears, can reach AI assistants like Siri and force them to perform actions their owners never intended. The research was highlighted in a piece today by The New York Times, suggesting that these subliminal commands can dial phone numbers, open websites, and more potentially malicious actions if placed in the wrong hands.
A group of students from the University of California, Berkeley and Georgetown University published a research paper this month, stating that they could embed commands into music recordings or spoken text. When played near an Amazon Echo or Apple iPhone, a person would just hear the song or someone speaking, while Siri and Alexa "might hear an instruction to add something to your shopping list." Or, more dangerous, unlock doors, wire money from your bank, and purchase items online.
The method by which the students were able to accomplish the hidden commands shouldn't be a concern for the public at large, but one of the paper's authors, Nicholas Carlini, believes malicious parties could already be making inroads with similar technology.
Last year, researchers based at Princeton University and Zheijiang University in China performed similar tests, demonstrating that AI assistants could be activated through frequencies not heard by humans. In a technique dubbed "DolphinAttack," the researchers built a transmitter to send the hidden command that dialed a specific phone number, while other tests took pictures and sent text messages. DolphinAttack is said to be limited in terms of range, however, since it "must be close to the receiving device."
In yet another set of research, a group at the University of Illinois at Urbana-Champaign proved this range limitation could be increased, showing off commands received from 25 feet away. For the most recent group of researchers from Berkeley, Carlini told The New York Times that he was "confident" his team would soon be able to deliver successful commands "against any smart device system on the market." He said the group wants to prove to companies that this flaw is a potential problem, "and then hope that other people will say, 'O.K. this is possible, now let's try and fix it.'"
For security purposes, Apple is stringent with certain HomeKit-related Siri commands, locking them behind device passcodes whenever users have passcodes enabled. For example, if you want to unlock your front door with a connected smart lock, you can ask Siri to do so, but you'll have to enter your passcode on an iPhone or iPad after issuing the command. The HomePod, on the other hand, purposefully lacks this functionality.
Article Link: Researchers Demonstrate Subliminal Smart Device Commands That Have Potential for Malicious Attacks
If they prevent ultrasound from being heard by a smartphone that is a major blow to indoor GPS apps such as Sonitor’s Forkbeard app. Hope the fix is more elegant than that.
[doublepost=1526269608][/doublepost]
This is not true at all. Ultrasound is a major player in indoor GPS.
Would it be possible for them to split the signal path? First using a high-pass filter that only looks at the ultrasound range and if the signal contains indoor GPS information process as such? And then using a low-pass filter to only look at the range below ultrasound to process for any voice commands?
The ultrasound contains a 10 bit binary string so it should be easy to allow this. Also it can be limited at the app level. Seems more of a fix to Siri than to the API.
jagooch
macrumors 6502a
This is pretty old news. We discussed this frequently over the last 4 years as it was discovered soon after the Amazon Echo was release. It’s education ( for them ) that some students decided to repeat the research already done by others, and it’s definitly something that Smart Speaker manufaturers should keep in mind when designing products .