To be fair, a major issue you didn't list (and probably the other guy's point) is being in public with other voices around. The voice software can't exactly control the microphone to block other voices. Not yet, anyway. And it will never be able to cope with noise that simply drowns out parts of your speech.
That's the smallest of challenges that needed to be overcome. Even Apple's relatively rudimentary speech recognition has little problem hearing my voice when I ask it to play tracks. Hierarchy of voices based on distances from the mic is pretty easy to determine now -- even using the iPhone's main mic, easier still with the noise reduction mic in play and trivial if you're using a headset.