Maybe you just don't hear it? That happens to me sometimes, when I have the phone in my pocket and I'm in a loud space (like the local pub).
Of course, if you have the phone on silent, you don't hear it at all.
The only annoying thing I have is that sometimes I answer the call before the iAnnounce sentence is finished and I can still hear it in the background. But that's very rare.