Interesting. The person it happens to with most for me is when talking to a friend with an iPhone 4 (on Verizon, I'm on ATT).
To the other reply, yeah I haven't figured out any pattern either. I try hanging up and calling back; restarting...sometimes it's just there and won't go away...other times sounds great. Frustrating.