I've now had 3 iPhone 4's and all of them have had the same issue you described. Last week the Genius told me that he gets about 10 people a week with the same issue (its not many, but enough them to consider it an issue) and he said normally when he does an exchange, it fixes the problem, but that its a hit or miss situation and no guarantees. This is now my third time doing an exchange (factory new, not refurb) and no fix.
I believe its an issue with the mic and not reception, because I don't have this issue when I use Headphones or a Bluetooth.