Exactly. In fact, the person next to you helps your driving since he will react if something is wrong.
What has been shown by research is that having a conversation over the phone exhausts our ability to process information. For instance, our eyes move a lot less and the result is a kind of tunnel vision. Likewise, our ability to process unusual information quickly plummets - for instance, it will take very long to identify something on the road that doesn't look as what you expect on the road (an animal, someone in a wheelchair). As a result, the driver can be surprised, doesn't see things in the periphery and reacts slowly.
This is the real danger. An headset or anything doesn't help that at all.