If by "imagination" you mean "I've actually experienced this", then yes. My connection is 23 down/ 3 up and I'm using the G.711u codec through a Linksys/Cisco PAP2T. Without QoS, drop outs are introduced into the VOIP connection as latency is crucial.
Since that PAP2T has numerous issues it is most likely this one is acting up. Using normal VoIP phones will give you better performance. Most other VoIP issues are being caused by slow internet connections, misconfigurations, using wireless (latency too high, slow connections, etc.), problems with the telco, hardware defect, etc. QoS will not resolve any of these problems although it can make the situation a little bit better with certain slow internet connections.
Do you run VOIP on an airport extreme?
On a slower device: the Linksys WRT54GL and I've used it with dd-wrt and tomato (which is what I'm currently using). There were no issues with 3 phones and 1 fax when streaming audio and surfing the web in a business setting. I've had it with two connections: 20 Mbit adsl which unfortunately does not go past the 12 Mbit downstream (and about 800~1000 kbit upstream) and later on a 10 Mbit glass fibre connection (10 up AND down).
Users did not notice any differences. The only thing they complained about was 1 phone not working properly (it was broken and replaced) and the fax (fax with VoIP is troublesome). I once used QoS and it had a very negative effect on VoIP and the connection speed. The device was simply too slow causing network slowdowns which in turn caused problems with VoIP and anything else.
People who need QoS are people with large and complex networks and/or very slow internet connections (do not overlook upload speed!). If you want to use QoS you also need a very fast device since QoS will have a very big performance hit on the device. You also have to have some networking knowledge since you need to configure QoS. If you don't have good knowledge you can achieve the exact opposite: things becoming even slower. None of these things can be found in the average consumer home which is why it is in professional gear and not in most consumer gear.
