Definitely a noticeable difference in many locations. When I'm in my home town with my AT&T I only get 2-3Mbps, which is what Verizon will get AT BEST - and it's quite a bit more sluggish on web browsing than when I go into the city where I get 5-6Mbps. Not unusable, but noticeable.
Anxiously awaiting my backordered 4S so I can see what HSPA+ will do for me in the city
It really depends on your signal strength and what you are doing. On average, AT&T's download speed is twice that of Verizon's, but I've seen cases where it's 10-15 times faster.
If you load a simple web page or app, then you might not notice a different, but if you download a song or app that takes 5 minutes on Verizon but only 1-2 minutes on AT&T, I would say that's substantial.