OP, I live in a city where I get AT&T's LTE, but sadly it doesn't cover were I work. So, instead of LTE I get on AT&T's 4G network which gets me average speeds of 7 MBs and the max being 12 MBs, but it never bothers me. I can do everything happily, and once I get home with my LTE coverage where I can get up to 28 MBs down, I don't really see any difference in speed while I'm using my phone unless I'm watching a video. And with videos, the only difference is that they just buffer faster, but I don't have that problem with AT&T's 4G network here at work, which is comparible to what you're getting on Verizon.
Since you have unlimited data on Verizon, and they will not cap you, I highly suggest you stay with Verizon and enjoy their great LTE coverage. Those AT&T LTE speeds your friends are pulling mean nothing in the real world when you're doing the most basic things like web browsing, facebooking, tweeting, etc.