Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
I just picked up a refurb iPod Touch 5, which has more or less the same hardware as the 4s, and set it up identically with apps, so it matches my iPhone 5s. In real world use, I can't tell any difference. Maybe if I think about it really really hard I can imagine that it might be a little less smooth when scrolling? Keep the 4s.
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.