I definitely agree that they won't completely push the iPad/iPhone 4 users to the curb. But I do think we'll start seeing some apps and games that come out that will take advantage of the hardware. They will still run on the older devices, but they will take a little longer to start the games, and the load screens will become a bit longer on those games. I think it'll be pretty similar to how the iPhone 3G -> 3GS conversion took place. They will be alright at first, but more and more games will begin to drift onto the app store that will be quite a bit slower on the iPhone 4. The one thing I would worry about though is that it may take a little less time this time around, with a device utilizing the A5 chip already being out for a decent amount of time.
Also, we will likely see many more apps like Real Racing 2 coming out, which support the iPhone 5 being the controller for a game on the TV. The games will still run on the iPhone 4, but they won't have those types of features. And that is just one feature we'll likely see a lot more of. I expect to see quite a few more little features that the developers come up with that will require the dual-core A5. The apps and games will run on the older devices, but you won't get to have the full experience.
But, will app developers still push the envelope like that if the uptake of the iPhone 5 isn't as great/record-setting as the iPhone 4? Because I can envision a likely scenario where a lot of users don't upgrade to the "5", especially if the external design doesn't change that much from the current iP4. The average user associates "big upgrade" with the external look of the device, not what's under the hood. Just look at all the people who continuously (incorrectly) state that the 3GS was a "minor" upgrade from the 3G, and how many people state that the iPad 2 was a "minor" upgrade from the iPad 1. Even though the processing and graphical power more than doubled in each instance. So I could very likely see the iPhone 5 not taking off quite as well as the iPhone 4 did, if the external design doesn't differ dramatically.