This understates the difference.
The difference isn't just noticeable in intensive, iPad 2-optimized, 3D games like Infinity Blade, but throughout the device. Applications launch much faster. Switching between applications is now fast enough that it feels like the device has real multitasking.
Simple tasks like PDF rendering have been vastly improved. On the old iPad, iBooks would often take a fraction of a second after a page turn to render the new page (transitioning from a blurry pre-render to the actual rendered page). On iPad 2, it feels like the page is already rendered before you turn to it. This small improvement in the responsiveness of iBooks translates into an enormous improvement in the overall experience iBooks provides. This has been my experience throughout the device.
The iPad 2 is much, much faster and better at everything. No one who has seriously used both iPads would suggest otherwise. Whether this speed actually means much to the median user (for whom the iPad probably felt plenty responsive already) is less clear.