I notice a pretty huge improvement in speed and responsiveness. It's my favorite thing about the iPad 2.
Granted, I'm a "power user". I do a lot of multitasking (switching back and forth frequently), loading of graphic-heavy websites, and productivity apps. I do a fair amount of productive work on my iPad. For comparison, my work desktop PC is a dual core with two large LCD screens. That's how I roll...
The iPad 1 was "no slouch", but there were a lot of little delays and stutters that added up over time. In many ways I compare the iPad 2 upgrade to moving from the iPhone 3G to the 3GS. The 3GS wasn't radically faster, but it was just enough to eliminate all the annoying little delays that marred the experience. Not that the 3G was a bad phone -- it was pretty great for its time -- but the smoothness of the 3GS really put a polish on the whole thing.
The iPad 2 is the same deal. But the speed improvement you'll see has everything to do with how you use the device.
If you are a casual user and take your time moving between apps and do a lot of book reading and movie watching, you'll notice almost no difference.
If, however, you use it in a way that taxes the CPU, graphics, and RAM, like a lot of app switching, 3D games, and browsing complex websites, you will see a big difference.
Bottom line, if you have an iPad 1 and have never been bothered by lag or performance, don't upgrade for the speed. You're fine. But if you have been annoyed by lag (I was), then definitely do consider upgrading.