I upgraded, and I can honestly say it's not worth it. I'd take my iPad 1 with it's perfect display back over the multiple iPad 2's I've had with faulty displays. There's really not much of a noticable difference between the two.
I don't know what you are exactly doing with your iPads, but for me the difference is very big. No lags, no slowdowns, switching between apps is really fast, Safari much faster than before and Games look and play better. I am more than happy.
How many apps do you have installed on your iPad? That might help to better understand how you are using it. I got 90 apps installed and I use more than 15 every day.