Glad I did. The display itself is well worth it if you do a lot of reading. Text is incredibly sharp.
I notice a definite improvement in performance when switching apps or even loading many tabs and/or images in Safari. I don't have the tools to actually measure the usage, but I have no doubt it's due to the additional 512MB of RAM. I had to drop back to the iPad 1 for the week and a half between the launch and receiving the iPad 3. The performance hit when switching apps was very obvious.
The graphics performance is also a great improvement. Think of what's required to drive this many pixels on a desktop or portable. Personally, I find it pretty amazing. I haven't tried any of the updated retina games, yet, but I'm sure I'll be impressed (apps such as Real Racing HD 2).
Though I probably won't use the rear camera, it's at least useable. The old saying "the best camera you own is the one with you". Typically, I have an iPhone 4S sitting next to me, so I would lean towards using that. The new iPhoto's beaming feature makes it easy to get the images from the iPhone to iPad. It's much easier to do the editing on the iPad.
If you can afford the upgrade, I don't think you'd be disappointed. App performance is similar, but as I mentioned, the extra ram does speed some things up.
Edit: misread the last of your OP and gave the reasons I was glad I did upgrade, instead of not.