If you have no other devices that use the Lightning connector, I'd say save the $100 and go with the 3. This is coming from a guy who just upgraded from the 3 to a 4. I did it mostly because I got $60 off plus $30 in rewards at a Bestbuy open box deal (but very curiously the thing was brand new). I figure I should get $450 for my mint iPad 3 with accessories so this upgrade will cost me ~$100.
It kind of makes sense for me since now I can completely get rid of the 30-pin cable, and now I have an extra Lightning cable and a 12W power adapter to charge my Mini and iPhone 5, both of which I was going to buy anyway which would have cost me $40. Performance wise I do notice that the browser is a bit more peppy, and I have an iPhoto journal with 400+ photos that shows up faster.
The combination of $90 in savings, consolidating to the Lightning cable, and the performance increase made it all worth it for me. This is my justification to myself