Mate, they done more than "just add a new connector". The A6X destroys the A5X in terms of raw performance power. While Geekbench scores measure potential, a difference of 1000 points isn't to be taken lightly.
As you rightly say though, the performence on iPad models in general when it comes to iOS 7 is concerning. It's certainly not 100% optimised but hopefully 7.0.1 will address that.
Normal users couldn't care less about benchmarks on a processor though.
I mean, its not like the iPad 3 was sluggish. All they really do see are the tangibles (which happen to include a new connector)