I suppose the real question to ask is what you're trying to do, to see if you'll actually notice a difference in the speed. Otherwise, if you want an "overall" type of pictures, various benchmarking software tends to collect scores on machines to offer a quantifiable way to compare speeds. For example, going by
GeekBench, here're some speed comparisons.
Retina iMac | late 2012 iMac
Single-core 32-bit: 3886 | 3185
Multi-core 32-bit: 14560 | 12229
Single-core 64-bit: 4351 | 3593 (Just for fun, note that the Late 2013 Mac Pro with 6 cores scores 3600 here)
Multi-core 64-bit: 16547 | 13908 (Just for fun, note that the Late 2013 Mac Pro with 12 cores scores a 32148 on this one)
In other words, there's a difference, but whether you'll really notice it will depend on what you're doing and how sensitive you are to those changes.