Also I find it funny how a lot of people can say "my Mac/PC has Core i7, it's much better than my old core 2 duo" but as soon as we touch tablets, same people will claim that specs don't matter.
It's not that they don't matter, it's that they are just one factor out of a number that matter.
Your example is a poor one. Comparing newer/older macs, faster CPU is going to give better performance since it's the same OS on both devices. But in the case of tablets you're comparing different OS, and a more efficient OS may make more difference in performance to the user than a faster CPU.