Yeah the 2GB of RAM will make a huge difference, but also having the A8 in a 4" device will also improve performance. I did a little calculation with the Geekbench scores and with the A8 in the iPod Touch 6th gen being clocked over 20% lower than the one in the iPhone 6, it scores just under 15% less; so the A8 in a 4" iPhone (clocked the same as the iP6) should outscore the iPhone 6 by maybe 8%. I know it isn't much but its something. Graphics would be better too. But man what a beast the A9 would be in it!!!
Anything better than the 5S in terms of ram would be good. I mean not that the 5S is bad, but going from my 5 to a 5S wouldn't be that much of an upgrade...