The 320M has some weird cult following, but all of the assertions of it being better than HD3000/4000/5000 for gaming or other performance related metrics are just false. HDx000 destroys it in almost every category.
You'd think so, but I found quite the contrary.
I owned both the 320M and HD4000 MBAs simultaneously, and even with an extra 2gb of RAM, the HD4000 machine ran hugely behind the 320M in actual gaming performance, especially with the detail / resolution cranked up.
I mostly play older titles so it's easy to run at higher settings. I found that the more detailed the scenes got, the greater advantage the 320M had. Complex scenes would hit 10-15fps and become unplayable on the HD4000 while the 320M would push through them at 25-35fps.
Basically the HD4000 would have FPS spikes in simpler scenes but then drown as soon as the going got too heavy.
I ran clean OSX installs on both machines, and honestly thought I was missing something until I read a few matching stories from others that had actually owned both machines.
Why did I buy the newer model? Because like you I assumed that significantly higher benchmark scores meant significantly higher gaming performance. The reality was the opposite. I suspect the i5 is artificially boosting the scores compared to the c2d.
The HD5000 might have finally matched or slightly exceeded the old 320M, but I'd honestly sooner buy an Asus ultrabook (nVidia or Iris) instead of the current MBA that has graphics performance dating back three years.
I don't think you can even call the MBA an ultrabook anymore (there is little 'ultra' about it apart from battery life), which is a real shame because it essentially coined the term.