I had a 2011 i5/4gb/256gb MBA running Lion (10.7) for two years. I never saw any problems. And I ran software like Logic Pro, Final Cut Pro, VectorWorks, Photoshop, FileMaker Pro.
I'm sure there was swapping going on, but I was never aware of it with the fast SSD. Since the new CPUs and disks are faster, it should be even less of a problem. Remember, 4gb was the maximum memory available on the MBA until 2012.
My system had an HDD, so it was very noticeable. An SSD would help tremendously I imagine.