The VM size is not real. That is, it doesn't actually correspond to the amount of memory in use or the amount of swap space on the disk. It's simply the total amount of memory address space that has been requested by your applications. Many applications will request much more space than they need "just in case". The memory isn't actually taken or marked as in-use unless the application actually does something with it.
Think of it as a theoretical upper bound for memory usage if every open application went nuts and used every bit of memory it asked for. This will never happen in the real world.
As MacsRgr8 suggested, the contents of /var/vm will tell the real story in terms of actual swap space used on disk. The files "swapfile0", "swapfile1" etc are what you want to look at; if you have a "sleepimage" file, ignore that - it's a copy of your entire RAM the last time the computer went to sleep (for safe sleep purposes).
My Macbook currently shows a VM size of 10.66 GB, but the swapfile total is only 128 MB. This is not unusual.
[Edit: Bah! Beaten by WildCowboy! 😀]