What kind of third party software are you running?
You might get a bit of a boost if you can run it on XP or 2000 since they are less resource intensive than Vista.
Also note that if you dedicate 2/3 Gb to your VM you don't have much left for OS X. Don't try to run anything intensive while your VM is active.
What do you mean by "If I do not swap between op systems, should I allocate more?"
B