Do you have more than 4GB of RAM installed in your machine?
The biggest improvement in 64bit is in memory addressing. You'll be able to take advantage of as much RAM as you can throw at the program. If the answer is no, then it's not a really big gain.
Also consider how inconvenient it is to have to switch operating systems. Just how heavy is your work in Premiere? If it's casual, I would definitely stay in OSX.