I've not heard of this performance hit based on the amount of RAM you have. However, the 32-bit version of Windows is limited to an address space of 4GB, but this doesn't mean that it can use 4GB of RAM--it can't. In actuality, it's limited to somewhere between 2GB-3GB, depending on hardware configurations. Additionally, your processor has to be capable of running 64-bit software. If you have an older Intel Mac that has either a Core Solo or a Core Duo processor, then your machine isn't 64-bit anyway. The Core2 Duo or better is required to use 64-bit software.
Lastly, the 64-bit version of XP isn't really Windows XP, and the 32-bit version of Vista never should have been released, as Vista really needs more than 32-bit addressing is capable of providing. 64-bit in general should be slower than 32-bit since it's twice as much data, but Intel did 32-bit so poorly that 64-bit is actually a little faster in general. Additionally, more RAM is always better, so if your machine is capable of handling more than 3GB of RAM, then 64-bit is more future-proof.
As you can see, it's not a straight-forward answer.