Having more physical memory only helps if your working set (set of memory pages being actively used by the applications, etc. you are running) is larger then the amount of physical memory in the system (aka paging to disk). Physical memory is an orthogonal issue to using LP64...Does 6 GB count? Only until recently can we get 8 GB in an Apple notebook. So yes the majority of users are screwed regardless of memory addressing abilities.
You don't need huge amounts of physical memory or even physical memory above the 4 GiB barrier to get all of the many non-memory related benefit of switching to a 64b process and/or 64b kernel. Heck having a 64b virtual address space and say only 2 GiB or physical RAM you will still gain the ability to memory map huge files, etc. greatly simplifying coding and file access while letting the system deal with managing the physical memory page pool for you.
(see the post right above yours and others for more details).