Well, a few things come to mind.
First, what makes you think your drive isn't being used when a file transfer is taking place? I don't know your setup, don't know whether your OS is on the drive you mention, etc., but the OS has to be doing something while that transfer is taking place. It may not be doing much - depends on whether or not your card uses bus mastering (handling on-card transfers itself, with minimal input from the OS), and how good that mastering is. One good way to find out is to launch Activity Monitor while doing a transfer, see how much your CPU and memory are being impacted, and by what.
Second, maybe you just have a bad card. It happens, even with NEC chipsets. I've had to throw cards away before.
Third, maybe you have one or more bad drives. Whether we're talking about USB flash drives, external hard drives, whatever - each of them is going to have its own drive controller that regulates I/O. It may not be that one of them is bad, it may be that it just doesn't "like" your card's chipset, or something.
Fourth, perhaps you've unknowingly added a USB 1.1 device somewhere in this mix? That will slow things down, even if the device works perfectly fine.
Fifth, maybe your Mac's PCI controller is failing. In which case none of the above matters. Unlikely but possible.
As you can hopefully see, there are just a whole lot of variables here. I haven't even touched all of them. To find the problem, you're going to have to test, test, test, until you've empirically eliminated enough of them that we, or you, can reason our way to a conclusion.
