I used FF before Safari 4 came out and didn't get that problem. I don't think it's a FF issue (but then again I was using the beta version for the gesture support and not the official release version).
When I first got the replacement it wasn't doing it, but I think I started noticing it only when I use Adium (MSN Mac was being really really REALLY stupid, not letting me send messages). Unfortunately I can't turn that off right now cause I need to get in touch with some people from school.
I have the same problem with my white macbook and safari 4, it usually gets worse the more you use it. I just restart the computer and everything is fine. I didn't have this problem with safari 3 so I just chalked it up to 4 being a beta.