Apart from start up (and that is notoriously a Firefox thing) TFF is quite fast for me. Maybe it's beacause of extensions? What add-ons are you using?
I have installed Adblock Plus (with fanboy tracking and annoyance filters, I use GlimmerBlocker to take care of ads), Flashblock, UnPlug, DownloadHelper, OpenWith, QuicktimeEnabler, Status-4-Evar.
As for Safari (at least for Leopard users) you can try downloading Webkit builds from here:
http://code.google.com/p/leopard-webkit/
This is from one of TFF developers. He is actually trying to port Webkit 2 on Leopard.
I'm using build 535.20, and I find it much faster than regular Safari (version 1 of Peacekeeper test confirms this). There are issues with passwords (for example I cannot log-in on Macrumors when using it), but I think it is a really good companion to TFF.