My Mac passed all the shields up tests. I'm on 10.5. So, just because the Leopard firewall isn't as informative as the Tiger firewall doesn't mean it's not working.
Actually, it does. If nobody understands what's going on, nobody can trust it. Everything I'm reading here indicates that I'm far from alone in not having a clue about how this Firewall actually works. People have quoted the help text twice, and I've read it about a dozen times myself, and it still isn't clear. Apple has always been bad at documentation, but here's a place where it would be really useful.
To the best of my understanding, "allow all incoming connections" should essentially turn the firewall off. "Block all incoming connections" should actually say "block all unsolicited connections", because I think it will still allow responses to outgoing requests (ie. Safari). That last option, is a freaking mess. I don't get how it works or what it does. Everytime I open an app, Leopard asks me if I want to allow it to receive incoming connections, but why would it ask me that about Excel? I think it just asks as a matter of form, unless Excel is really requesting a port connection. I can choose between "always allow" which scares the bejeezus out of me, or "deny and affect performance", which isn't inviting in regards to an app I just launched for a reason.
Then there's that whole caveat about how Leopard may let all kinds of other stuff talk out the ports without user intervention (system processes, processes launched by other processes-- all the kinds of stuff you'd actually want to know about because you didn't explicitly launch the process yourself). I can't quite tell if that note applies only to the selection list or if it's a general warning about all the operating modes.
I have the feeling that Apple came up with a reasonably user friendly system here-- tying access to applications and services, which people understand, rather than port ranges and protocols, which people don't. The problem is I can't figure out what it's doing and simply can't trust it.
For completeness, I should mention that I'm not behind a router-- I've relied on the Tiger firewall as the only defense between me and my ISP. Logs indicate port scans and mostly ssh attacks, but no indication that anyone's been successful yet. I only run the services I'm interested in using, so I'd punch them through a router anyway.
Next question: Why would we EVER need a software firewall on a Mac? Seriously. Is there a scenario where it helps? Lets go back to FTP. Lets say I have a Mac directly connected to the Internet with no firewall. Someone finds my computer and tries to open a connection of Port 21. What will happen? Nothing, because I'm not running FTP and not listening on 21. So if I block 21 or not I get the same result, nothing. So why block it?
Firewalls make the most sense in a larger organization where we don't know if some idiot is running a misconfigured FTP server on his PC, so we add a firewall which makes that imposable.
Firewalls do allow local services to send requests and receive responses. They just don't allow unsolicited incoming requests. There is still value in having it running.