Perhaps in the strict situation of an individual attempting to gain access to your computer while you stare at them, but there are a number of ways for them to access the unencrypted volume:
- Pull it out and put it in another computer, as you say
- Bring the mac up as a firewire target disk on another computer (just requires plugging in a firewire cable and rebooting)
- Rebooting off a system DVD and resetting the password
This is why, if you have sensitive data, just login / password control is not considered enough.
As I've said many times before, though, there is a difference between paranoia and plausible situations. I wouldn't bother encrypting your Mac just because you've used Safari to buy something with your credit card. I encrypt mine because there may be confidential (HIPAA governed) data related to patients in my e-mail. For that kind of application, just passwording it is not enough.
Ugh, speaking of which, there was just a news article on NPR about a surgeon (plastics) at the hospital at which I recruit for my dissertation having given a notebook computer to a family member without securely deleting patient records from 2,000 patients, including digital pictures, etc. Probably no intrusion occurs, but you "do not want to be that guy" about whom the news carries a story because you breached patient confidentiality inadvertently....