Filevault creates an encrypted sparse disk image and dumps all your files in it. That's how it protects them.
It then transparently mounts this disk image as your home folder. That's essentially all FileVault does, in a nutshell.
It's secure, because even if your hard drive were removed and mounted on another Mac, the data inside the FileVault image would not be accessible without the password.
There are two related problems associated with this. The more innocuous one is that, just like a USB drive, this adds another layer of complexity in the running of the system. The system has to close all writing of files to the hard drive, in any situation, or else risk loss of data. Now the system has to commit its changes to the disk image and the disk image in turn has to be committed to the drive. So, if the computer crashes, for instance, there seems to be a higher rate of losing / corrupting preference files than there is without FileVault.
Second, and much less innocuous, is that it is theoretically possible that this single disk image could become corrupted in such a way that it fails to mount. If this happens, all your files are now in an unmounted image that is encrypted using fairly hefty encryption. Now what do you do? How can you get the files out? For security reasons, the image was built in the first place so that you have to be able to mount it to be able to decrypt files from it. I.E. You are SOL.
I don't know how often this has happened -- I haven't seen a lot of reports of it. But that's the danger. And that's why people recommend you be careful -- e.g. use FileVault on a notebook computer when you are also capable of making routine backups to some other secure location. But don't use it just for fun.
