How much slower it is depends on a couple of things.
First, if you have a CPU that supports AES-NI, OSX should use that (I believe it does, at least) and the CPU bit won't be slowing anything down.
Second: some SSD drives such as those based on Sandforce chips actually compress data before writing it to NAND. (Interestingly, they also encrypt it before doing so but OSX does not use that capability, so a filevault system's data is actually encrypted twice before going to NAND) This means that it's usually writing less data to NAND than you're feeding it, so it appears to be writing faster than it actually is. It has the added effect of making your NAND last longer because it is used less.
This does mean, though, that write performance on non-compressable data is less than compressable data. AES-encrypted data is, almost by definition, non-compressable because data that has been encrypted with a good algorithm is indistinguishable from random data.
So if you have a drive with a controller that gets its high write performance by compressing data, you'll find it's a bit slower with Filevault. Read performance also suffers slightly due to the same effect, but less so.
An SSD that does not compress data should be limited only by how fast your CPU can encrypt and decrypt the data.