Nah, don't bother with Filevault. Most average people don't need it, although the option is nice to have available for those who really do need it, or for the paranoid...
And yes, it slows down your computer. I've also read that it's somewhat flawed, although I don't know what these flaws are, specifically. I've had it turned on for a week last July, but eventually I just didn't bother.