Do you have the fear of getting your Mac mini stolen?
I never set a Master Password nor do I use FileVault, as I don't have a need for this, even though I have mobile Macs since 2004. Only one got stolen two years ago, but even since then, I don't have that much crucial data on my Mac, that it needs to be protected. I keep the important document in DropBox, thus in the event of something being stolen, a password change is quickly done.
Other, bigger data, is stored on external HDDs.
And if you have Mac OS X 10.6.8 and earlier, FileVault still has some bugs that can turn a day to ****.