If I use FileVault and DON'T use Secure Erase, is it recoverable?

TO answer your main question, the files could be recovered by someone who knows your password. But in that case they would have access to all your files anyway.

I don't think you need to use secure erase if you are using FileVault.

 

Since you have an SSD, I would not recommend using secure erase, as it writes zeroes over the file it's deleting. I would recommend you enable trim if it's not already turned on and your drive is compatible. If it has an aggressive garbage collection, I wouldn't worry.

Also, deleted files would remain encrypted regardless because of full drive encryption.

 

AFAIK, in FileVault 2, Apple encrypts your already encrypted encryption keys (encrypted with your hashed password) with a long random key. Because no one knows this random key, not even Apple, it is nearly impossible to recover the unencrypted data. This works like a OTP.