Very recently, a lot of simple things have been making Finder restart for me. It didn't happen until the past week or so. 🙄 Little dumb things, like deleting folders and doing Exposé hotcorner drag and drops with spring-loaded folders.
Are you pretty sure that it's only accessing your FAT32 that crashes finder?
Is there a plist file worth deleting?