If it's defragged as you think it is, copy the whole thumb drive to your computer. Back it up once in a method of your choosing. Delete all files on the thumb drive. Then copy all the files back to your thumb drive. They are now defragged.
Also, do you need FAT32 compatibility with PCs? If not, format the thumb drive in HFS+. It will be faster than FAT32.
The medium of storage (flash based, disk based, etc.) has nothing to do with storage techniques. Any well used drive over time will acquire small spaces in between used storage space (files) that cannot be easily used. Read the wikipedia entry.