Mac OS X does defrag files automatically. But there is a difference between defragging and optimizing. Defragging takes all of the pieces of a file and literally, physically reassembles them in one contiguous location on the drive. That way the system doesn't have to perform a juggling act keeping track of all of the parts of a file. Again, OS X does this automatically, provided you have enough empty space for the Mac to work its magic.
Optimizing is the act of moving all defragged data files into one section of the drive, ditto for all fonts, all applications, etc. Optimizing will then give you a nice big, empty, contiguous space for a BootCamp partition.
You can optimize with a utility like TechTools. OS X can't optimize itself. That being said, you are going to have problems because you don't have enough free space to begin with. You need a bigger drive, a second drive or and external drive. Or you need to delete all of those YouTube videos!