Well your music folder is not the only location of audio files on your computer. If you play games or use other interactive programs, these audio files have to be stored somewhere (they may not necessarily be .mp3 format either)
You mentioned defragmenting ... Can you share your end goal with us? Something tells me there's an easier way than deleting all your music.