Well, I bit the bullet and tried it earlier. I backed up my iTunes Library files, selected all tracks in iTunes, Moved to Trash, closed iTunes, replaced it's library files with the backups, moved the music from Trash to one folder in /Music and iTunes found everything. I kept all my ratings, playcounts etc.
I have /Music set as my iTunes folder so it might indeed be OK to move files around in there.