Yes they do.
No there is not a clean way to remove everything if you want to restore from backup.
The only clean way is to restore and set the phone as new, that will be a JB file free, but you can't restore from backup.
For the records, 99 out of 100 those files that persist won't bother you, you won't even know that they are there.