Yeah, with that much room you have to weigh in the time you've spent messing with it; the slow down due to a large number of files is pretty negligible once it's indexed.
You might wanna check out Back in Time. It's a utility that gives you considerably more control over Time Machine; you can copy stuff, remove it, etc etc.
The problem with archiving on your iMac is that when the disk goes kaput so does your stuff. I have business stuff, so I obviously have to be more careful, but it helps to have an overall strategy for keeping stuff that's important, and getting rid of junk. Each type of backup, archives, offsite or cloud storage, iterative backups, clones, etc has its benefits and weaknesses, and each are used for different types of data.
Rob