TimeTracker is a great utility that will let you browse through your TM backups and see what's changed. If you find something big that's changing a lot, you can go into the settings as people suggest and exclude that. Pinpointing what it is will help you target the exclusion more precisely.
Don't mind the "extremely early state" stuff -- I've been using this for easily 10 years if not more and it's always been "prerelease" LOL. It works fine, but give it a while on first launch to scan through stuff.