I've been struggling with deduping my image files for years. It seems every program I find simply does ok until it is presented with a large amount of data, then it fills up memory (8G on my MacPro) and falls over.
Well, I think I may finally have found the answer to my problem that doesn't involve me spending a week pulling my hair out trying to write my own solution. Singlemizer seems to do the right stuff no matter what.
http://minimalisticdev.com/Singlemizer/
It meets my basic functional requirements:
1. Can filter what it's looking for.
2. Uses a good hash mechanism to actually compare file data.
3. Doesn't store everything in memory as it goes along.
4. Can be purchased with PayPal.
5. Performance is very good.
I have no connection with the company other than just having forked out the $20 registration fee despite still being under the 20 free trials.
It chunked along on 1.1TB of data in a reasonable amount of time, and the system did waaay better than with my last attempts with other software. I was able to dedup an entire hard drive in a single run without crashing or grinding the system to a swapping halt on the very first try.
The application seems to do the right thing deletion-wise. It organizes everything in the trash by original folder, so undoing things before emptying the trash will be sane if necessary and the directory is date/time named too, so multiple runs can be figured out. This means keeping things like my Zenfolio duplicates can be done post-deletion rather than having to specify the folder name in an exclusion list, though long-term the list is the way to go.
One license is good for every Mac I own too- it's nice to not have to worry about using it on my laptop drives too- so far I'm a happy camper.
Paul
Well, I think I may finally have found the answer to my problem that doesn't involve me spending a week pulling my hair out trying to write my own solution. Singlemizer seems to do the right stuff no matter what.
http://minimalisticdev.com/Singlemizer/
It meets my basic functional requirements:
1. Can filter what it's looking for.
2. Uses a good hash mechanism to actually compare file data.
3. Doesn't store everything in memory as it goes along.
4. Can be purchased with PayPal.
5. Performance is very good.
I have no connection with the company other than just having forked out the $20 registration fee despite still being under the 20 free trials.
It chunked along on 1.1TB of data in a reasonable amount of time, and the system did waaay better than with my last attempts with other software. I was able to dedup an entire hard drive in a single run without crashing or grinding the system to a swapping halt on the very first try.
The application seems to do the right thing deletion-wise. It organizes everything in the trash by original folder, so undoing things before emptying the trash will be sane if necessary and the directory is date/time named too, so multiple runs can be figured out. This means keeping things like my Zenfolio duplicates can be done post-deletion rather than having to specify the folder name in an exclusion list, though long-term the list is the way to go.
One license is good for every Mac I own too- it's nice to not have to worry about using it on my laptop drives too- so far I'm a happy camper.
Paul
