Time Machine re-backups same files after moved to different folder on same drive!

[Mac Write]

Background
I have a Mac Pro 1,1 and now that I finally got a new computer (took 5 1/2 years to get it, a rMBP 2.6/16/256) my first ever portable (and 15" is way to small for this legally blind person who sits less then 4 inches from his monitor). My Mac Pro has 250GB Boot, 2x2TB (Secondary drive and RAW Footage) and a 1TB "Primary" drive.

I use my Mac Pro as my server (Lion Server) for everything that doesn't need to be on my MBP. I have a Drobo FS exclusively as a Time Machine drive, which had a bad year in 2011 but has been allot better in 2012 until Time Machine gave me headaches.

Drobo FS
Since this is used exclusively for Time Machine, speed ins't that big a deal since it's just for backup (when Time Machine isn't being a problem which it never used to be). The Drobo has 4x2TB+500GB hard drives

The Problem
I have a 2TB (1.4TB used) "RAW Footage" drive for videos and decided to move everything out of the sub folder RAW Footage to the drive's top level. Well Time Machine didn't like that and decided to re-backup the all the moved files (1.4TB worth), causing me to run out of space.

The Questions
Since the files were being moved on the same drive, why would Time Machine need to re-backup the files? I though it would only mark the path change in the backup since it was a file move, not copy? I am going to have to start from scratch, but want to do it right this time.

Will running Lion Server be of any help with this Time Machine setup? (The Drobo FS is attached to my AEBS and soon a switch).

I really love Time Machines hourly backups so please don't suggest CCC, etc. I want to know what the problem is and now to prevent it in the future.

 

[switon]

RE: ln -s ...

Hi Mac Write,

What if you did a symbolic link instead of a move? In other words, link the "Raw Footage" directory to the top level. (Alternatively, you could even link all of the video clips to the top level directory.) Use the "ln -s ..." commands to perform the links. Time Machine will then back up the links and not the videos that have already been backed up.

Regards,
Switon

P.S. When you moved the videos, the volume's directory structure was changed which meant that Spotlight had to re-index the changes. When this happened Spotlight marked the moved video files as needing to be backed up by Time Machine. The directory where the videos were originally now had no videos, and so in the TM differential backup this directory was now empty and the links back to an earlier TM backup were broken. If you TM backup volume had not run out of space doing the new differential backup of all of your videos, then TM would have eventually eliminated the second old copy of your videos when it ran out of space. I think you have two options: (i) delete the old Raw Footage directory and all of its videos from your TM backup -- this would recover the 1.4TB of space on your TM backup volume. Then continue with a new differential backup which would backup the 1.4TB of moved videos. Or (ii) you could just start the TM backup from scratch.

To prevent this problem from reoccurring in the future, use symbolic links instead of moves when you wish to "change" the location of files, i.e., when you wish to access the files from a different directory, link the files to the new directory instead of moving them.

 

[Bear]

Background...
The Problem
I have a 2TB (1.4TB used) "RAW Footage" drive for videos and decided to move everything out of the sub folder RAW Footage to the drive's top level. Well Time Machine didn't like that and decided to re-backup the all the moved files (1.4TB worth), causing me to run out of space.
...

Since it's a RAW footage drive, I presume you won't be doing any more major rearrangements like this again?

If so delete the backups of the old RAW folder (as suggest by someone else) and let Time Machine back up the stuff in its new location.

 

[Mac Write]

The problem with deleting the old RAW footage on Time Machine, is the sparsebundle will still be too big for the drive

it used to be
/Volumes/RAW Footage/RAW Footage but that seemed pointless so I moved it to
/Volumes/RAW Footage and don't get why OS X wouldn't tell TM that the files were moved to a new location on the same drive

 

[switon]

RE: That move...

Hi Mac Write,

Someone please correct me if I am wrong, but I think the problem is how Unix handles its filesystem. When you "moved" the files, the actual data (bits on the drive) did not move, rather new entries (essentially pointers) in a special directory file were added for the moved files and old entries for those moved files in a different directory file were deleted. This caused Apple's Spotlight utility (mds, mdworker) to reindex these directories, and part of Spotlight's job is to mark for Time Machine what files need to be backed up. It does this by examining any changes to the directory structure. Therefore Spotlight informed TM through its index that your "moved" files needed to be backed up. There is no way of getting around this, as this is how the unix filesystems in conjunction with Spotlight indexing works.

As I mentioned in my earlier post, the next time you find yourself needing to move large quantities of files (such as your 1.4TB of video files), then a symbolic link from one directory to the other (a single "ln -s ..." command) would accomplish your "move" without changing the directory structure on the volume. Time Machine would then not re-back up all of those files, rather it would just back up the symbolic links (a very small amount of data relative to the large file sizes).

Lastly, you say that sparsebundle would be too big for the drive...but if it was initially big enough then after deleting the 1.4TB of video files from the sparsebundle it should once again be big enough to make your backup. By "delete" I mean to actually permanently remove these video files from the TM backup. This is accomplished by a special command in the TM GUI.

Having said this, I suspect that your best option at this point is to connect your TM backup volume by the fastest possible connection (Thunderbolt, USB3, Firewire 800, Gigabit Ethernet) and start a new TM backup. If your TM is normally going to be a network backup, as it sounds like it is since you speak of a sparsebundle, then you must first start your new TM backup as a networked TM backup, then stop the backup after the sparsebundle has been created, reattach the TM volume via TB, USB3, Gigabit, etc. to your Mac, and continue the TM backup over the faster connection. When the initial backup is finished, you disconnect the TM volume and move it back to its networked location. All subsequent differential TM backups will then be accomplished as network backups.

...just a suggestion...

Good luck,
Switon

P.S. For large quantities of files that seldom change, perhaps, say, your 1.4TB of video files, then instead of using TM to backup you might think about using "rsync" instead. Say you have your 1.4TB of videos on an external drive. You then rsync them to a second external drive -- this takes a fair amount of time to copy the 1.4TB of data. Now the next time you add a new video, you just use "rsync" to back up the new video to the second drive --- rsync works and knows only to transfer the new video (differential backup). This way you keep the 1.4TB of videos out of your TM backup and thus better utilize your TM backup for your daily content generation backup --- TM does perform hourly backups, as you know. This would allow your TM backup to potentially contain several years of historical backups of your daily work since most of its size would not be taken up by the 1.4TB of videos. At the same time, the "rsync" backup of your 1.4TB of videos provides you with a redundant copy of your videos.

 

[Mac Write]

I was thinking of splitting my TM into either 2 partitions on the Drobo (one thin provisioning) and one fixed for TM. I have a 880GB iTunes Library (7-years of podcasts no way am I deleting that beautiful an archive). I was thinking of moving more of my stuff into iTunes (videos) for AppleTV. Should I consider iTunes active or static……

I am going to redo my TM but do it right this time. I am also planning to replace my WD 1TB Black with a 3TB (4TB too pricy) and go back to a single home folder (right now I have 4 drives)

• .25TB - Boot
• 1.0TB - Primary
• 2.0TB - Secondary
• 2.0TB - RAW Footage

As soon as I can get my friend over we are going to use the 2 free SATA ports. One for my Optical Drive that is all but working 100%, and the second for an SSD 40-60GB boot drive (this is a 1.1 Mac Pro, but is now my home server until it dies). Once this change is done I will have 4 slots for Data+Drobo FS as my Time Machine which is currently 5.8TB of space (when I replace the 1TB I will then put that in the drobo to replace the 500GB).

Focusing on my TM setup. I think I should have "Archive" as a thin provisioning partition (upto 16TB) and then the fixed Time Machine partition (6TB). hopefully that will last until the drobo dies (as right now I can easily expand it to 16TB+).

Since I have a backup on the Drobo right now, I have the freedom of time to figure out the best way to do this all.

Thanks for the ideas and support. Once it is all setup I will need to setup a rsycn to monitor the "Archive" folders.