Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Warning: Arq Backup removes 'thinning'
- Thread starter gilby101
- Start date
-
- Tags
- backup apps
- Sort by reaction score
Yes, Arq is now correctly thinning backups. Per backup set logging is now just two lines:
07-Oct-2023 01:03:21 AEDT Retention: Thinning backup records according to backup plan settings.
07-Oct-2023 01:03:22 AEDT Retention: Photos: Condensed backup record list from 85 to 84 backup record(s)
I am also having a lot of errors. What does "materialize" mean? Does it mean download the file from cloud so that the file can be backup?
Sorry, plse ignore question. ARQ gives the answer, when I select "materialize".
As a non-computer user of ARQ, I must be using thinning and storage limit wrong. Would appreciate your correction.
I am using Version 7.32 for macOS to backup hourly and did the following experiments.
Before the experiments, I deleted all my ARQ backups using the ARQ app, as well as deleted all the ARQ files on Google Drive (where I stored my ARQ backup records). Before the experiments, I did the very first backup by unchecking both the thinning and storage limit. About 48GB of data was uploaded to Google Drive. I then changed the parameters on the retention page given below.
1) Test Thinning
On the retention page, I unchecked the storage limit, set “thin backup records as they age” to "Keep hourly backup" for just "1" hour and the rest of the parameters were set to "0". Then let ARQ do hourly backup. I expected that at any given time, I should just have one backup record which is also the latest backup record.
Result: ARQ kept making backups and created many hourly backup records which were shown under “Restore” Google Drive on the ARQ app. They were stored on Google Drive. The thinning parameters appeared to have no effect on curtailing the number of backup records.
2) Test storage limit (limiting the GB that could be used on Google Drive)
I set "Keep hourly backup" to just "24" hours the rest of parameter were set to "0". Then set the storage limit to “1 GB”, a deliberately low value. Then let ARQ do hourly backup as before.
Result: Just like the first case, ARQ kept making backups and created hourly backup records which were shown under “Restore” Google Drive. Again, the “storage limit” appeared to have no effect on curtailing the number of backup records.
The Arq Help says:
"If you check “Thin backup records as they age”, Arq will keep fewer backup records as they get older.
For example, if you check “Thin backup records as they age” and choose to keep hourly backups for 24 hours, daily backups for 30 days, weekly backups for 52 weeks and monthly backups for 60 months, each time Arq backs up it will:
- keep one backup record per hour for the past 24 hours
- keep one backup record per day for 30 days after that (starting at 1 day after the current date)
- keep one backup record per week for 52 weeks after that (starting at 31 days after the current date)
- keep one backup record per month for 60 months after that (starting at 31 days + 52 weeks after the current date)"
You could ask Arq Support what is expected to happen with n,0,0,0.
The minimum might well be 24,1,1,1 - one backup for per hour for 24 hours, then daily backup for 1 day, then weekly backup for 1 week, and then monthly backups for 1 month. With that I believe you would get hourly backups for a day and then 3 more backups with the oldest backup 1month+1week+1day old. That would be my choice for testing.
My settings:
1. I do daily backups to cloud storage.
2. My retention settings vary a bit, but most of the backup sets use 24,30,26,60.
There was a time when Arq kept the oldest backup whatever the thinning numbers. Stefan, who has always been very conservative about what gets deleted, changed that following a discussion I (and likely others) had with him.
----
I don't have a storage limit set for any of my backups, but when interpreting what happens note that:
1) Help says "Arq keeps at least one backup record no matter what the budget setting".
2) The budget refers to storage used - not the size of the data being backup up. Apart from some overhead, doing 2 backups of 48GB data will only use 48GB plus size of changed data - not 96GB. Just like Time Machine, CCC, etc.
----
Final thought. Thinning due to time or size only takes place at the end of an error free backup.
I appreciate your in depth and detailed reply (also your past replies in which you were generous with your time and effort.)
You were correct an arq backup has to be “error free” for the thinning and budget to work and this makes sense. I had 1 error and that had been my problem!
When I saw your reply, I checked the backup log. It said:
"Errors occurred during backup; not applying retention rules.....not enforcing budget or removing unreferenced data" The error came from my using Google Drive. ARQ couldn’t get the content there.
The log also gave the solution: “To stop the Cloud file contents not present on disk error from being reported, please edit your backup plan, click on the Options tab, and change the dataless-files behavior to ignore or materialize as you wish”
When I set the cloud file to “ignore”, both the budget and thinning seemed to work correctly, except when thinning was set to 1 backup per hour, I got 3 backup records instead. It stayed at 3 backup records, when more backups were made. But I am not complaining.
Thanks again for your help and my problem was solved!
After doing hourly backups overnight, when I use 1,0,0,0 in thinning, ARQ now shows 1 backup record per hour, as expected.
I am so sorry about the fuss about nothing and wasting everyone's time.
I am so sorry about the fuss about nothing and wasting everyone's time.
Don't worry about the fuss. These edge cases are interesting. I genuinely did not know what to expect with the zeros.
According to what you sent me about arq’s thinning policy, 1,1,1,1 should have given the same result as 1,0,0,0. Apparently, ARQ generated the wrong result.
ARQ detected an “error” is odd, because to use ARQ, I have to use a cloud provider such as Google drive. But when ARQ detected GD, it generated one error which, as you had point out, disactivated thinning and budget. Your insight of something to do with error was what solved my problem. After putting the “cloud files” to “ignore” (you had another post that used “materialized” which I had read, that was why when u said “error”, I knew what u meant,and vola problem solved) both thinning and budget work correctly.
I wish ARQ was more descriptive,for example, instead of error, uses error(retention ignored).
ARQ detected an “error” is odd, because to use ARQ, I have to use a cloud provider such as Google drive. But when ARQ detected GD, it generated one error which, as you had point out, disactivated thinning and budget. Your insight of something to do with error was what solved my problem. After putting the “cloud files” to “ignore” (you had another post that used “materialized” which I had read, that was why when u said “error”, I knew what u meant,and vola problem solved) both thinning and budget work correctly.
I wish ARQ was more descriptive,for example, instead of error, uses error(retention ignored).