Slow transfer speed with Carbon Copy Cloner.

[Heyes]

*****UPDATE: I'm now convinced my issue ISN'T A PROBLEM WITH CCC, and has much to do with fragmented external hdd.*****

I miss the old days when things worked and I knew how to fix 'em when they didn't.

So, I have my 2017 iMac (tiny 8gb memory) and Carbon Copy Cloner (7.0.4), hooked-up (USB 3) to a WD Elements.

I get transfer speeds of approx 3 mb/s when cloning my internal hd. It's the usual mix of relatively small file sizes (images, BB Edit files, etcetera).

And I get similar speeds when using CCC to clone files from that WD Elements to another WD Elements (file sizes typically 150mb to 1gb).

I realise speeds depend on many variables - and I don't know what speeds I've usually had.

I've only become aware of this after one of the WD Elements misbehaved and required a Disk Utility erase - and I'm now faced with cloning 5tb of files from one drive to another - with a suggested transfer time of many days.

Help appreciated, please and thanks.

 

[Heyes]

And perhaps someone can tell me why those two external drives have just spun-up when all I'm doing is browsing. There's nothing which should be using them right now.

 

[Bigwaff]

Do your WD Element enclosures have the latest firmware?

 

[eyoungren]

I miss the old days when things worked and I knew how to fix 'em when they didn't.

So, I have my 2017 iMac (tiny 8gb memory) and Carbon Copy Cloner (7.0.4), hooked-up (USB 3) to a WD Elements.

I get transfer speeds of approx 3 mb/s when cloning my internal hd. It's the usual mix of relatively small file sizes (images, BB Edit files, etcetera).

And I get similar speeds when using CCC to clone files from that WD Elements to another WD Elements (file sizes typically 150mb to 1gb).

I realise speeds depend on many variables - and I don't know what speeds I've usually had.

I've only become aware of this after one of the WD Elements misbehaved and required a Disk Utility erase - and I'm now faced with cloning 5tb of files from one drive to another - with a suggested transfer time of many days.

Help appreciated, please and thanks.

Might try turning off Safety Net. That's a per backup task setting.

 

[ignatius345]

So, I have my 2017 iMac (tiny 8gb memory) and Carbon Copy Cloner (7.0.4), hooked-up (USB 3) to a WD Elements.

I get transfer speeds of approx 3 mb/s when cloning my internal hd. It's the usual mix of relatively small file sizes (images, BB Edit files, etcetera).

And I get similar speeds when using CCC to clone files from that WD Elements to another WD Elements (file sizes typically 150mb to 1gb).

I'm assuming these WD drives are formatted APFS? Apparently APFS does not work very well with HDDs because it actually increases fragmentation (meaning more seek time from the data being physically fragmented across sectors of the drive platter). Gory details: https://eclecticlight.co/2019/10/19/should-you-enable-defragmentation-on-apfs-hard-drives/

Maybe you'd have better luck formatting them as HFS+, or dropping some cash and getting SSDs instead .

And perhaps someone can tell me why those two external drives have just spun-up when all I'm doing is browsing. There's nothing which should be using them right now.

Might be Spotlight indexing them.

 

[chabig]

It could also be because the iMac's USB ports only support 5Gbps transfer rate (according to to MacTracker).

 

[chabig]

It could also be because the iMac's USB ports only support 5Gbps transfer rate. In real world use, you shouldn't expect to see much more than 3Gbps (300 MB/s). If it's less than that, you could also have a cable problem.

 

[Heyes]

My thanks to all who've responded...

Do your WD Element enclosures have the latest firmware?

I assume so - one is just a few months old.

Might try turning off Safety Net. That's a per backup task setting.

Although it's an essential feature for ongoing backups, I could try it on the initial clone to the freshly-wiped drive.

I'm assuming these WD drives are formatted APFS? Apparently APFS does not work very well with HDDs because it actually increases fragmentation (meaning more seek time from the data being physically fragmented across sectors of the drive platter). Gory details: https://eclecticlight.co/2019/10/19/should-you-enable-defragmentation-on-apfs-hard-drives/

Maybe you'd have better luck formatting them as HFS+, or dropping some cash and getting SSDs instead .


Might be Spotlight indexing them.

Yes, they're APFS - HFS+ doesn't appear to be an option in Disk Utility.

For me, price has always been an issue with SSD and still is to some degree.
Plus, other than my daily clone, the contents of my external drives doesn't change much and so slower back-up time isn't an issue - but the time to restore 5gb after a disk erase *is*, and although I expect it to require days I don't recall things ever having been as slow as they currently are.

The external drives are excluded from Spotlight, so that shouldn't be why they suddenly spun-up.

It could also be because the iMac's USB ports only support 5Gbps transfer rate. In real world use, you shouldn't expect to see much more than 3Gbps (300 MB/s). If it's less than that, you could also have a cable problem.

It's 3mb/s I'm getting - around one-hundredth of your suggested speed.

And the cables are those supplied with the drives - no extensions or hubs are involved.

 

[ignatius345]

Yes, they're APFS - HFS+ doesn't appear to be an option in Disk Utility.

For me, price has always been an issue with SSD and still is to some degree.
Pus, other than my daily clone, the contents of my external drives doesn't change much and so slower back-up time isn't an issue - but the time to restore 5gb after a disk erase *is*, and although I expect it to require days I don't recall things ever having been as slow as they currently are.

I've experienced some very slow transfer speeds with Carbon Copy Cloner using HDDs as well, and I wasn't able to figure out exactly why either (sorry, that's of no help to you). If you're dealing with portable 2.5" drives, that could be a factor, according to this: https://support.bombich.com/hc/en-us/articles/20686495048215-Choosing-a-backup-drive

I've been moving to SSDs for most things, but it gets a bit costly once you start looking at larger capacities. Still, they've come down in price a lot over the past few years and if you happen to catch a decent one on sale, it could be worth it just to streamline things. I've found DiskPrices is a decent resource for shopping (though it is limited to Amazon).

 

[bombich]

Although it's [SafetyNet] an essential feature for ongoing backups, I could try it on the initial clone to the freshly-wiped drive.

SafetyNet isn't actually a part of the backup strategy, it's only a safety mechanism. I recommend disabling it any time the destination is dedicated to backing up the source. That said, it's not going to affect the performance of your backup.

ignatius345 is on the right track with the comment about slower HDDs, WD Element devices are on the slower end of the spectrum. Rotational HDDs, in general, are "alright", but I'd try to find one of the faster-spinning devices, especially if you have a bunch of smaller files. The file-size-distribution factor is huge too, I blogged about this one a while back (and this blog post also touches on "It's 3mb/s I'm getting - around one-hundredth of your suggested speed."):

File Copying Olympics: How File Size Impacts the Race for Performance Gold

If you're using a rotational disk for the source, APFS might not be a great choice for that volume. For larger files, I'd prefer APFS on any device. For a backup volume, I prefer APFS for any device. For a production source that has lots of small files, well, personally, I'd never use anything other than an SSD right now, but if an HDD is what you have, the legacy HFS+ format may be preferable until that device is obsolete.

Yes, they're APFS - HFS+ doesn't appear to be an option in Disk Utility.

Choose "Show All Devices" from Disk Utility's View menu, then select the parent container. If you select an already-APFS formatted volume, that volume exists inside of an APFS container, and that volume can't be reformatted as HFS+. But if you select the parent container or the whole parent device in the sidebar, HFS+ ("Mac OS Extended (Journaled)") will be a formatting option.

Mike (developer of CCC)

 

[Fishrrman]

Try different cables.

Try different USB ports (you may have done that already).

Try an SSD instead. It costs more, yes. But it's worth it.

 

[Heyes]

I've experienced some very slow transfer speeds with Carbon Copy Cloner using HDDs as well, and I wasn't able to figure out exactly why either (sorry, that's of no help to you). If you're dealing with portable 2.5" drives, that could be a factor, according to this: https://support.bombich.com/hc/en-us/articles/20686495048215-Choosing-a-backup-drive

I've been moving to SSDs for most things, but it gets a bit costly once you start looking at larger capacities. Still, they've come down in price a lot over the past few years and if you happen to catch a decent one on sale, it could be worth it just to streamline things. I've found DiskPrices is a decent resource for shopping (though it is limited to Amazon).

Again, thankyou for this.

I've always realised that portables are slower, and it's something I accept because of fewer cables and previous issues with mains-powered externals not powering down/sleeping when they should.

Also, I increasingly use portables rather than the onboard hd in my Mac - my use is light: word processing, email, a bit of image processing, etcetera and I like the literal portability of 'grab a drive and go'. And backups being slower than a powered drive or SSD isn't a problem because they usually run at end-of-day or if earlier are small/tiny.

Because it's only rarely that I transfer higher quantities of data between external drives, I've not previously noticed issues with speed. And I can live with perhaps relative slowness - I've previously noticed just 50gb/hour transfer) and I've been ok with that for when a drive fails and I have to write 5tb to a replacement. But the perhaps 15gb/hour I'm getting makes things painfully slow - particularly because I want to batch-write with rest periods to enable the drives to cool.

I'm encouraged by the falling prices of SSD.

Try different cables.

Try different USB ports (you may have done that already).

Try an SSD instead. It costs more, yes. But it's worth it.

Thanks.
They're the OED, and there's no difference between ports.

 

[Heyes]

SafetyNet isn't actually a part of the backup strategy, it's only a safety mechanism. I recommend disabling it any time the destination is dedicated to backing up the source. That said, it's not going to affect the performance of your backup.

ignatius345 is on the right track with the comment about slower HDDs, WD Element devices are on the slower end of the spectrum. Rotational HDDs, in general, are "alright", but I'd try to find one of the faster-spinning devices, especially if you have a bunch of smaller files. The file-size-distribution factor is huge too, I blogged about this one a while back (and this blog post also touches on "It's 3mb/s I'm getting - around one-hundredth of your suggested speed."):

File Copying Olympics: How File Size Impacts the Race for Performance Gold

If you're using a rotational disk for the source, APFS might not be a great choice for that volume. For larger files, I'd prefer APFS on any device. For a backup volume, I prefer APFS for any device. For a production source that has lots of small files, well, personally, I'd never use anything other than an SSD right now, but if an HDD is what you have, the legacy HFS+ format may be preferable until that device is obsolete.


Choose "Show All Devices" from Disk Utility's View menu, then select the parent container. If you select an already-APFS formatted volume, that volume exists inside of an APFS container, and that volume can't be reformatted as HFS+. But if you select the parent container or the whole parent device in the sidebar, HFS+ ("Mac OS Extended (Journaled)") will be a formatting option.

Mike (developer of CCC)

Thankyou.

I'm keen to learn and better understand... why do recommend 'disabling SafetyNet any time the destination is dedicated to backing up the source'?

On some small tests, the transfer varies from perhaps 3 - 90mb/s, and throughputs of up to pro-rata 100gb/hour - and I'd be happy with that upper figure. But on a current write (of files largely approx 150-500mb), it's taken 6+ hours to write 100gb. (I usually write in smaller batches, to help keep the drives cooler.)

And I've noticed that sometimes similar batches vary significantly in the time taken.
And that they slow as the write progresses - although the destination has plenty of free space.
Maybe this variance is because the source (with about just 10% free space) is likely heavily fragmented.

And thanks for the advice on HFS+.

Although I realise potential slowness of portables, I increasingly use them rather than the onboard hd in my Mac - my use is light: word processing, email, a bit of image processing, etcetera and I like the literal portability of 'grab a drive and go'.

And backups being slower than a powered drive or SSD isn't a problem because they usually run at end-of-day or if earlier are small/tiny. It's when a drive fails and I have to write 5tb to a replacement that I wince - knowing it'll take days.

Anyway... this current experience has been useful to me, in that I know a little more than previously and so can better plan strategy going forward. And also that perhaps I'd be better with a smaller quantity of retained files - some of which'll almost certainly never get looked at again.

 

[Fishrrman]

I'm using CarbonCopyCloner on a Mac Mini (m4), OS is 15.5 "Sequoia".

My backup drive is a Samsung t7 "Shield". This is a USB3.1 gen2 speed SSD.

My internal drive is "split" into 4 "hard partitions" (NOT APFS "containers").
Boot (accounts & applications)
Main (general, personally-created data, many small files)
Media
Music

For this thread, I'm going to run a backup of the Boot and Main partitions right now.
(be right back).

Boot backup:

Main backup:

Is that "fast enough" for you?

 

[ignatius345]

Although I realise potential slowness of portables, I increasingly use them rather than the onboard hd in my Mac - my use is light: word processing, email, a bit of image processing, etcetera and I like the literal portability of 'grab a drive and go'.

If you're using portable drives a lot, I just can't recommend enough biting the bullet and getting a decent SSD.

I'm still backing up to a 4 TB HDD just because it's there and I don't feel like ponying up ~$200 for an SSD version just yet. But eventually that and a couple other odd HDDs will die or I'll just phase them out.

 

[Fishrrman]

OP wrote:
"UPDATE: I'm now convinced my issue ISN'T A PROBLEM WITH CCC, and has much to do with fragmented external hdd."

That could very well be the case -- APFS is hard on HDDs, and CAN fragment the drive severely.

If CCC is your backup of choice, there's an easy way to fix this:
1. ERASE the HDD ... clean it off (a quick erase is all you need, you don't need to "securely erase" data)
2. Run a completely new, fresh CCC backup on it.

ADVISORY:
IF you are using CCC's "safety net" feature, you'll lose older, archived files. All you'll have after the new backup is what is on your Mac at the time of backup.

HOWEVER:
If you don't use the safety net feature (I don't use it myself), you won't "lose anything", because each successive CCC backup updates to the state the internal drive is in, and nothing more.

It's up to you.

Again, take a look at my reply 15 above.
A 1tb Crucial X9 USB3.1 gen2 SSD will run less than $100.
Worth the money.

 

[Heyes]

OP wrote:
"UPDATE: I'm now convinced my issue ISN'T A PROBLEM WITH CCC, and has much to do with fragmented external hdd."

That could very well be the case -- APFS is hard on HDDs, and CAN fragment the drive severely.

Thanks.

Yes, I'm increasingly learning this.
Although, for compatibility and going forward, the format seems wise.

If CCC is your backup of choice, there's an easy way to fix this:
1. ERASE the HDD ... clean it off (a quick erase is all you need, you don't need to "securely erase" data)
2. Run a completely new, fresh CCC backup on it.

I'm firmly in favor of CCC.

I did a wipe **although only erasing the content of the volume with which I'd noticed the problem** before posting initially posting here - Disk Utility had reported a 'space manager' issue after some problems (slow to mount and/or display folders).

And I'm about to do another - this time also removing and recreating all volumes and containers on the hdd (I've read of one person having success after such a full reformat.)

 

[Heyes]

Again, take a look at my reply 15 above.
A 1tb Crucial X9 USB3.1 gen2 SSD will run less than $100.
Worth the money.

Your results are impressive.

And I see the value in the Crucial SSDs - but my storage requirement is for a pair of 5tb drives, and the cost is beyond me.

 

[bombich]

I hope I'm not straying too far off topic, but:

ADVISORY:
IF you are using CCC's "safety net" feature, you'll lose older, archived files. All you'll have after the new backup is what is on your Mac at the time of backup.

HOWEVER:
If you don't use the safety net feature (I don't use it myself), you won't "lose anything", because each successive CCC backup updates to the state the internal drive is in, and nothing more.

I think you have these swapped – when SafetyNet is enabled, old content is archived on the "current" backup. When SafetyNet is disabled, the "current" backup remains identical to the source (i.e. "clean"). SafetyNet isn't related to backup versioning, and typically SafetyNet should be disabled. SafetyNet was only designed to be a safety mechanism, it doesn't really have a place in the backup strategy.

For versioning (i.e. access to older versions of your files and files that were deleted from the source), CCC uses snapshots. When you select a destination volume, CCC (v7) will prompt you to use the destination exclusively for the backup, and will give you the option of enabling backup versioning (and that's enabled by default). When you choose the option to use the volume exclusively for your backup, SafetyNet will be disabled, and versioning will be enabled or disabled per your choice.

 

[Fishrrman]

Thanks Mr. Bombich.

But... I had suggested (in reply 16) that the OP fully ERASE his backup drive, and "start from scratch".

There wouldn't be anything left on it, if he does that... ?

 

[Heyes]

To cap-off what I started, and hopefully leave something of use to others...

I've now switched back to desktop drives, and the speed difference is huge.
I'd originally expected the portables might be half the speed of desktop units... but, and on a freshly-wiped disk, I'm currently getting at best about *one-tenth* the write speed - 60gb/hour versus often 500-600gb/hour - and often less (dropping to 30gb/hour). Read speeds of the portables are fine (500-600gb/hour). I hope the write speeds of the desktops don't fall too much as they fill.

So, while I was happy to accept slower speeds because of fewer wires and power points under my desk, I no longer am.
My experience has been that when a unit fails (as they inevitably will, eventually) it's almost impossibly slow to write terrabytes (in my case about 4-5) of content from a portable hdd to another portable hdd.

So, a 'big yay!' to CarbonCopyCloner - to whom I apologise... because it's not been at fault, and the error in opening this thread has been entirely mine. I thank everyone who's replied.

And please, before anyone again suggests SSD - as I've previously noted, I simply can't afford the 10tb I'd need.