Repairing a Fusion drive, which fsck_cs/hfs switches?

toke lahti · Feb 7, 2023

I noticed that I haven't printed all of my old Messages to pdf's.
Turned on my old mini2012, which has been in a box for 2 years.
Didn't boot up, didn't see the ssd side of the fusion drive.
Macos in that fusion drive was 10.12.6.

I took ssd+hdd out from my mini2012.
Plugged them to my mini2018.
Fusion disk will mount, but:

Disk Utility can't repair (I'm running Monterey).
Disk Utility can't make an image out of it.
Diskutil says:

Code:

TokesMini2018:~ ext-toke$ diskutil cs list
CoreStorage logical volume groups (1 found)
|
+-- Logical Volume Group 1CC00242-E367-4994-BF2E-EBF073A99401
    =========================================================
    Name:         coldFusion
    Status:       Online
    Size:         2127230877696 B (2.1 TB)
    Free Space:   73728 B (73.7 KB)
    |
    +-< Physical Volume 48AF315D-33CA-4F69-BBDE-2BCE600E77E3
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk7s2
    |   Status:   Online
    |   Size:     127691702272 B (127.7 GB)
    |
    +-< Physical Volume 0011F803-C840-4200-815A-062221A03AC2
    |   ----------------------------------------------------
    |   Index:    1
    |   Disk:     disk6s2
    |   Status:   Online
    |   Size:     1999539175424 B (2.0 TB)
    |
    +-> Logical Volume Family E2CDFACD-F9C1-4960-AADB-5823A1D0513D
        ----------------------------------------------------------
        Encryption Type:         None
        |
        +-> Logical Volume 342DDA56-0D49-44B2-8D24-717386848766
            ---------------------------------------------------
            Disk:                  disk8
            Status:                Online
            Size (Total):          2121338847232 B (2.1 TB)
            Revertible:            No
            LV Name:               miniFusion
            Volume Name:           miniFusion
            Content Hint:          Apple_HFS
            LVG Type:              Fusion, Sparse

and:

Code:

TokesMini2018:~ ext-toke$ diskutil verifyVolume disk8
Started file system verification on disk8 (miniFusion)
Verifying storage system
Performing fsck_cs -n -x --lv --uuid 1CC00242-E367-4994-BF2E-EBF073A99401
Checking volume
disk6s2: Scan for Volume Headers
disk7s2: Scan for Volume Headers
disk6s2: Scan for Disk Labels
disk7s2: Scan for Disk Labels
Logical Volume Group 1CC00242-E367-4994-BF2E-EBF073A99401 spans 2 devices
disk7s2+disk6s2: Scan for Metadata Volume
Logical Volume Group has a 79 MB Metadata Volume with double redundancy
Start scanning metadata for a valid checkpoint
Load and verify Segment Headers
Load and verify Checkpoint Payload
Load and verify Transaction Segment
Incorporate 0 newer non-checkpoint transactions
Load and verify Virtual Address Table
Load and verify Segment Usage Table
Load and verify Metadata Superblock
Load and verify Logical Volumes B-Trees
Logical Volume Group contains 1 Logical Volume
Load and verify E2CDFACD-F9C1-4960-AADB-5823A1D0513D
Load and verify 342DDA56-0D49-44B2-8D24-717386848766
Load and verify Freespace Summary
Load and verify Block Accounting
Load and verify Live Virtual Addresses
Newest transaction commit checkpoint is valid
Load and verify Segment Cleaning
The volume 1CC00242-E367-4994-BF2E-EBF073A99401 appears to be OK
Storage system check exit code is 0
Verifying file system
Volume was successfully unmounted
Performing fsck_hfs -fn -x /dev/rdisk8
Executing fsck_hfs (version hfs-583.100.10)
Checking Journaled HFS Plus volume
The volume name is miniFusion
Checking extents overflow file
Checking catalog file
The volume miniFusion could not be verified completely
File system check exit code is 8
Restoring the original state found as mounted
Problem -69842 occurred while restoring the original mount state
Error: -69845: File system verify or repair failed
Underlying error: 8

Checking the disks with diskutil verifyDisk and they are okay.

Code:

The partition map appears to be OK
Finished partition map verification on disk6/7

I guess CS LVG can't be checked?
Checking the fusion drive:

Code:

TokesMini2018:~ ext-toke$ diskutil verifyDisk disk8
You cannot specify a CoreStorage logical volume (and a logical volume should not have a partition map), also note that Core Storage is deprecated
TokesMini2018:~ ext-toke$ diskutil verifyVolume disk6
Error starting file system verification for disk6: Invalid request (-69886)
TokesMini2018:~ ext-toke$ diskutil verifyVolume disk8
Started file system verification on disk8 (miniFusion)
Verifying storage system
Performing fsck_cs -n -x --lv --uuid 1CC00242-E367-4994-BF2E-EBF073A99401
Checking volume
disk6s2: Scan for Volume Headers
disk7s2: Scan for Volume Headers
disk6s2: Scan for Disk Labels
disk7s2: Scan for Disk Labels
Logical Volume Group 1CC00242-E367-4994-BF2E-EBF073A99401 spans 2 devices
disk7s2+disk6s2: Scan for Metadata Volume
Logical Volume Group has a 79 MB Metadata Volume with double redundancy
Start scanning metadata for a valid checkpoint
Load and verify Segment Headers
Load and verify Checkpoint Payload
Load and verify Transaction Segment
Incorporate 0 newer non-checkpoint transactions
Load and verify Virtual Address Table
Load and verify Segment Usage Table
Load and verify Metadata Superblock
Load and verify Logical Volumes B-Trees
Logical Volume Group contains 1 Logical Volume
Load and verify E2CDFACD-F9C1-4960-AADB-5823A1D0513D
Load and verify 342DDA56-0D49-44B2-8D24-717386848766
Load and verify Freespace Summary
Load and verify Block Accounting
Load and verify Live Virtual Addresses
Newest transaction commit checkpoint is valid
Load and verify Segment Cleaning
The volume 1CC00242-E367-4994-BF2E-EBF073A99401 appears to be OK
Storage system check exit code is 0
Verifying file system
Volume is already unmounted
Performing fsck_hfs -fn -x /dev/rdisk8
Executing fsck_hfs (version hfs-583.100.10)
Checking Journaled HFS Plus volume
The volume name is miniFusion
Checking extents overflow file
Checking catalog file
The volume miniFusion could not be verified completely
File system check exit code is 8
Restoring the original state found as unmounted
Error: -69845: File system verify or repair failed
Underlying error: 8

I do have some TM-backups, but haven't got time to test if they install ok.
But if I have enough faith on those, I might be brave enough to try to repair the Fusion drive.

Should I try some fsck_cs or fsck_hfs first or just go with diskutil repairVolume or repairDisk? Which one?

SMART says that the physical disks are okay, but the amounts of Total Writes are amazing:

Code:

smartctl 6.6 2017-11-05 r4594 [Darwin 21.4.0 x86_64] (sf-6.6-1)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Marvell based SanDisk SSDs
Device Model:     SanDisk SD8SB8U128G1122
Serial Number:    163102425286
LU WWN Device Id: 5 001b44 4a4282e84
Firmware Version: X4140000
User Capacity:    128,035,676,160 bytes [128 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Tue Jan 31 21:54:59 2023 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (    0) seconds.
Offline data collection
capabilities:              (0x11) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    No Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  10) minutes.

SMART Attributes Data Structure revision number: 4
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0032   100   100   ---    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   ---    Old_age   Always       -       9825
 12 Power_Cycle_Count       0x0032   100   100   ---    Old_age   Always       -       230161
165 Total_Write/Erase_Count 0x0032   100   100   ---    Old_age   Always       -       96519388635845
166 Min_W/E_Cycle           0x0032   100   100   ---    Old_age   Always       -       841
167 Min_Bad_Block/Die       0x0032   100   100   ---    Old_age   Always       -       24
168 Maximum_Erase_Cycle     0x0032   100   100   ---    Old_age   Always       -       1196
169 Total_Bad_Block         0x0032   100   100   ---    Old_age   Always       -       139
170 Unknown_Attribute       0x0032   100   100   ---    Old_age   Always       -       0
171 Program_Fail_Count      0x0032   100   100   ---    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   ---    Old_age   Always       -       0
173 Avg_Write/Erase_Count   0x0032   100   100   ---    Old_age   Always       -       952
174 Unexpect_Power_Loss_Ct  0x0032   100   100   ---    Old_age   Always       -       1941
184 End-to-End_Error        0x0032   100   100   ---    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   ---    Old_age   Always       -       2368
188 Command_Timeout         0x0032   100   100   ---    Old_age   Always       -       109
194 Temperature_Celsius     0x0022   060   056   ---    Old_age   Always       -       40 (Min/Max 23/56)
199 SATA_CRC_Error          0x0032   100   100   ---    Old_age   Always       -       0
230 Perc_Write/Erase_Count  0x0032   100   100   ---    Old_age   Always       -       130865640278016
232 Perc_Avail_Resrvd_Space 0x0033   100   100   004    Pre-fail  Always       -       100
233 Total_NAND_Writes_GiB   0x0032   100   100   ---    Old_age   Always       -       111654
234 Perc_Write/Erase_Ct_BC  0x0032   100   100   ---    Old_age   Always       -       177400
241 Total_Writes_GiB        0x0030   253   253   ---    Old_age   Offline      -       63401
242 Total_Reads_GiB         0x0030   253   253   ---    Old_age   Offline      -       59943
244 Thermal_Throttle        0x0032   000   100   ---    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

Code:

smartctl 6.6 2017-11-05 r4594 [Darwin 21.4.0 x86_64] (sf-6.6-1)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Samsung SpinPoint M9T
Device Model:     ST2000LM003 HN-M201RAD
Serial Number:    S321J9EG804079
LU WWN Device Id: 5 0004cf 21047aa5e
Firmware Version: 2BC10001
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Jan 31 21:59:59 2023 EET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (22620) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 377) minutes.
SCT capabilities:            (0x003f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   051    Pre-fail  Always       -       2
  2 Throughput_Performance  0x0026   252   252   000    Old_age   Always       -       0
  3 Spin_Up_Time            0x0023   091   091   025    Pre-fail  Always       -       2732
  4 Start_Stop_Count        0x0032   001   001   000    Old_age   Always       -       238242
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   252   252   051    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0024   252   252   015    Old_age   Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       7736
 10 Spin_Retry_Count        0x0032   252   252   051    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   001   001   000    Old_age   Always       -       226100
191 G-Sense_Error_Rate      0x0022   100   100   000    Old_age   Always       -       4
192 Power-Off_Retract_Count 0x0022   252   252   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0002   064   051   000    Old_age   Always       -       28 (Min/Max 21/49)
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   252   252   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x002a   100   100   000    Old_age   Always       -       29
223 Load_Retry_Count        0x0032   100   100   000    Old_age   Always       -       7
225 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       1150155
241 Total_LBAs_Written      0x0032   099   094   000    Old_age   Always       -       1595797
242 Total_LBAs_Read         0x0032   096   094   000    Old_age   Always       -       5609688

SMART Error Log Version: 1
No Errors Logged

bogdanw · Feb 8, 2023

Apple - "How to fix a split Fusion Drive" https://support.apple.com/en-us/HT207584

These steps permanently delete all data stored on the drives that make up your Fusion Drive. Make sure that you have a backup before continuing.

toke lahti · Feb 8, 2023

bogdanw said:
Apple - "How to fix a split Fusion Drive" https://support.apple.com/en-us/HT207584

Ouch,
I forgot to write at the beginning, that I would like to REPAIR the drive, not delete it.

toke lahti · Feb 8, 2023

Sidenote about ssd:
9 Power_On_Hours 0x0032 100 100 --- Old_age Always - 9825
233 Total_NAND_Writes_GiB 0x0032 100 100 --- Old_age Always - 111654
241 Total_Writes_GiB 0x0030 253 253 --- Old_age Offline - 63401

Doesn't this mean, that the actual amount of writes is 109TiB?
(Which was needed to keep written 62TiB of data.)

And since the drive has been power on for 9825 hours, it has written 11GiB in average.
In every hour it has been powered on.
That is amazing amount of swapping and cacheing!

toke lahti · Feb 8, 2023

If I would be wise and copy that corrupted fusion volume to an image, so I could test how fsck would work, is there any certain switches to be used with dd?

toke lahti · Feb 9, 2023

I just keep talking to myself:

There are tons of advice in the net, that if you want to fix filesystem corruption, you should make an image from that volume and work with that, so the orginal remains untouched.

My current understanding is that you can't make an image of corrupt hfs+ with dd.

Is there any guides on how to do this and with what?

I can't decide if posting this to another topic would help.
You can't know if you don't try...

toke lahti · Feb 9, 2023

First umount of fusion drive in diskutil or Disk Util.
Then
sudo dd if=/dev/disk9(fusion) of=/Volumes/destination/minifusion.iso conv=noerror,sync
results
first a lot of "Input/output error"
then after a while fusion drive mounts with new disk# and then dd will just repeat
"Device not configured".

toke lahti · Feb 9, 2023

Same problem with ddrescue:
Fusion drive keeps mounting, which results ddrescue to stop: "Fatal error reading the input file: Device not configured".

bogdanw · Feb 9, 2023

Save whatever you can from it and erase it.

toke lahti · Feb 9, 2023

Oh well, this will keep it from mounting:

GitHub - aburgh/Disk-Arbitrator: A Mac OS X forensic utility which manages file system mounting in support of forensic procedures.

A Mac OS X forensic utility which manages file system mounting in support of forensic procedures. - aburgh/Disk-Arbitrator

github.com

...but even that doesn't help... maybe it is a physical fault, despite of SMART...

Search

Search

Repairing a Fusion drive, which fsck_cs/hfs switches?

toke lahti

macrumors 68040

bogdanw

macrumors 603

toke lahti

macrumors 68040

toke lahti

macrumors 68040

toke lahti

macrumors 68040

toke lahti

macrumors 68040

toke lahti

macrumors 68040

toke lahti

macrumors 68040

bogdanw

macrumors 603

toke lahti

macrumors 68040

GitHub - aburgh/Disk-Arbitrator: A Mac OS X forensic utility which manages file system mounting in support of forensic procedures.

Our Staff