[{"data":1,"prerenderedAt":359},["ShallowReactive",2],{"article-investigating-zfs-cksum-errors":3},{"article":4,"prev":353,"next":356},{"id":5,"title":6,"alt":7,"authors":8,"body":28,"createdAt":334,"description":33,"dummy":335,"extension":14,"img":336,"meta":337,"navigation":342,"path":343,"readingTime":344,"seo":345,"stem":346,"tags":347,"updatedAt":350,"__hash__":351,"slug":352},"articles\u002Farticles\u002Finvestigating-zfs-cksum-errors.md","Investigating inconspicuous ZFS CKSUM Errors","Investigating inconspicuous ZFS CKSUM errors",{"Markus Ressel":9},{"id":10,"bio":11,"cite":12,"country":13,"extension":14,"image":15,"jobtitle":16,"meta":17,"name":25,"stem":26,"__hash__":27},"authors\u002Fauthors\u002Fmarkusressel.md","Some super nice bio",null,"Germany","md","\u002Fimages\u002Favatar.jpg","Software Engineer",{"body":18},{"type":19,"value":20,"toc":21},"minimark",[],{"title":22,"searchDepth":23,"depth":23,"links":24},"",3,[],"Markus Ressel","authors\u002Fmarkusressel","cdZR-tMqLASDmxL9q7xXk60honEv8HVVlN770x0e7M4",{"type":19,"value":29,"toc":324},[30,34,39,47,57,68,72,87,93,100,104,115,121,127,131,141,147,150,156,159,166,189,195,202,206,215,222,228,231,237,247,253,256,262,268,274,278,289,295,301,307,311,318,321],[31,32,33],"p",{},"My home NAS was giving READ and CKSUM errors on two distinct HDDs in different mirrors of the same pool, which turned out to be a much bigger issue than anticipated. Let's investigate.",[35,36,38],"h2",{"id":37},"what-happened","What happened",[31,40,41,42,46],{},"After using a FreeNAS managed ZFS pool for many years, in January 2021 I switched to an Arch Linux \u002F OpenZFS based system for my personal NAS solution at home. Moving the 4x 4TB disks between \"BSD-ZFS\" and OpenZFS and importing the pool ",[43,44,45],"em",{},"just worked",", so a couple months later I went ahead and added two much older 2TB disks as a third mirror to the pool. While this has been working fine so far, a recent scrub revealed some READ and CKSUM errors on two drives in the pool:",[48,49,54],"pre",{"className":50,"code":52,"language":53},[51],"language-text","  pool: vol1\n state: DEGRADED\nstatus: One or more devices are faulted in response to persistent errors.\n        Sufficient replicas exist for the pool to continue functioning in a\n        degraded state.\naction: Replace the faulted device, or use 'zpool clear' to mark the device\n        repaired.\n  scan: resilvered 72.4G in 00:28:58 with 0 errors on Thu Apr 15 22:07:17 2021\nconfig:\n\n        NAME                                                STATE     READ WRITE CKSUM\n        vol1                                                DEGRADED     0     0     0\n          mirror-0                                          DEGRADED     0     0     0\n            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450        FAULTED     22     0     0  too many errors\n            ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n          mirror-1                                          ONLINE       0     0     0\n            ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n          mirror-2                                          DEGRADED     0     0     0\n            ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  FAULTED     10     0    35  too many errors\n            ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n","text",[55,56,52],"code",{"__ignoreMap":22},[31,58,59,60,67],{},"A small amount of errors can happen due to a variety of reasons, like bad cabling, software hickups, etc. and can be unproblematic. And even though ",[61,62,66],"a",{"href":63,"rel":64},"https:\u002F\u002Fjrs-s.net\u002F2015\u002F02\u002F03\u002Fwill-zfs-and-non-ecc-ram-kill-your-data\u002F",[65],"nofollow","you don't really need ECC memory for ZFS",", this particular server has 32G of ECC memory in it. While the numbers I saw were not alarming, they were enough for me to investigate further if there was more going on behind the scenes.",[35,69,71],{"id":70},"investigating-zfs-errors","Investigating ZFS errors",[31,73,74,75,78,79,82,83,86],{},"First of all I initiated a ",[55,76,77],{},"zpool scrub vol1"," to let ZFS check every single block of all disks and verify it.\nLooking at ",[55,80,81],{},"zpool status -v"," after the finished scrub revealed an ",[43,84,85],{},"alarming"," amount of CKSUM errors:",[48,88,91],{"className":89,"code":90,"language":53},[51],"ZFS has finished a scrub:\n\n   eid: 3587\n class: scrub_finish\n  host: Frittenbude\n  time: 2021-04-20 01:22:37+0200\n  pool: vol1\n state: DEGRADED\nstatus: One or more devices are faulted in response to persistent errors.\n Sufficient replicas exist for the pool to continue functioning in a\n degraded state.\naction: Replace the faulted device, or use 'zpool clear' to mark the device\n repaired.\n  scan: scrub repaired 0B in 08:47:03 with 0 errors on Tue Apr 20 01:22:37 2021\nconfig:\n\n NAME                                                STATE     READ WRITE CKSUM\n vol1                                                DEGRADED     0     0     0\n   mirror-0                                          DEGRADED     0     0     0\n     ata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450        FAULTED     22     0 1.86M  too many errors\n     ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n   mirror-1                                          ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n     ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n   mirror-2                                          DEGRADED     0     0     0\n     ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  FAULTED     10     0 1.27M  too many errors\n     ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,92,90],{"__ignoreMap":22},[31,94,95,96,99],{},"As mentioned before, some amount of errors in each category is probably nothing to worry about, but having millions of checksum blocks corrupted really wasn't the outcome I was hoping for. Sure, both disks aren't exactly factory new, but since their age is years apart, they have different capacities, and they weren't even used in the same system for the same amount of time, I found it very unlikely that both disks failed on a hardware level. Still, I had to do ",[43,97,98],{},"something"," about it.",[35,101,103],{"id":102},"trying-to-resolve-the-reported-zfs-errors","Trying to resolve the reported ZFS errors",[31,105,106,107,110,111,114],{},"While researching online about small amounts of errors, I found suggestions about replacing the cabling and\u002For possibly even the drive controller. Since I wasn't able to do either without waiting for a package delivery, I wanted to give ZFS a shot at handling the errors that it detected using the available parity data and investigate wether this was a one-off thing or possibly an ongoing, slow corruption mess lateron. So I went ahead and did what ",[55,108,109],{},"zpool status"," suggested and cleared the error counts using ",[55,112,113],{},"zpool clear vol1",". This took a couple seconds and automatically started a resilver of the pool right after, which also took a considerable amount of time (about 12 hours) and greeted me with the following:",[48,116,119],{"className":117,"code":118,"language":53},[51],"ZFS has finished a resilver:\n\n   eid: 3932\n class: resilver_finish\n  host: Frittenbude\n  time: 2021-04-20 13:46:56+0200\n  pool: vol1\n state: ONLINE\n  scan: resilvered 4.64T in 11:43:38 with 0 errors on Tue Apr 20 13:46:56 2021\nconfig:\n\n NAME                                                STATE     READ WRITE CKSUM\n vol1                                                ONLINE       0     0     0\n   mirror-0                                          ONLINE       0     0     0\n     ata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450        ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n   mirror-1                                          ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n     ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n   mirror-2                                          ONLINE       0     0     0\n     ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  ONLINE       0     0     0\n     ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,120,118],{"__ignoreMap":22},[31,122,123,124],{},"Alright, so everything is fine again... or is it? ",[43,125,126],{},"vsaucemusic",[35,128,130],{"id":129},"setback","Setback",[31,132,133,134,136,137,140],{},"To verify that the errors were gone for real I ran another ",[55,135,77],{},"... which resulted in yet another (heartrate increasing) error message from ",[55,138,139],{},"zed",":",[48,142,145],{"className":143,"code":144,"language":53},[51],"The number of checksum errors associated with a ZFS device\nexceeded acceptable levels. ZFS has marked the device as\ndegraded.\n\n impact: Fault tolerance of the pool may be compromised.\n    eid: 4009\n  class: statechange\n  state: DEGRADED\n   host: Frittenbude\n   time: 2021-04-20 14:38:52+0200\n  vpath: \u002Fdev\u002Fdisk\u002Fby-id\u002Fata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450-part2\n  vphys: id1,enc@n3061686369656d30\u002Ftype@0\u002Fslot@7\u002Felmdesc@Slot_06\u002Fp2\n  vguid: 0x0107D2E37F936AEF\n   pool: 0x3746073E539CB11B\n",[55,146,144],{"__ignoreMap":22},[31,148,149],{},"Additionally, the scrub seemed to slow down more and more. Even after almost 20 hours, progress was only at 36.89% with the estimated time remaining growing bigger and bigger:",[48,151,154],{"className":152,"code":153,"language":53},[51],"  pool: vol1\n state: DEGRADED\nstatus: One or more devices has experienced an unrecoverable error.  An\n        attempt was made to correct the error.  Applications are unaffected.\naction: Determine if the device needs to be replaced, and clear the errors\n        using 'zpool clear' or replace the device with 'zpool replace'.\n   see: https:\u002F\u002Fopenzfs.github.io\u002Fopenzfs-docs\u002Fmsg\u002FZFS-8000-9P\n  scan: scrub in progress since Tue Apr 20 13:49:30 2021\n        3.20T scanned at 46.9M\u002Fs, 2.76T issued at 40.4M\u002Fs, 7.47T total\n        3.27G repaired, 36.89% done, 1 days 09:57:14 to go\nconfig:\n\n        NAME                                                STATE     READ WRITE CKSUM\n        vol1                                                DEGRADED     0     0     0\n          mirror-0                                          DEGRADED     0     0     0\n            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450        DEGRADED 12.2K    18 17.4K  too many errors  (repairing)\n            ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n          mirror-1                                          ONLINE       0     0     0\n            ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n          mirror-2                                          ONLINE       0     0     0\n            ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  ONLINE       0     0     0\n            ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,155,153],{"__ignoreMap":22},[31,157,158],{},"Looking at the server monitoring, the disk wasn't moving any meaningful data despite trying to:",[31,160,161],{},[162,163],"img",{"alt":164,"src":165,"title":164},"Grafana dashboard showing 24 hours disk activity","\u002Fimages\u002Fblog\u002Finvestigating-zfs-cksum-errors\u002Fzfs_failed_drive_grafana.png",[31,167,168,169,172,173,176,177,180,181,184,185,188],{},"Although the old ",[43,170,171],{},"WDC_WD20EARX"," 2TB disk from ",[55,174,175],{},"mirror-2"," was seemingly successfully corrected, the much newer ",[43,178,179],{},"WDC_WD40EFRX"," was, again, running into problems. Issuing ",[55,182,183],{},"smartctl -A \u002Fdev\u002Fsdg"," revealed that the disk had an abnormally high ",[55,186,187],{},"Raw_Read_Error_Rate",", strongly indicating a hardware level fault of the disk:",[48,190,193],{"className":191,"code":192,"language":53},[51],"=== START OF READ SMART DATA SECTION ===\nSMART Attributes Data Structure revision number: 16\nVendor Specific SMART Attributes with Thresholds:\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\n  1 Raw_Read_Error_Rate     0x002f   001   001   051    Pre-fail  Always   FAILING_NOW 23484\n  3 Spin_Up_Time            0x0027   178   164   021    Pre-fail  Always       -       6066\n  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       84\n  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0\n  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0\n  9 Power_On_Hours          0x0032   054   054   000    Old_age   Always       -       33897\n 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0\n 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0\n 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       84\n192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       56\n193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       298\n194 Temperature_Celsius     0x0022   113   098   000    Old_age   Always       -       37\n196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0\n197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0\n198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0\n199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0\n200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0\n",[55,194,192],{"__ignoreMap":22},[31,196,197,198,201],{},"There was no reason to continue with the scrub, since it probably wouldn't have finished in a timely matter, and even if it did there would still be errors on the disk itself. So I aborted the scrub with ",[55,199,200],{},"zpool scrub -s vol1",". As a last effort to revive the disk without replacing it, I shutdown the server completely, allowing all disks to also fully shutdown and reinitialize on the next boot. This didn't help though, in fact importing the pool took minutes instead of seconds, probably due to the failing disk having problems to read data.",[35,203,205],{"id":204},"acceptance","Acceptance",[31,207,208,209,214],{},"So I went online to buy a new disk. Yeah I know, having a spare handy (or even a hot-spare) would have been much better, but I didn't have the money to do that. Since the failing disk was a Western Digital model, I wanted to replace it with a similar disk from the same manufacturer. This was easier said than done though, since WD changed their marketing due to the whole ",[61,210,213],{"href":211,"rel":212},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=aztTf2gI55k",[65],"CMR\u002FSMR scandal",". Reviews on Amazon were all over the place, with users receiving hardware with different model numbers than advertised, even on WD RED Plus product pages. This made me reconsider purchasing Western Digital again, as well as using Amazon as a shop. I ended up purchasing the Seagate Ironwolf 4TB (ST4000VN008) from another shop instead.",[31,216,217,218,221],{},"After 5 days of painful nervousness (actually it wasn't that bad) the new disk finally arrived since, of course, there was a weekend in between. So I opened the package, shut down the server, removed the old disk and inserted the new one in the same spot and restarted the server. A ",[55,219,220],{},"zpool status vol1"," made me aware of the fact that the disk was now completely unavailable:",[48,223,226],{"className":224,"code":225,"language":53},[51],"  pool: vol1\n state: DEGRADED\nstatus: One or more devices is currently being resilvered.  The pool will\n        continue to function, possibly in a degraded state.\naction: Determine if the device needs to be replaced, and clear the errors\n        using 'zpool clear' or replace the device with 'zpool replace'.\n   see: https:\u002F\u002Fopenzfs.github.io\u002Fopenzfs-docs\u002Fmsg\u002FZFS-8000-9P\nconfig:\n\n        NAME                                                STATE     READ WRITE CKSUM\n        vol1                                                DEGRADED     0     0     0\n          mirror-0                                          DEGRADED     0     0     0\n            74259793414679279                               UNAVAIL      0     0     0  was \u002Fdev\u002Fdisk\u002Fby-id\u002Fata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450-part2\n            ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n          mirror-1                                          ONLINE       0     0     0\n            ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n          mirror-2                                          ONLINE       0     0     0\n            ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  ONLINE       0     0     0\n            ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,227,225],{"__ignoreMap":22},[31,229,230],{},"With the new disk installed, I had to figure out its name to be able to tell ZFS to use it as a replacement. To do that I used the following command, which outputs both the short and long device name\u002Fpath:",[48,232,235],{"className":233,"code":234,"language":53},[51],"> lsblk -r|awk 'NR==1{print $0\" DEVICE-ID(S)\"}NR>1{dev=$1;printf $0\" \";system(\"find \u002Fdev\u002Fdisk\u002Fby-id -lname \\\"*\"dev\"\\\" -printf \\\" %p\\\"\");print \"\";}'\n",[55,236,234],{"__ignoreMap":22},[31,238,239,240,243,244,246],{},"To make sure I had the right disk, I compared the drive name to the ones already inside of the pool and made sure it wasn't already part of another mirror. Then I went ahead and let ZFS use the new disk using ",[55,241,242],{},"zpool replace vol1 \u002Fdev\u002Fdisk\u002Fby-id\u002Fata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450-part2 \u002Fdev\u002Fdisk\u002Fby-id\u002Fata-ST4000VN008-2DR166_ZDH9KMBQ",". The command took a couple of seconds and then finished without an error. Looking at ",[55,245,220],{}," once again revealed that ZFS was already resilvering the new disk:",[48,248,251],{"className":249,"code":250,"language":53},[51],"  pool: vol1\n state: DEGRADED\nstatus: One or more devices is currently being resilvered.  The pool will\n        continue to function, possibly in a degraded state.\naction: Wait for the resilver to complete.\n  scan: resilver in progress since Mon Apr 26 18:35:04 2021\n        3.99T scanned at 10.9G\u002Fs, 2.50T issued at 6.86G\u002Fs, 7.40T total\n        706M resilvered, 33.78% done, 00:12:11 to go\nconfig:\n\n        NAME                                                STATE     READ WRITE CKSUM\n        vol1                                                DEGRADED     0     0     0\n          mirror-0                                          DEGRADED     0     0     0\n            replacing-0                                     DEGRADED     0     0     0\n              74259793414679279                             UNAVAIL      0     0     0  was \u002Fdev\u002Fdisk\u002Fby-id\u002Fata-WDC_WD40EFRX-68N32N0_WD-WCC7K3RUX450-part2\n              ata-ST4000VN008-2DR166_ZDH9KMBQ               ONLINE       0     0     0  (resilvering)\n            ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n          mirror-1                                          ONLINE       0     0     0\n            ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n            ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n          mirror-2                                          ONLINE       0     0     0\n            ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  ONLINE       0     0     0\n            ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,252,250],{"__ignoreMap":22},[31,254,255],{},"After about 9 hours I was greeted with a promising result:",[48,257,260],{"className":258,"code":259,"language":53},[51],"ZFS has finished a resilver:\n\n   eid: 994\n class: resilver_finish\n  host: Frittenbude\n  time: 2021-04-27 03:38:44+0200\n  pool: vol1\n state: ONLINE\n  scan: resilvered 2.89T in 09:03:40 with 0 errors on Tue Apr 27 03:38:44 2021\nconfig:\n\n NAME                                                STATE     READ WRITE CKSUM\n vol1                                                ONLINE       0     0     0\n   mirror-0                                          ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZDH9KMBQ                 ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n   mirror-1                                          ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n     ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n   mirror-2                                          ONLINE       0     0     0\n     ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  ONLINE       0     0     0\n     ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,261,259],{"__ignoreMap":22},[31,263,264,265,267],{},"The resilver was finished and there were no errors reported. To make sure everything was working as expected, I once again ran a scrub using ",[55,266,77],{},", which took an additional 8 hours - and found no issues:",[48,269,272],{"className":270,"code":271,"language":53},[51],"ZFS has finished a scrub:\n\n   eid: 1648\n class: scrub_finish\n  host: Frittenbude\n  time: 2021-04-28 03:35:08+0200\n  pool: vol1\n state: ONLINE\n  scan: scrub repaired 0B in 08:09:13 with 0 errors on Wed Apr 28 03:35:08 2021\nconfig:\n\n NAME                                                STATE     READ WRITE CKSUM\n vol1                                                ONLINE       0     0     0\n   mirror-0                                          ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZDH9KMBQ                 ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414J8T-part2           ONLINE       0     0     0\n   mirror-1                                          ONLINE       0     0     0\n     ata-ST4000VN008-2DR166_ZM414FPG-part2           ONLINE       0     0     0\n     ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6TEHZD2-part2  ONLINE       0     0     0\n   mirror-2                                          ONLINE       0     0     0\n     ata-WDC_WD20EARX-00PASB0_WD-WMAZA5058866-part2  ONLINE       0     0     0\n     ata-ST2000DM001-1CH164_Z1E532WY-part2           ONLINE       0     0     0\n\nerrors: No known data errors\n",[55,273,271],{"__ignoreMap":22},[35,275,277],{"id":276},"verifying-the-defective-disk","Verifying the defective disk",[31,279,280,281,284,285,288],{},"To be sure that the old disk was indeed defective, I attached it to my desktop and ran ",[55,282,283],{},"sudo badblocks -v \u002Fdev\u002Fsdc > badsectors.txt"," for about 24 hours on it. While ",[55,286,287],{},"badblocks"," itself didn't report anything, the fact that it wasn't done after such a long time was telling. Aborting the command also revealed that it wasn't able to achieve even 1% of progress in all that time:",[48,290,293],{"className":291,"code":292,"language":53},[51],"Checking blocks 0 to 3907018583\nChecking for bad blocks (read-only test):\n^C\nInterrupted at block 32281728\n",[55,294,292],{"__ignoreMap":22},[31,296,297,298,300],{},"I also checked the SMART values again and the ",[55,299,187],{}," had increased even further:",[48,302,305],{"className":303,"code":304,"language":53},[51],"SMART Attributes Data Structure revision number: 16\nVendor Specific SMART Attributes with Thresholds:\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\n  1 Raw_Read_Error_Rate     0x002f   001   001   051    Pre-fail  Always   FAILING_NOW 24511\n  3 Spin_Up_Time            0x0027   178   162   021    Pre-fail  Always       -       6066\n  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       88\n  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0\n  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0\n  9 Power_On_Hours          0x0032   054   054   000    Old_age   Always       -       33915\n 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0\n 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0\n 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       88\n192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       60\n193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       298\n194 Temperature_Celsius     0x0022   111   098   000    Old_age   Always       -       39\n196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0\n197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       3\n198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0\n199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0\n200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0\n",[55,306,304],{"__ignoreMap":22},[35,308,310],{"id":309},"conclusion","Conclusion",[31,312,313,314,317],{},"A defective disk is mostly a non-issue when using the right file system in the right configuration. When using a file system like ZFS, hardware errors are fully transparent to the system and everything on it. With ZFS I was able to detect and replace a faulty disk with only a couple of simple commands. However, this transparency comes with the responsibility to regularly check the status of the pool. Before writing this article my self-made Telegram notification bridge for ZED was broken. I was made aware of the errors by pure luck because I happend to check my server randomly at the right time. Even though ZFS already indicated corrupted blocks even before the SMART values, I probably wouldn't have noticed it for quite some time without a functioning notification system in place. Since I only use mirrors in my pools, a second drive failure in the same mirror would have been catastrophic and although I intenionally placed disks of different manufacturers and age in a given mirror, there is still a chance of it happening, ",[43,315,316],{},"especially"," if the system continues to run unsupervised for a longer period of time.",[31,319,320],{},"The lesson I took from this experience is that my ZED (ZFS Event Daemon) notification system is cruicial to avoid catastrophics damage, and I really need to setup a way to \"monitor my monitoring\". Currently I am thinking about a prometheus exporter on the host, which is then scraped by the prometheus instance that is already running on the host inside a k3s cluster, but this is a topic for another day.",[31,322,323],{},"Thank you for reading.",{"title":22,"searchDepth":23,"depth":23,"links":325},[326,328,329,330,331,332,333],{"id":37,"depth":327,"text":38},2,{"id":70,"depth":327,"text":71},{"id":102,"depth":327,"text":103},{"id":129,"depth":327,"text":130},{"id":204,"depth":327,"text":205},{"id":276,"depth":327,"text":277},{"id":309,"depth":327,"text":310},"2021-04-28",false,"\u002Fimages\u002Fblog\u002Finvestigating-zfs-cksum-errors\u002Fheader.jpg",{"excerpt":338},{"type":19,"value":339},[340],[31,341,33],{},true,"\u002Farticles\u002Finvestigating-zfs-cksum-errors",7,{"title":6,"description":33},"articles\u002Finvestigating-zfs-cksum-errors",[348,349],"linux","zfs","2023-12-16","_V5qIejUUBgq0FhU3PkbueanuFnpJ1Xr58JXKWP42n8","investigating-zfs-cksum-errors",{"title":354,"slug":355},"The (tech) life of Markus Ressel","the-tech-life-of-markus-ressel",{"title":357,"slug":358},"Booting ZFS on Linux on Asus ROG Zephyrus M16 (2021)","booting-zfs-on-linux-on-asus-rog-zephyrus-m16",1793999264915]