ZFS on Linux боевые тесты на бажном железе!

В контексте моей повальной увлеченности ZFS последние лет 7 решил поднять его на Linux да не просто поднять - а выбрать самую бажную машину из парка и поставить на ней.

Насколько бажна эта машина и диски на ней? За последний год на ней диски менялись 12 раз! 2 раза менялся контроллер и 2 раза полностью пересоздавалась фс (ext4 поверх LSI/RAID-10) в связи с тотальным повреждением. 

Итак, после недели работы и активной записи перезаписи ZFS-таки перешел в состояние DEGRADED!

И порадовал меня 23 тыщами ошибок записи на /dev/sdd:

zpool status
  pool: data
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        DEGRADED     0     0     0
      mirror-0  ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdc     ONLINE       0     0     0
      mirror-1  DEGRADED     0     0     0
        sdd     FAULTED      3 23.0K     0  too many errors
        sde     ONLINE       0     0     0
      mirror-2  ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0

При этом в dmesg творился настоящий ад!

[919595.775408] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.775492] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 07 67 c7 40 00 00 08 00
[919595.775551] end_request: I/O error, dev sdd, sector 124241728
[919595.775622] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.775698] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 08 59 39 50 00 00 08 00
[919595.775756] end_request: I/O error, dev sdd, sector 140065104
[919595.775808] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.775884] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 09 15 92 b8 00 00 08 00
[919595.775940] end_request: I/O error, dev sdd, sector 152408760
[919595.775987] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.776061] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 09 24 1c 40 00 00 08 00
[919595.776118] end_request: I/O error, dev sdd, sector 153361472
[919595.776165] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.776240] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 08 56 f5 00 00 00 08 00
[919595.776297] end_request: I/O error, dev sdd, sector 139916544
[919595.776349] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.776424] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 06 40 4a 30 00 00 08 00
[919595.776481] end_request: I/O error, dev sdd, sector 104876592
[919595.776529] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.776605] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 02 7f ef 10 00 00 08 00
[919595.776662] end_request: I/O error, dev sdd, sector 41938704
[919595.776709] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.776783] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 09 3e 8d 98 00 00 08 00
[919595.776841] end_request: I/O error, dev sdd, sector 155094424
[919595.776889] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.776965] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 09 8a 21 a8 00 00 08 00
[919595.777021] end_request: I/O error, dev sdd, sector 160047528
[919595.777070] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919595.777145] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 08 86 97 80 00 00 08 00
[919595.777202] end_request: I/O error, dev sdd, sector 143038336
[919596.519092] megaraid_sas: scanning for scsi0...
[919610.884574] __ratelimit: 94 callbacks suppressed
[919610.884625] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919610.884691] sd 0:2:3:0: [sdd] CDB: Read(10): 28 00 00 00 0a 10 00 00 10 00
[919610.884741] __ratelimit: 94 callbacks suppressed
[919610.884776] end_request: I/O error, dev sdd, sector 2576
[919610.884821] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919610.884887] sd 0:2:3:0: [sdd] CDB: Read(16): 88 00 00 00 00 01 5d 3f b4 10 00 00 00 10 00 00
[919610.884969] end_request: I/O error, dev sdd, sector 5859423248
[919610.885009] sd 0:2:3:0: [sdd]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[919610.885075] sd 0:2:3:0: [sdd] CDB: Read(16): 88 00 00 00 00 01 5d 3f b6 10 00 00 00 10 00 00
[919610.885155] end_request: I/O error, dev sdd, sector 5859423760

После этого выполняем замену сбойного диска и ждем ресилверинга :)

Для чистоты эксперимента нагрузка (тесты) с машины не снимается:

 zpool status
  pool: data
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jan 21 15:04:27 2015
    7.78G scanned out of 2.36T at 9.38M/s, 73h4m to go
    2.60G resilvered, 0.32% done
config:

    NAME             STATE     READ WRITE CKSUM
    data             DEGRADED     0     0     0
      mirror-0       ONLINE       0     0     0
        sdb          ONLINE       0     0     0
        sdc          ONLINE       0     0     0
      mirror-1       DEGRADED     0     0     0
        replacing-0  FAULTED      0     0     0
          sdd        FAULTED      3 23.0K     0  too many errors
          sdh        ONLINE       0     0     0  (resilvering)
        sde          ONLINE       0     0     0
      mirror-2       ONLINE       0     0     0
        sdf          ONLINE       0     0     0
        sdg          ONLINE       0     0     0

errors: No known data errors