A copy about ZFS Internals(p1-p5)

ZFS Internals It ‘s dangerous operations

After 8 years, I ‘m try to study openzfs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
$ dd if=/dev/zero of=disk0 bs=1M count=100
104857600 bytes (105 MB) copied, 0.0582985 s, 1.8 GB/s
$ dd if=/dev/zero of=disk1 bs=1M count=100
104857600 bytes (105 MB) copied, 0.0587067 s, 1.8 GB/s
$ zpool create test-pool /tank/test/disk0 /tank/test/disk1

# I don 't know why only 128M could be used. loss a lot of capacity
$ zpool list test-pool
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
test-pool 160M 244K 160M - 1% 0% 1.00x ONLINE -

$ zfs get compression test-pool
NAME PROPERTY VALUE SOURCE
test-pool compression off default
$ zfs get recordsize test-pool
NAME PROPERTY VALUE SOURCE
test-pool recordsize 128K default

$ touch /test-pool/empty
$ du -hs /test-pool/empty
512 /test-pool/empty

# ashift=0 ?
test-pool ashift 0 default

$ rm -f testfile ;for i in {0..65535}; do echo $i >> testfile; done

$ ls -li
7 -rw-r--r-- 1 root root 382106 12月 28 12:45 testfile

$ zdb -dddddddd test-pool/ 7
Dataset test-pool [ZPL], ID 21, cr_txg 1, 404K, 7 objects, rootbp DVA[0]=<0:16600:200> DVA[1]=<1:89200:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=77L/77P fill=7 cksum=b706a1be7:479be3ec2c2:e63cf120702c:1fa949c8882cfd

Object lvl iblk dblk dsize lsize %full type
7 2 16K 128K 385K 384K 100.00 ZFS plain file (K=inherit) (Z=inherit)
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 2
path /testfile
uid 0
gid 0
atime Wed Dec 28 12:45:10 2016
mtime Wed Dec 28 12:45:12 2016
ctime Wed Dec 28 12:45:12 2016
crtime Wed Dec 28 12:45:10 2016
gen 77
mode 100644
size 382106
parent 4
links 1
pflags 40800000004
Indirect blocks:
0 L1 0:2400:200 1:2400:200 4000L/200P F=3 B=77/77
0 L0 1:28000:20000 20000L/20000P F=1 B=77/77
20000 L0 1:48000:20000 20000L/20000P F=1 B=77/77
40000 L0 1:68000:20000 20000L/20000P F=1 B=77/77

segment [0000000000000000, 0000000000060000) size 384K

$ awk 'BEGIN{print strtonum("0x"20000)}'
131072
  • 0 0 L1 0:2400:200 1:2400:200 4000L/200P F=3 B=77/77

    • L1 means two levels of indirection (number of block pointers which need to be traversed to arrive at this data)
    • 0:2400:200 = disk0 , first raidz volume ?
    • the offset from the begining of the disk (2400)
    • and the size of the block (0x200 = 512 bytes)
    • 0:2400 is the Data virtual Address 1 (dva1, Data virtual address )
    • F=3 is the fill count, and describes the number of non-zero block pointers under this block pointer
    • B=77/77 is the birth time, what is the same as the txg number, or some body said it ‘s a transaction number ?
  • 0 L0 1:28000:20000 20000L/20000P F=1 B=77/77

    • L0 is the block level that holds data (we can have up to six levels)
    • the fill count has a little different interpretation. Here the F= means if the block has data or not (0 or 1), what is different from the levels 1 and above, where it means “how many” non-zero block pointers under this block pointer.

Output first 128K of our file
16 bytes per line

-R poolname vdev:offset:size[:flags]

1
2
3
4
5
6
7
8
9
10
11
12
13
awk 'BEGIN{print strtonum("0x"20000)}'
131072

$ zdb -R test-pool/ 1:28000:20000 | head
1:28000:20000
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
000000: 0a330a320a310a30 0a370a360a350a34 0.1.2.3.4.5.6.7.
000010: 310a30310a390a38 0a33310a32310a31 8.9.10.11.12.13.
000020: 36310a35310a3431 310a38310a37310a 14.15.16.17.18.1
000030: 0a31320a30320a39 34320a33320a3232 9.20.21.22.23.24
000040: 320a36320a35320a 0a39320a38320a37 .25.26.27.28.29.
000050: 32330a31330a3033 330a34330a33330a 30.31.32.33.34.3
000060: 0a37330a36330a35 30340a39330a3833 5.36.37.38.39.40
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
$ zdb -R tank/test 1:28000:20000:r | head
0
1
2
3
4
5
6
7
8
9

$ zdb -R test-pool/ 1:28000:20000:r > file1
Found vdev: /tank/test/disk1
$ zdb -R test-pool/ 1:48000:20000:r > file2
Found vdev: /tank/test/disk1
$ zdb -R test-pool/ 1:68000:20000:r > file3
Found vdev: /tank/test/disk1

$ ls -lh
-rw-r--r-- 1 root root 128K 12月 28 12:51 file1
-rw-r--r-- 1 root root 128K 12月 28 12:53 file2
-rw-r--r-- 1 root root 128K 12月 28 12:53 file3
-rw-r--r-- 1 root root 374K 12月 28 12:45 testfile

$ head -n 2 file1
0
1
$ head -n 2 testfile
0
1
$ tail -n 2 file1
23695
23696
$ head -n 2 file2
23697
23698
$ tail -n 2 file2
45541
45$ head -n 2 file3
542
45543
$ tail -n 2 file3
65535

# why output single line ?
$ tail -n 2 file3 | hexdump -C
00000000 36 35 35 33 35 0a 00 00 00 00 00 00 00 00 00 00 |65535...........|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002b60 00 00 00 00 00 00 00 00 00 00 00 00 |............|
00002b6c
$ tail -n 2 testfile | hexdump -C
00000000 36 35 35 33 34 0a 36 35 35 33 35 0a |65534.65535.|
0000000c

#delete zero last line in file3, you could get same md5sum

$ md5sum testfile
43a1b3e2e25c9410daefdaa1d78678bf testfile
$ cat file1 file2 file3 | md5sum
43a1b3e2e25c9410daefdaa1d78678bf -

Test copy on write

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
$ cp testfile testfile-1
$ ls -i testfile-1
22 testfile-1
$ zdb -dddddddd test-pool/ 22
Dataset test-pool [ZPL], ID 21, cr_txg 1, 1.14M, 11 objects, rootbp DVA[0]=<0:8e00:200> DVA[1]=<1:14e00:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=631L/631P fill=11 cksum=b211ccbf0:3ff67b6c728:bddd89c4f0bb:183789ced23226

Object lvl iblk dblk dsize lsize %full type
22 2 16K 128K 385K 384K 100.00 ZFS plain file (K=inherit) (Z=inherit)
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 2
path /testfile-1
uid 0
gid 0
atime Wed Dec 28 13:31:22 2016
mtime Wed Dec 28 13:31:22 2016
ctime Wed Dec 28 13:31:22 2016
crtime Wed Dec 28 13:31:22 2016
gen 631
mode 100644
size 382106
parent 4
links 1
pflags 40800000004
Indirect blocks:
0 L1 0:2600:200 1:2600:200 4000L/200P F=3 B=631/631
0 L0 1:adc00:20000 20000L/20000P F=1 B=631/631
20000 L0 1:cdc00:20000 20000L/20000P F=1 B=631/631
40000 L0 1:edc00:20000 20000L/20000P F=1 B=631/631

segment [0000000000000000, 0000000000060000) size 384K

$
$ echo haha >> testfile-1
$ zdb -dddddddd test-pool/ 22
Dataset test-pool [ZPL], ID 21, cr_txg 1, 1.14M, 11 objects, rootbp DVA[0]=<0:5e00:200> DVA[1]=<1:6a00:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=637L/637P fill=11 cksum=bcaba3f58:44283babf90:cb3dd715909e:1a01d96e52cd6b

Object lvl iblk dblk dsize lsize %full type
22 2 16K 128K 385K 384K 100.00 ZFS plain file (K=inherit) (Z=inherit)
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 2
path /testfile-1
uid 0
gid 0
atime Wed Dec 28 13:31:22 2016
mtime Wed Dec 28 13:31:49 2016
ctime Wed Dec 28 13:31:49 2016
crtime Wed Dec 28 13:31:22 2016
gen 631
mode 100644
size 382111
parent 4
links 1
pflags 40800000004
Indirect blocks:
0 L1 0:5800:200 1:15400:200 4000L/200P F=3 B=637/637
0 L0 1:adc00:20000 20000L/20000P F=1 B=631/631
20000 L0 1:cdc00:20000 20000L/20000P F=1 B=631/631
40000 L0 0:7ca00:20000 20000L/20000P F=1 B=637/637 ### copy on write

segment [0000000000000000, 0000000000060000) size 384K

Find the physical address

Because there are some different with solaris zfs, it ‘s not 4MiB in header.
I dd first 5MiB data from disk0.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ dd if=/tank/test/disk0  of=/tmp/test bs=1M count=5

# Found the data in disk0
$ hexdump -C /tmp/test | more
00418fc0 13 00 00 00 00 00 00 80 00 00 00 00 00 00 2e 66 |...............f|
00418fd0 69 6c 65 33 2e 73 77 70 00 00 00 00 00 00 00 00 |ile3.swp........|
00418fe0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00419000 30 0a 31 0a 32 0a 33 0a 34 0a 35 0a 36 0a 37 0a |0.1.2.3.4.5.6.7.|
00419010 38 0a 39 0a 31 30 0a 31 31 0a 31 32 0a 31 33 0a |8.9.10.11.12.13.|
00419020 31 34 0a 31 35 0a 31 36 0a 31 37 0a 31 38 0a 31 |14.15.16.17.18.1|
00419030 39 0a 32 30 0a 32 31 0a 32 32 0a 32 33 0a 32 34 |9.20.21.22.23.24|
00419040 0a 32 35 0a 32 36 0a 32 37 0a 32 38 0a 32 39 0a |.25.26.27.28.29.|

$ awk 'BEGIN{print strtonum("0x"419000)}'
4296704
$ $ awk 'BEGIN{print strtonum("0x"419000)/1024}'
4196

$ awk 'BEGIN{print strtonum("0x"28000)}'
163840
$ awk 'BEGIN{print strtonum("0x"20000)}'
131072
$ awk 'BEGIN{print (4296704-163840)/1024}'
4036

$ dd if=/tank/test/disk0 of=/tmp/test-1 bs=1k skip=4196 count=1
$ head -n 2 /tmp/test-1
0
1

which means the offset from the begining of the physical vdev (starting after the vdev labels, plus the boot block), 4036KiB total in openzfs on linux
4132864 = 3f100 ?

Compare with ext3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
cd /tank/test
$ dd if=/dev/zero of=disk2 bs=1M count=100
104857600 bytes (105 MB) copied, 0.0600031 s, 1.7 GB/s

$ mkfs.ext3 -b 4096 /tank/test/disk2
mke2fs 1.41.12 (17-May-2010)
/tank/test/disk2 is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
25600 inodes, 25600 blocks
1280 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=29360128
1 block group
32768 blocks per group, 32768 fragments per group
25600 inodes per group

Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

$ mount disk2 /mnt -o loop
$ df -h
Filesystem Size Used Avail Use% Mounted on
/tank/test/disk2 97M 4.5M 88M 5% /mnt

$ cp -a /test-pool/testfile ./
$ filefrag -v /mnt/testfile
Filesystem type is: ef53
Filesystem cylinder groups is approximately 1
File size of /mnt/testfile is 382109 (94 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 1841 12 merged
1 12 1854 1853 82 merged,eof
/mnt/testfile: 2 extents found, perfection would be 1 extent

$ umount /mnt
$ debugfs /tank/test/disk2
1.41.12 (17-May-2010)
debugfs: stats
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: d6f9c3d7-e56b-433d-9830-69b15553029b
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 25600
Block count: 25600
Reserved block count: 1280
Free blocks: 23664
Free inodes: 25588
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 6
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 25600
Inode blocks per group: 800
Filesystem created: Wed Dec 28 16:03:47 2016
Last mount time: Wed Dec 28 16:04:10 2016
Last write time: Wed Dec 28 16:04:52 2016
Mount count: 1
Maximum mount count: 24
Last checked: Wed Dec 28 16:03:47 2016
Check interval: 15552000 (6 months)
Next check after: Mon Jun 26 16:03:47 2017
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 58b4c828-2e74-4add-91bb-4007efd5187f
Journal backup: inode blocks
Directories: 2
Group 0: block bitmap at 8, inode bitmap at 9, inode table at 10
23664 free blocks, 25588 free inodes, 2 used directories

debugfs: ls
2 (12) . 2 (12) .. 11 (20) lost+found 12 (4052) testfile

debugfs: show_inode_info testfile
Inode: 12 Type: regular Mode: 0644 Flags: 0x0
Generation: 933995507 Version: 0x00000000
User: 0 Group: 0 Size: 382109
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 760
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x58637206 -- Wed Dec 28 16:04:22 2016
atime: 0x58637206 -- Wed Dec 28 16:04:22 2016
mtime: 0x58637206 -- Wed Dec 28 16:04:22 2016
BLOCKS:
(0-11):1841-1852, (IND):1853, (12-93):1854-1935
TOTAL: 95

$ dd if=/tank/test/disk2 of=/tmp/file-1 bs=4K count=12 skip=1841

$ head -n 2 /tmp/file-1
0
1

$ tail -n 2 /tmp/file-1
10042
1004

$if=/tank/test/disk2 of=/tmp/file-2 bs=4K count=82 skip=1854
$ head -n 2 /tmp/file-2
3
10044
$ tail -n 2 /tmp/file-2
12
$ tail -n 3 /tmp/file-2
65535
12 # I have echo 12 >> to testfile.

# delete zero line(last line) in file2
$ cat /tmp/file-1 /tmp/file-2 | md5sum
48688f1bef04b9b59905409e9226ba65 -
$ md5sum /mnt/testfile
48688f1bef04b9b59905409e9226ba65 /mnt/testfile

#Modify file

$ filefrag -v /mnt/testfile
Filesystem type is: ef53
Filesystem cylinder groups is approximately 1
File size of /mnt/testfile is 382109 (94 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 4104 12 merged
1 12 4117 4116 82 merged,eof
/mnt/testfile: 2 extents found, perfection would be 1 extent

$ dd if=/tank/test/disk2 of=/tmp/disk-2-1 bs=4k count=4105
4105+0 records in
4105+0 records out
16814080 bytes (17 MB) copied, 0.161878 s, 104 MB/s
You have new mail in /var/spool/mail/root
$ tail /tmp/disk-2-1
1032
1033
1034
1035
1036
1037
1038
1039
1040

1$dd if=/tank/test/disk2 of=/tmp/disk-2-2 bs=4k skip=4105
21495+0 records in
21495+0 records out
88043520 bytes (88 MB) copied, 0.66258 s, 133 MB/s
$ head /tmp/disk-2-2
041
1042
1043
1044
1045
1046
1047
1048
1049
1050

$ echo 1111111111 >> /tmp/disk-2-1

$ cat /tmp/disk-2-1 > /tank/test/disk2
$ du -hs /tank/test/disk2
17M /tank/test/disk2
$ cat /tmp/disk-2-2 >> /tank/test/disk2
$ du -hs /tank/test/disk2
100M /tank/test/disk2
$ fsck /tank/test/disk2
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/tank/test/disk2: clean, 12/25600 files, 1936/25600 blocks

$ mount /tank/test/disk2 /mnt -o loop

$ head -n 1042 /mnt/testfile | tail -n 2
1040
11111111111
# got it , file has changed.

Same with openzfs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
-bash-4.1# zpool destroy test-pool
$ zpool create test-pool /tank/test/disk0
$ zpool create test-pool -o ashift=9 /tank/test/disk0
$ rm -f testfile ;for i in {0..65535}; do echo $i >> testfile; done

0040f6a0 ff ff ff a6 50 00 00 00 00 00 00 00 00 00 00 00 |....P...........|
0040f6b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
0040f800 30 0a 31 0a 32 0a 33 0a 34 0a 35 0a 36 0a 37 0a |0.1.2.3.4.5.6.7.|
0040f810 38 0a 39 0a 31 30 0a 31 31 0a 31 32 0a 31 33 0a |8.9.10.11.12.13.|
0040f820 31 34 0a 31 35 0a 31 36 0a 31 37 0a 31 38 0a 31 |14.15.16.17.18.1|
0040f830 39 0a 32 30 0a 32 31 0a 32 32 0a 32 33 0a 32 34 |9.20.21.22.23.24|


$ ls -li /test-pool/testfile
7 -rw-r--r-- 1 root root 382106 12月 28 17:31 /test-pool/testfile

$ zdb -dddddd test-pool/ 7
Dataset test-pool [ZPL], ID 21, cr_txg 1, 404K, 7 objects, rootbp DVA[0]=<0:71c00:200> DVA[1]=<0:71e00:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=7L/7P fill=7 cksum=d25a82905:4fd05e01a15:f9b25d4668de:21890ed460fd11

Object lvl iblk dblk dsize lsize %full type
7 2 16K 128K 385K 384K 100.00 ZFS plain file (K=inherit) (Z=inherit)
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 2
path /testfile
uid 0
gid 0
atime Wed Dec 28 17:31:11 2016
mtime Wed Dec 28 17:31:13 2016
ctime Wed Dec 28 17:31:13 2016
crtime Wed Dec 28 17:31:11 2016
gen 7
mode 100644
size 382106
parent 4
links 1
pflags 40800000004
Indirect blocks:
0 L1 0:4800:200 0:4a00:200 4000L/200P F=3 B=7/7
0 L0 0:f800:20000 20000L/20000P F=1 B=7/7
20000 L0 0:2f800:20000 20000L/20000P F=1 B=7/7
40000 L0 0:4f800:20000 20000L/20000P F=1 B=7/7

segment [0000000000000000, 0000000000060000) size 384K

$ 40f800=4257792
$ f800=63488
$ awk 'BEGIN{print (4257792-63488)/1024}'
4096

Header is 4MiB, and offset 63488 begin to write real data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
awk 'BEGIN{print strtonum("0x40f800")/1024}'
4158

$ zdb -R test-pool 0:f800:20000:r > /tmp/file1
-bash-4.1# head /tmp/file1
0
1
2
3
4
5
6
7
8
9

# modify /tmp/file1
$ head -n 4 /tmp/file1
#0
#1
2
3

$ dd if=/tank/test/disk0 of=/tmp/disk0-1 bs=1k count=4158
$ dd if=/tank/test/disk0 of=/tmp/disk0-2 bs=1k skip=4158 count=1
$ dd if=/tank/test/disk0 of=/tmp/disk0-3 bs=1k skip=4159

$ head /tmp/disk0-2
0
1
2
3
4
5
6
7
8
9

-bash-4.1# tail /tmp/disk0-2
274
275
276
277
278
279
280
281
282
28

$ ls -l /tmp/disk0-*
-rw-r--r-- 1 root root 4257792 12月 28 19:59 /tmp/disk0-1
-rw-r--r-- 1 root root 1024 12月 28 19:59 /tmp/disk0-2
-rw-r--r-- 1 root root 100598784 12月 28 20:03 /tmp/disk0-3

# modify disk0-2
$ vi /tmp/disk0-2
-bash-4.1# head -n 2 /tmp/disk0-2
#0
#1

$ cp /tmp/disk0-1 /tank/test/disk0
$ cat /tmp/disk0-2 >> /tank/test/disk0
$ cat /tmp/disk0-3 >> /tank/test/disk0
$ zpool import -d /tank/test/ test-pool
cannot import 'test-pool': I/O error
Destroy and re-create the pool from
a backup source.

$ zdb -l /tank/test/disk0
zdb -l /tank/test/disk0
--------------------------------------------
LABEL 0
--------------------------------------------
version: 5000
name: 'test-pool'
state: 0
txg: 1277
pool_guid: 2913619627256210242
errata: 0
hostname: 'nas-57-11.local'
top_guid: 14729225611287826913
guid: 14729225611287826913
vdev_children: 1
vdev_tree:
type: 'file'
id: 0
guid: 14729225611287826913
path: '/tank/test/disk0'
metaslab_array: 35
metaslab_shift: 24
ashift: 9
asize: 100139008
is_log: 0
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
--------------------------------------------
LABEL 1
--------------------------------------------
version: 5000
name: 'test-pool'
state: 0
txg: 1277
pool_guid: 2913619627256210242
errata: 0
hostname: 'nas-57-11.local'
top_guid: 14729225611287826913
guid: 14729225611287826913
vdev_children: 1
vdev_tree:
type: 'file'
id: 0
guid: 14729225611287826913
path: '/tank/test/disk0'
metaslab_array: 35
metaslab_shift: 24
ashift: 9
asize: 100139008
is_log: 0
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

$ zpool import -N -o readonly=on -f -d /tank/test/ -F -T 1277
pool: test-pool
id: 2913619627256210242
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://zfsonlinux.org/msg/ZFS-8000-72
config:

test-pool FAULTED corrupted data
/tank/test/disk0 ONLINE

Raidz size

1
2
3
4
5
6
awk 'BEGIN{print 128/7}'
18.2857
awk 'BEGIN{print 18.2857/20}'
0.914285
awk 'BEGIN{print 18.2857/18.5}'
0.988416