Boyang Yan

ZFS

7 min read

Identify disks by stable names

root@server5:~# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 10 Dec 14 18:32 dm-name-pve-root -> ../../dm-1
lrwxrwxrwx 1 root root 10 Dec 14 18:32 dm-name-pve-swap -> ../../dm-0
lrwxrwxrwx 1 root root 10 Dec 14 18:34 dm-name-pve-vm--100--disk--0 -> ../../dm-8
lrwxrwxrwx 1 root root 10 Dec 14 18:33 dm-name-pve-vm--104--disk--0 -> ../../dm-7
lrwxrwxrwx 1 root root 11 Dec 14 20:18 dm-name-pve-vm--106--disk--0 -> ../../dm-12
lrwxrwxrwx 1 root root 10 Dec 14 18:34 dm-name-pve-vm--108--disk--0 -> ../../dm-9
lrwxrwxrwx 1 root root 11 Dec 14 18:34 dm-name-pve-vm--109--disk--0 -> ../../dm-10
lrwxrwxrwx 1 root root 11 Dec 14 18:34 dm-name-pve-vm--111--disk--0 -> ../../dm-11
lrwxrwxrwx 1 root root 10 Dec 14 18:33 dm-name-pve-vm--114--disk--0 -> ../../dm-6
lrwxrwxrwx 1 root root 11 Dec 18 10:02 dm-name-pve-vm--115--disk--0 -> ../../dm-13
lrwxrwxrwx 1 root root 10 Dec 14 18:32 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigAMH9jrBov3S7bY5GMiHMyWrfKLEYonBH -> ../../dm-0
lrwxrwxrwx 1 root root 10 Dec 14 18:33 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigHS2YQ9Vq1JmzrsPeWkDDhISOKyBWMma3 -> ../../dm-6
lrwxrwxrwx 1 root root 11 Dec 14 20:18 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigUO344DW26c8ljcDP2Rl3KNNEeuMud2q1 -> ../../dm-12
lrwxrwxrwx 1 root root 10 Dec 14 18:32 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigXSAbVxAmD7wqcDDizu14hZXJIRedjzpi -> ../../dm-1
lrwxrwxrwx 1 root root 10 Dec 14 18:33 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigZh8LJY7CqNamLXY7jg2xoGhch2QMoqtk -> ../../dm-7
lrwxrwxrwx 1 root root 10 Dec 14 18:34 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigadGzoR8vCKWEpE8sTLy9J1FAivaAnIOA -> ../../dm-9
lrwxrwxrwx 1 root root 11 Dec 18 10:02 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigmsEPMe4QakO7yR9xNOovy1efWNvl25eJ -> ../../dm-13
lrwxrwxrwx 1 root root 11 Dec 14 18:34 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuiguyDhuFgi0ITqHmF9yAcoWpc7auF9kCdF -> ../../dm-10
lrwxrwxrwx 1 root root 10 Dec 14 18:34 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigv9xbiqxXUegkgeJ6k9ZQvn8cSU8GoskV -> ../../dm-8
lrwxrwxrwx 1 root root 11 Dec 14 18:34 dm-uuid-LVM-V6jNtUPHf7SbGrd305N72BwhrFxabuigyYiqUhIeuUaqJ3TLqytzseXIUI2Oa0VZ -> ../../dm-11
lrwxrwxrwx 1 root root 10 Dec 14 18:32 lvm-pv-uuid-FsfQ03-Nz7a-xj5c-d1Ab-9fMu-WRh7-EwyfkF -> ../../sda3
lrwxrwxrwx 1 root root  9 Dec 14 18:32 scsi-35000c50084f8e953 -> ../../sdd
lrwxrwxrwx 1 root root  9 Dec 14 18:32 scsi-35000c500850521a3 -> ../../sdb
lrwxrwxrwx 1 root root  9 Dec 14 18:32 scsi-35000c5008eefb7e3 -> ../../sdc
lrwxrwxrwx 1 root root  9 Dec 14 18:32 scsi-35000cca04eaaf9e0 -> ../../sda
lrwxrwxrwx 1 root root 10 Dec 14 18:32 scsi-35000cca04eaaf9e0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Dec 14 18:32 scsi-35000cca04eaaf9e0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Dec 14 18:32 scsi-35000cca04eaaf9e0-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Dec 14 18:32 wwn-0x5000c50084f8e953 -> ../../sdd
lrwxrwxrwx 1 root root  9 Dec 14 18:32 wwn-0x5000c500850521a3 -> ../../sdb
lrwxrwxrwx 1 root root  9 Dec 14 18:32 wwn-0x5000c5008eefb7e3 -> ../../sdc
lrwxrwxrwx 1 root root  9 Dec 14 18:32 wwn-0x5000cca04eaaf9e0 -> ../../sda
lrwxrwxrwx 1 root root 10 Dec 14 18:32 wwn-0x5000cca04eaaf9e0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Dec 14 18:32 wwn-0x5000cca04eaaf9e0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Dec 14 18:32 wwn-0x5000cca04eaaf9e0-part3 -> ../../sda3

Use the dev/disk/by-id… paths in all ZFS commands (more stable than /dev/sdX).

For three unused SAS disks have stable IDs:

  • sdb → /dev/disk/by-id/wwn-0x5000c500850521a3
  • sdc → /dev/disk/by-id/wwn-0x5000c5008eefb7e3
  • sdd → /dev/disk/by-id/wwn-0x5000c50084f8e953

They are two different naming schemes for the same underlying disk, provided by udev.

  1. wwn-0x5000c500850521a3

WWN = World Wide Name: a globally unique identifier assigned to the device (common in SAS/SCSI and also in some SATA/SATA-in-SAS contexts). It is intended to be stable across reboots, and usually stable across controller ports as well.

  1. scsi-35000c500850521a3

This is the SCSI device identifier representation that udev exposes. You can see it contains the same unique value as the WWN, just formatted differently: wwn-0x5000c500850521a3 scsi-3 5000c500850521a3 (udev flattens it into scsi-35000c500850521a3) So these two are essentially two “aliases” that both resolve to /dev/sdb.

Wipe any existing signatures (destructive)

Only do this if you’re sure those disks are not needed:

wipefs -a /dev/disk/by-id/wwn-0x5000c500850521a3
wipefs -a /dev/disk/by-id/wwn-0x5000c5008eefb7e3
wipefs -a /dev/disk/by-id/wwn-0x5000c50084f8e953
sgdisk --zap-all /dev/disk/by-id/wwn-0x5000c500850521a3 /dev/disk/by-id/wwn-0x5000c5008eefb7e3 /dev/disk/by-id/wwn-0x5000c50084f8e953

NOTE: sgdisk is part of the GPT/partitioning toolset (“gdisk”). The option:

  • —zap-all: wipes the disk’s partition-table metadata, specifically:
    • the GPT header and partition entries (both the primary at the start of the disk and the backup GPT at the end), and
    • the MBR/protective MBR area.

After —zap-all, the disk has no valid partition table. It is effectively “unpartitioned,” from the perspective of partitioning tools.

wipefs -a targets known filesystem / RAID / LVM / ZFS signatures that it recognizes (superblocks, magic strings). It does not guarantee removal of:

  • full GPT structures (especially the backup GPT at end-of-disk in all cases),
  • odd/partial partition-table remnants,
  • or metadata that doesn’t match wipefs’s signature list.

Create a striped ZFS pool (RAID0)

zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  vmstripe \
  /dev/disk/by-id/wwn-0x5000c500850521a3 \
  /dev/disk/by-id/wwn-0x5000c5008eefb7e3 \
  /dev/disk/by-id/wwn-0x5000c50084f8e953

NOTE: ashift is the “allocation shift” for a vdev. It controls the pool’s minimum block size in powers of two:

  • ashift=9 → 2^9 = 512 bytes
  • ashift=12 → 2^12 = 4096 bytes (4K)
  • ashift=13 → 2^13 = 8192 bytes (8K)

Why it matters: ZFS writes in multiples of the device’s physical sector size. If you pick an ashift smaller than the real physical sector size, you can create read-modify-write penalties and worse performance/longevity.

root@server5:~# fdisk -l /dev/sdb
Disk /dev/sdb: 1.09 TiB, 1200243695616 bytes, 2344225968 sectors
Disk model: X425_STBTE1T2A10
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 131072 bytes

ZFS can compress data on the fly. Compression is set per dataset (we set it at pool root so it inherits). Why lz4:

  • lz4 is a very fast compression algorithm with low CPU overhead and good real-world gains.
  • It’s widely used as the “safe default” because it’s hard to make performance worse with it; when data is not compressible, ZFS stores it uncompressed (or with minimal overhead).

atime is “access time.” When enabled, every time a file is read, the filesystem updates metadata to record the last access time. Why turn it off:

  • For VM storage, you don’t usually need precise “last accessed” timestamps.
  • Updating atime causes extra writes (metadata churn), which is wasted I/O and can degrade performance—especially on HDDs.

So atime=off is a common performance optimization for VM datastores.

xattr controls how ZFS stores extended attributes (metadata like ACLs, SELinux labels, Samba metadata, etc.). Two common modes:

  • xattr=dir (or “on” historically): stores xattrs in hidden files (can increase metadata I/O)
  • xattr=sa: stores xattrs in system attributes (in inode-like metadata structures)

Why sa:

  • Usually faster and more space-efficient for workloads that use xattrs/ACLs (common with SMB, some Linux features, containers).
  • Reduces the need for extra hidden files and metadata operations.

Verify:

root@server5:~# zpool status
  pool: vmstripe
 state: ONLINE
config:
 
        NAME                      STATE     READ WRITE CKSUM
        vmstripe                  ONLINE       0     0     0
          wwn-0x5000c500850521a3  ONLINE       0     0     0
          wwn-0x5000c5008eefb7e3  ONLINE       0     0     0
          wwn-0x5000c50084f8e953  ONLINE       0     0     0
 
errors: No known data errors
root@server5:~# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
vmstripe   444K  3.16T    96K  /vmstripe

Add it to Proxmox VE (PVE) as VM storage

pvesm add zfspool vmstripe --pool vmstripe --content images,rootdir --blocksize 16k
pvesm status

What blocksize (volblocksize) is A VM disk on ZFS is typically a zvol (a block device). volblocksize is the internal block size ZFS uses to store that zvol’s data. Proxmox uses this as the default when provisioning VM disks on a ZFS storage.

Why it matters It’s a trade-off between:

  • IOPS / latency for small random I/O (smaller can help),
  • metadata overhead and write amplification (too small can waste space / increase metadata),
  • throughput for larger I/O (larger can help).

Also important: volblocksize is fixed at zvol creation time. Changing it later usually means creating a new zvol (e.g., by disk migration/clone to a new storage) and copying data.

Why people use 16K 16K is commonly recommended as a “reasonable default” for general VM disks because:

  • It aligns well with modern 4K-sector disks (your drives are 4K physical), while not being as small as 4K (which ZFS warns can waste space / increase overhead).
  • OpenZFS moved toward 16K as a better default than 8K for VM-like workloads, and Proxmox’s UI/documentation discussions reflect this direction (the Proxmox UI shows 16K as the suggested/default placeholder in recent versions).

Using RAM to make ZFS faster

https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

ARC (RAM read cache) — the big lever

Check current ARC limit:

root@server5:~# cat /sys/module/zfs/parameters/zfs_arc_max
17179869184
root@server5:~# grep -E '^(size|c_max|c_min) ' /proc/spl/kstat/zfs/arcstats
c_min                           4    8448397440
c_max                           4    17179869184
size                            4    3020608

On many Proxmox installs, ARC is capped. Check:

root@server5:~# cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=17179869184

If you want ZFS to use more RAM for caching, set a higher zfs_arc_max (bytes). Example: 64 GiB:

  1. Edit: 
    vi /etc/modprobe.d/zfs.conf
# add (or change) this line:
options zfs zfs_arc_max=68719476736

Apply:

update-initramfs -u -k all
reboot

Sizing guidance (practical):

  • For a VM host, set ARC to something like 32–128 GiB, depending on how much RAM you have after VM allocations.
  • ARC helps mostly with reads and repeated access patterns.

Graph View

Backlinks