--- title: "Root ZFS Encryption, Mirroring, and Remote Unlocking on Ubuntu" tags: ['Operating Systems', 'Linux', 'Security'] date: 2023-07-26 author: Tommy --- While Unbutu supports ZFS on root filesystems with an easy snapshot and rollback mechanism called ZSYS, ZSYS is [soon going to be deprecated](https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1966773) and the installer does not offer an easy way to setup mirroring. In this guide, I will walk you through how to set up Ubuntu with root on ZFS, mirroring with both the root and EFI partitions, and remote unlocking + boot into a snapshot with ZFSBootMenu. ![ZFSBootMenu](/images/zfsbootmenu.png) ## Enter the Shell Enter the shell on your Ubuntu Installer: ## Partitioning the Disk Esentially, we need a 512MB ESP partition for ZFSBootMenu and a `/` partition for the rest of the system. If you are using a single disk, just make those 2 partitions on your disk. If you are planning to do mirroring, set up both on of these partitions on each disk. There are a variety of tools you can use, but an easy to use one would be `cfdisk`. ```bash cfdisk /dev/nvme0n1 cfdisk /dev/nvme0n2 ``` ![cfdisk](/images/cfdisk.png) ## Mirroring the ESP partition _Skip this if you are not doing mirroring_ While the EFI specs do not support `mdadm`, we can setup mdadm with metadata v1.0, which will be put at the end of the parition and allows it to boot. ```bash mdadm --create /dev/md0 --level 1 --raid-disks --metadata 1.0 /dev/nvme0n1p1 /dev/nvme0n1p2 mkfs.fat -F 32 /dev/md0 ``` ## Setup the ZFS partition This part is mostly based on the [official ZFSBootMenu guide](https://docs.zfsbootmenu.org/en/v2.2.x/guides/ubuntu/uefi.html) with some changes to work around some not-so-great instructions there. ### Creating the zpool #### Getting the Disk ID. First, we must get the disk IDs from `/dev/disk/by-id`. The official guide uses the dynamically assigned drive identifier (`/dev/sda`, `/dev/nvme0n1`, etc), which is not what we want to do with zpools, since it will cause problems later on. ![/dev/disk/by-id](/images/disk-by-id.png) #### Installing ZFS-Utils ```bash sudo apt install zfsutils-linux -y ``` #### Create the encryption key ```bash echo 'SomeKeyphrase' > /etc/zfs/zroot.key chmod 000 /etc/zfs/zroot.key ``` #### For Non-Mirrored Setups ```bash sudo zpool create -o ashift=12 -O compression=zstd -O acltype=posixacl -O xattr=sa -O atime=off -O encryption=on -O keylocation=file:///etc/zfs/zroot.key -O keyformat=passphrase -o autotrim=on -o failmode=panic compatibility=openzfs-2.1-linux -m none zroot /dev/disk/by-id/nvme-SAMSUNG_MZQL21T9HCJR-00A07_XXXXXXX-part2 ``` #### For Mirrored Setups ```bash zpool create -o ashift=12 -O compression=zstd -O acltype=posixacl -O xattr=sa -O atime=off -O encryption=on -O keylocation=file:///etc/zfs/zroot.key -O keyformat=passphrase -o autotrim=on -o failmode=panic compatibility=openzfs-2.1-linux -m none zroot mirror /dev/disk/by-id/nvme-SAMSUNG_MZQL21T9HCJR-00A07_XXXXXXX-part2 /dev/disk/by-id/nvme-SAMSUNG_MZQL21T9HCJR-00A07_YYYYYYY-part2 ``` #### Notes We use slightly different options than the official guide. Most notably, `atime` is disabled as it has detrimental effect on performance and unnecessarily increases write operations. `compression` is changed from `lz4` to `zstd` as it has much better compression ratio than `lz4` while still maintaining good performance. We did not specify the encryption type here as `aes-256-gcm` is already the default with openZFS >= 0.8.4. failmode=panic makes sure that the kernel panics in case there is something wrong with the drive instead of taking risks with abnormal behavior. We specify compatibility=openzfs-2.1-linux to make sure that updates will not make the system unbootable in the future. ### Creating the filesystems ```bash zfs create -o mountpoint=none zroot/ROOT zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/ubuntu zfs create -o mountpoint=/home zroot/home zfs create -o mountpoint=/var/log zroot/ROOT/ubuntu/log zfs create -o mountpoint=/var/spool zroot/ROOT/ubuntu/spool zfs create -o mountpoint=/var/cache zroot/ROOT/ubuntu/cache zpool set bootfs=zroot/ROOT/ubuntu zroot ``` Here, we deviate from the official guide by splitting out `/var/log`, `/var/spool`, `/var/cache` out into their own datasets. These are directories which are parts of Ubuntu that we do not want to be rolled back along with the system in case we need to boot into a prior snapshot. ### Mounting the filesystem ```bash zpool export zroot zpool import -N -R /mnt zroot zfs load-key -L prompt zroot zfs mount zroot/ROOT/ubuntu zfs mount zroot/ROOT/ubuntu/log zfs mount zroot/ROOT/ubuntu/cache zfs mount zroot/ROOT/ubuntu/spool zfs mount zroot/home ``` ### Update device symlink ```bash udevadm trigger ``` ## Install Ubuntu We will deviate from the ZFSBootMenu's documentation here, as it only installs a minimal system with SysVinit. Instead, we can install ubuntu-server-minimal. ### Bootstrapping ```bash zgenhostid -f 0x00bab10c apt install -y debootstrap debootstrap jammy /mnt ``` ### Copy files into the new install ```bash cp /etc/hostid /mnt/etc/hostid cp /etc/resolv.conf /mnt/etc/ mkdir /mnt/etc/zfs cp /etc/zfs/zroot.key /mnt/etc/zfs ``` ### Chroot into the new OS ```bash mount -t proc proc /mnt/proc mount -t sysfs sys /mnt/sys mount -B /dev /mnt/dev mount -t devpts pts /mnt/dev/pts chroot /mnt /bin/bash ``` ### Setup the repositories Use the source list from the Ubuntu live ISO: ```bash cat < /etc/apt/sources.list # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to # newer versions of the distribution. deb http://archive.ubuntu.com/ubuntu/ jammy main restricted # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted ## Major bug fix updates produced after the final release of the ## distribution. deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted ## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu ## team. Also, please note that software in universe WILL NOT receive any ## review or updates from the Ubuntu security team. deb http://archive.ubuntu.com/ubuntu/ jammy universe # deb-src http://archive.ubuntu.com/ubuntu/ jammy universe deb http://archive.ubuntu.com/ubuntu/ jammy-updates universe # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates universe ## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu ## team, and may not be under a free licence. Please satisfy yourself as to ## your rights to use the software. Also, please note that software in ## multiverse WILL NOT receive any review or updates from the Ubuntu ## security team. deb http://archive.ubuntu.com/ubuntu/ jammy multiverse # deb-src http://archive.ubuntu.com/ubuntu/ jammy multiverse deb http://archive.ubuntu.com/ubuntu/ jammy-updates multiverse # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates multiverse ## N.B. software from this repository may not have been tested as ## extensively as that contained in the main release, although it includes ## newer versions of some applications which may provide useful features. ## Also, please note that software in backports WILL NOT receive any review ## or updates from the Ubuntu security team. deb http://archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse # deb-src http://archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse deb http://security.ubuntu.com/ubuntu/ jammy-security main restricted # deb-src http://security.ubuntu.com/ubuntu/ jammy-security main restricted deb http://security.ubuntu.com/ubuntu/ jammy-security universe # deb-src http://security.ubuntu.com/ubuntu/ jammy-security universe deb http://security.ubuntu.com/ubuntu/ jammy-security multiverse # deb-src http://security.ubuntu.com/ubuntu/ jammy-security multiverse EOF ``` ### Install the necessary packages ```bash apt update apt install --no-install-recommends linux-generic ubuntu-server-minimal ``` ### Configure packages to customize local and console properties ```bash dpkg-reconfigure locales tzdata keyboard-configuration console-setup ``` ## ZFS Configuration ### Install required packages ```bash apt install dosfstools zfs-initramfs zfsutils-linux ``` ### Enable systemd ZFS services ```bash systemctl enable zfs.target systemctl enable zfs-import-cache systemctl enable zfs-mount systemctl enable zfs-import.target ``` ### Enable systemd-networkd ```bash systemctl enable systemd-networkd ``` ### Configure `initramfs-tools` ```bash echo "UMASK=0077" > /etc/initramfs-tools/conf.d/umask.conf ``` ### Rebuild the initramfs ```bash update-initramfs -c -k all ``` ## Install and configure ZFSBootMenu ### Setup the EFI partition If you are doing mirroring: ```bash zfs set org.zfsbootmenu:commandline="quiet loglevel=4" zroot/ROOT zfs set org.zfsbootmenu:keysource="zroot/ROOT/ubuntu" zroot cat << EOF >> /etc/fstab $( blkid | grep /dev/md0 | cut -d ' ' -f 2 ) /boot/efi vfat defaults 0 0 EOF mkdir -p /boot/efi mount /boot/efi ``` If you are not, just replace `md0` in the commands above with your efi partition. ### Install ZFSBootMenu #### To use it without remote unlocking, just follow the official guide: ``` apt install curl -y mkdir -p /boot/efi/EFI/ZBM curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI ``` #### To use it with remote unlocking, you have to compile the package: ```bash git clone https://github.com/dracut-crypt-ssh/dracut-crypt-ssh apt install -y blah blah blah dependency here I forgot some build command here echo 'omit_dracutmodules+=" crypt-ssh "' >> /etc/dracut-config-location-idk mkdir -p /etc/dropbear ssh-keygen -t rsa -m PEM -f /etc/dropbear/ssh_host_rsa_key ssh-keygen -t ecdsa -m PEM -f /etc/dropbear/ssh_host_ecdsa_key ssh-keygen -t ed25519 -m PEM -f /etc/dropbear/ssh_host_ed25519_key mkdir -p /etc/cmdline.d echo "ip=dhcp rd.neednet=1" > /etc/cmdline.d/dracut-network.conf cat < /etc/zfsbootmenu/dracut.conf.d/dropbear.conf # Enable dropbear ssh server and pull in network configuration args add_dracutmodules+=" crypt-ssh " install_optional_items+=" /etc/cmdline.d/dracut-network.conf " # Copy system keys for consistent access dropbear_rsa_key=/etc/dropbear/ssh_host_rsa_key dropbear_ecdsa_key=/etc/dropbear/ssh_host_ecdsa_key dropbear_ed25519_key=/etc/dropbear/ssh_host_ed25519_key # User zbmuser is the authorized unlocker here dropbear_acl=/home/zbmuser/.ssh/authorized_keys EOF ``` ### Configure EFI boot entries ```bash mount -t efivarfs efivarfs /sys/firmware/efi/efivars apt install efibootmgr -y efibootmgr -c -d "/dev/nvme0n1" -p 1 \ -L "ZFSBootMenu" \ -l \\EFI\\ZBM\\VMLINUZ.EFI efibootmgr -c -d /dev/nvme0n1 -p 1 \ -L "ZFSBootMenu (Backup)" \ -l \\EFI\\ZBM\\VMLINUZ-BACKUP.EFI ### Skip this section if you are not doing mirroring efibootmgr -c -d "/dev/nvme0n2" -p 1 \ -L "ZFSBootMenu 2" \ -l \\EFI\\ZBM\\VMLINUZ.EFI efibootmgr -c -d /dev/nvme0n2 -p 1 \ -L "ZFSBootMenu 2 (Backup)" \ -l \\EFI\\ZBM\\VMLINUZ-BACKUP.EFI ``` ### Set the root password Set the root password: ``` passwd ``` ### Exit the environment ``` exit umount -n -R /mnt zpool export zroot reboot ```