PrivSec-dev/privsec.dev
1
0
Fork 0
mirror of https://github.com/PrivSec-dev/privsec.dev synced 2024-11-10 00:31:34 -05:00
Code
d68d4f2e08
privsec.dev/content/posts/linux/Root ZFS Encryption, Mirroring, and Remote Unlocking with Ubuntu.md
Tommy d68d4f2e08
Update Root ZFS Encryption, Mirroring, and Remote Unlocking with Ubuntu.md 
Signed-off-by: Tommy <contact@tommytran.io>
2023-10-21 23:02:25 -07:00

13 KiB
Raw Blame History

title tags date author
Root ZFS Encryption, Mirroring, and Remote Unlocking on Ubuntu
Operating Systems
Linux
Security
2023-07-26 Tommy

While Unbutu supports ZFS on root filesystems with an easy snapshot and rollback mechanism called ZSYS, ZSYS is soon going to be deprecated and the installer does not offer an easy way to setup mirroring. In this guide, I will walk you through how to set up Ubuntu with root on ZFS, mirroring with both the root and EFI partitions, and remote unlocking + boot into a snapshot with ZFSBootMenu.

ZFSBootMenu

Enter the Shell

Enter the shell on your Ubuntu Installer:

Partitioning the Disk

Esentially, we need a 512MB ESP partition for ZFSBootMenu and a / partition for the rest of the system. If you are using a single disk, just make those 2 partitions on your disk. If you are planning to do mirroring, set up both on of these partitions on each disk.

There are a variety of tools you can use, but an easy to use one would be cfdisk.

cfdisk /dev/nvme0n1
cfdisk /dev/nvme0n2

cfdisk

Mirroring the ESP partition

Skip this if you are not doing mirroring

While the EFI specs do not support mdadm, we can setup mdadm with metadata v1.0, which will be put at the end of the parition and allows it to boot.

mdadm --create /dev/md0 --level 1 --raid-disks --metadata 1.0 /dev/nvme0n1p1 /dev/nvme0n1p2
mkfs.fat -F 32 /dev/md0

Setup the ZFS partition

This part is mostly based on the official ZFSBootMenu guide with some changes to work around some not-so-great instructions there.

Creating the zpool

Getting the Disk ID.

First, we must get the disk IDs from /dev/disk/by-id. The official guide uses the dynamically assigned drive identifier (/dev/sda, /dev/nvme0n1, etc), which is not what we want to do with zpools, since it will cause problems later on.

/dev/disk/by-id

Installing ZFS-Utils

sudo apt install zfsutils-linux -y

Create the encryption key

echo 'SomeKeyphrase' > /etc/zfs/zroot.key
chmod 000 /etc/zfs/zroot.key

For Non-Mirrored Setups

sudo zpool create -o ashift=12 -O compression=zstd -O acltype=posixacl -O xattr=sa -O atime=off -O encryption=on -O keylocation=file:///etc/zfs/zroot.key -O keyformat=passphrase -o autotrim=on -o failmode=panic compatibility=openzfs-2.1-linux -m none zroot /dev/disk/by-id/nvme-SAMSUNG_MZQL21T9HCJR-00A07_XXXXXXX-part2

For Mirrored Setups

zpool create  -o ashift=12  -O compression=zstd  -O acltype=posixacl  -O xattr=sa  -O atime=off  -O encryption=on  -O keylocation=file:///etc/zfs/zroot.key  -O keyformat=passphrase  -o autotrim=on  -o failmode=panic compatibility=openzfs-2.1-linux -m none zroot mirror /dev/disk/by-id/nvme-SAMSUNG_MZQL21T9HCJR-00A07_XXXXXXX-part2 /dev/disk/by-id/nvme-SAMSUNG_MZQL21T9HCJR-00A07_YYYYYYY-part2

Notes

We use slightly different options than the official guide. Most notably, atime is disabled as it has detrimental effect on performance and unnecessarily increases write operations. compression is changed from lz4 to zstd as it has much better compression ratio than lz4 while still maintaining good performance. We did not specify the encryption type here as aes-256-gcm is already the default with openZFS >= 0.8.4. failmode=panic makes sure that the kernel panics in case there is something wrong with the drive instead of taking risks with abnormal behavior. We specify compatibility=openzfs-2.1-linux to make sure that updates will not make the system unbootable in the future.

Creating the filesystems

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/ubuntu
zfs create -o mountpoint=/home zroot/home
zfs create -o mountpoint=/var/log zroot/ROOT/ubuntu/log
zfs create -o mountpoint=/var/spool zroot/ROOT/ubuntu/spool
zfs create -o mountpoint=/var/cache zroot/ROOT/ubuntu/cache

zpool set bootfs=zroot/ROOT/ubuntu zroot

Here, we deviate from the official guide by splitting out /var/log, /var/spool, /var/cache out into their own datasets. These are directories which are parts of Ubuntu that we do not want to be rolled back along with the system in case we need to boot into a prior snapshot.

Mounting the filesystem

zpool export zroot
zpool import -N -R /mnt zroot
zfs load-key -L prompt zroot
zfs mount zroot/ROOT/ubuntu
zfs mount zroot/ROOT/ubuntu/log
zfs mount zroot/ROOT/ubuntu/cache
zfs mount zroot/ROOT/ubuntu/spool
zfs mount zroot/home

Update device symlink

udevadm trigger

Install Ubuntu

We will deviate from the ZFSBootMenu's documentation here, as it only installs a minimal system with SysVinit. Instead, we can install ubuntu-server-minimal.

Bootstrapping

zgenhostid -f 0x00bab10c
apt install -y debootstrap
debootstrap jammy /mnt

Copy files into the new install

cp /etc/hostid /mnt/etc/hostid
cp /etc/resolv.conf /mnt/etc/
mkdir /mnt/etc/zfs
cp /etc/zfs/zroot.key /mnt/etc/zfs

Chroot into the new OS

mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -B /dev /mnt/dev
mount -t devpts pts /mnt/dev/pts
chroot /mnt /bin/bash

Setup the repositories

Use the source list from the Ubuntu live ISO:

cat <<EOF > /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu/ jammy main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ jammy universe
# deb-src http://archive.ubuntu.com/ubuntu/ jammy universe
deb http://archive.ubuntu.com/ubuntu/ jammy-updates universe
# deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://archive.ubuntu.com/ubuntu/ jammy multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ jammy multiverse
deb http://archive.ubuntu.com/ubuntu/ jammy-updates multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu/ jammy-security main restricted
# deb-src http://security.ubuntu.com/ubuntu/ jammy-security main restricted
deb http://security.ubuntu.com/ubuntu/ jammy-security universe
# deb-src http://security.ubuntu.com/ubuntu/ jammy-security universe
deb http://security.ubuntu.com/ubuntu/ jammy-security multiverse
# deb-src http://security.ubuntu.com/ubuntu/ jammy-security multiverse
EOF

Install the necessary packages

apt update
apt install --no-install-recommends linux-generic ubuntu-server-minimal

Configure packages to customize local and console properties

dpkg-reconfigure locales tzdata keyboard-configuration console-setup

ZFS Configuration

Install required packages

apt install dosfstools zfs-initramfs zfsutils-linux -y

Enable systemd ZFS services

systemctl enable zfs.target
systemctl enable zfs-import-cache
systemctl enable zfs-mount
systemctl enable zfs-import.target

Enable systemd-networkd

systemctl enable systemd-networkd

Configure initramfs-tools

echo "UMASK=0077" > /etc/initramfs-tools/conf.d/umask.conf

Rebuild the initramfs

update-initramfs -c -k all

Install and configure ZFSBootMenu

Setup the EFI partition

If you are doing mirroring:


cat << EOF >> /etc/fstab
$( blkid | grep /dev/md0 | cut -d ' ' -f 2 ) /boot/efi vfat defaults 0 0
EOF

mkdir -p /boot/efi
mount /boot/efi

If you are not, just replace md0 in the commands above with your efi partition.

Set ZFSBootMenu properties

Next, we will set the kernel boot parameters and the encryption key source for ZFSBootMenu. Here, we will deviate from the official guide and use a hardened boot parameter for better security:

zfs set org.zfsbootmenu:commandline="quiet loglevel=4 spectre_v2=on spec_store_bypass_disable=on l1tf=full,force mds=full,nosmt tsx=off tsx_async_abort=full,nosmt kvm.nx_huge_pages=force nosmt=force l1d_flush=on mmio_stale_data=full,nosmt random.trust_bootloader=off random.trust_cpu=off intel_iommu=on amd_iommu=on efi=disable_early_pci_dma iommu.passthrough=0 iommu.strict=1 slab_nomerge init_on_alloc=1 init_on_free=1 pti=on vsyscall=none page_alloc.shuffle=1 randomize_kstack_offset=on extra_latent_entropy debugfs=off" zroot/ROOT
zfs set org.zfsbootmenu:keysource="zroot/ROOT/ubuntu" zroot

Install ZFSBootMenu

To use it without remote unlocking, just follow the official guide:

apt install curl -y
mkdir -p /boot/efi/EFI/ZBM
curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi
cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI

To use it with remote unlocking, you have to compile the package:

git clone https://github.com/zbm-dev/zfsbootmenu
cd zfsbootmenu
make
make install

echo 'Global:
  ManageImages: true
  BootMountPoint: /boot/efi
  DracutConfDir: /etc/zfsbootmenu/dracut.conf.d
  PreHooksDir: /etc/zfsbootmenu/generate-zbm.pre.d
  PostHooksDir: /etc/zfsbootmenu/generate-zbm.post.d
  InitCPIOConfig: /etc/zfsbootmenu/mkinitcpio.conf
Components:
  ImageDir: /boot/efi/EFI/zbm
  Versions: 3
  Enabled: false
  syslinux:
    Config: /boot/syslinux/syslinux.cfg
    Enabled: false
EFI:
  ImageDir: /boot/efi/EFI/zbm
  Versions: false
  Enabled: true
Kernel:
  CommandLine: ro quiet loglevel=0 quiet loglevel=4 spectre_v2=on spec_store_bypass_disable=on l1tf=full,force mds=full,nosmt tsx=off tsx_async_abort=full,nosmt kvm.nx_huge_pages=force nosmt=force l1d_flush=on mmio_stale_data=full,nosmt random.trust_bootloader=off random.trust_cpu=off intel_iommu=on amd_iommu=on efi=disable_early_pci_dma iommu.passthrough=0 iommu.strict=1 slab_nomerge init_on_alloc=1 init_on_free=1 pti=on vsyscall=none page_alloc.shuffle=1 randomize_kstack_offset=on extra_latent_entropy debugfs=off' | tee /etc/zfsbootmenu/config.yaml

git clone https://github.com/dracut-crypt-ssh/dracut-crypt-ssh
apt install -y libblkid-dev
cd dracut-crypt-ssh
./configure
make
make install
echo 'omit_dracutmodules+=" crypt-ssh "' >> /etc/dracut-config-location-idk
mkdir -p /etc/dropbear
ssh-keygen -t rsa -m PEM -f /etc/dropbear/ssh_host_rsa_key
ssh-keygen -t ecdsa -m PEM -f /etc/dropbear/ssh_host_ecdsa_key
ssh-keygen -t ed25519 -m PEM -f /etc/dropbear/ssh_host_ed25519_key
mkdir -p /etc/cmdline.d
echo "ip=dhcp rd.neednet=1" > /etc/cmdline.d/dracut-network.conf

cat <<EOF > /etc/zfsbootmenu/dracut.conf.d/dropbear.conf
# Enable dropbear ssh server and pull in network configuration args
add_dracutmodules+=" crypt-ssh "
install_optional_items+=" /etc/cmdline.d/dracut-network.conf "
# Copy system keys for consistent access
dropbear_rsa_key=/etc/dropbear/ssh_host_rsa_key
dropbear_ecdsa_key=/etc/dropbear/ssh_host_ecdsa_key
dropbear_ed25519_key=/etc/dropbear/ssh_host_ed25519_key
# User zbmuser is the authorized unlocker here
dropbear_acl=/home/zbmuser/.ssh/authorized_keys
EOF

generate-zbm

Configure EFI boot entries

mount -t efivarfs efivarfs /sys/firmware/efi/efivars
apt install efibootmgr -y

efibootmgr -c -d "/dev/nvme0n1" -p 1 \
  -L "ZFSBootMenu" \
  -l \\EFI\\ZBM\\VMLINUZ.EFI

efibootmgr -c -d /dev/nvme0n1 -p 1 \
  -L "ZFSBootMenu (Backup)" \
  -l \\EFI\\ZBM\\VMLINUZ-BACKUP.EFI

### Skip this section if you are not doing mirroring

efibootmgr -c -d "/dev/nvme0n2" -p 1 \
  -L "ZFSBootMenu 2" \
  -l \\EFI\\ZBM\\VMLINUZ.EFI

efibootmgr -c -d /dev/nvme0n2 -p 1 \
  -L "ZFSBootMenu 2 (Backup)" \
  -l \\EFI\\ZBM\\VMLINUZ-BACKUP.EFI

Set the root password

Set the root password:

passwd

Remove grub

apt purge grub* -y

Exit the environment

exit
umount -n -R /mnt
zpool export zroot
reboot