Installing Arch Linux with ZFS on a USB stick

Published on 2019-03-28. Modified on 2022-10-23.

In this tutorial I am going to show you how you can make a working Arch Linux USB stick with ZFS. This can be useful if you're running a homelab NAS and you want to boot of the USB stick and only use your drives for ZFS. It can also be used as a diagnostic or administration tool for dealing with ZFS on Arch Linux or some other Linux distribution.

Let me start by saying, running ZFS on Arch Linux is generally not a good idea. Not because it cannot be done, I have multiple servers that runs Arch Linux with ZFS, but the Arch Linux developers treat ZFS as a "second-class citizen" and as such, you either have to use something like zfs-dkms from AUR, use a non-official pacman repository like archzfs, or run your own custom repository. In either case, you need to keep an eye on things if you don't want stuff to break when you update the kernel.

TIP! Besides from all the Technical reasons to choose FreeBSD over GNU/Linux, if you want proper support for ZFS, run FreeBSD! FreeBSD treats ZFS as a true first-class citizen and no Linux distribution can compare in that regard. FreeBSD has tightly integrated ZFS into several of the system base tools. Even the top command displays ZFS ARC Stats.

Other and better options than Arch Linux are Void Linux, which has good support for ZFS. Also Debian and Devuan - if you enable their "contrib" repositories. Ubuntu also has good support for ZFS.

In this tutorial I am going to use two different methods in order to create a bootable USB stick. First I am going to use Archiso with the non-official archzfs respository to create an ISO which you can put unto an USB stick. Next I am going to use the method in which you mount an USB stick and install Arch Linux unto it from an already running Arch Linux machine with zfs-dkms installed from AUR.

The Archiso method

You can keep an eye on the build process of archzfs on https://ci.archzfs.com/.

First install Archiso:

# pacman -S archiso

Then create a temporary working directory and enter into that:

$ mkdir tmpwork
$ cd tmpwork

Copy the config files, syslinux, etc., into the directory (I am going to use the archlive version):

$ cp -r /usr/share/archiso/configs/releng/ archlive

Edit pacman.conf and enter the archzfs-testing repository from Archzfs in order to get the latest version of ZFS:

[archzfs-testing]
Server = https://archzfs.com/$repo/$arch

The ordering within pacman.conf matters. To give top priority to the Archzfs repository, place it above the other repository entries.

Edit packages.x86_64 to select which packages are to be installed on the live system image, listing packages line by line. I have added the following:

zfs-linux
linux-headers
neovim

Then become root, import the GPG key and build the ISO image:

$ su
# curl -O https://archzfs.com/archzfs.gpg
# pacman-key -a archzfs.gpg
# pacman-key --lsign-key DDF7DB817396A49B2A2723F7403BD972F75D9D76
# mkarchiso -v ./

When Archiso is done, it will have created a directory called out in which the resulting ISO file is placed. You can know use dd to put it unto an USB stick.

# dd if=out/archlinux-2022.10.23-x86_64.iso of=/dev/sdX bs=4M oflag=direct status=progress

Make sure you change the sdX to match your USB stick. You can check it by taking a look at the output of dmesg as root, when you insert the stick to the computer.

The running Arch Linux installation method

I am going to use the ext4 filesystem for the USB stick. The Arch Linux wiki recommends disabling the journal when using ext4 on a USB stick, but since all decent flash devices perform internal wear leveling I think the benefit of running with the journal outweighs the risk of premature wearing out the USB stick.

I am going to use the MBR setup and not UEFI in this tutorial.

First we need to install the Arch Linux install scripts:

# pacman -S arch-install-scripts

Next, locate your USB device:

# fdisk -l

In my case it's the sdc device.

Then partition the disk. If you're not comfortable using fdisk then cfdisk is a nice alternative partitioning tool.

# cfdisk /dev/sdc

Create a single bootable Linux partition, resulting in /dev/sdc1 in my case.

Create the ext4 filesystem:

# mkfs.ext4 /dev/sdc1

Use the following command instead if you want to disable the journal:

# mkfs.ext4 -O "^has_journal" /dev/sdc1

Then mount the USB stick:

# mount /dev/sdc1 /mnt

Then install the base package group, whatever custom packages you want, and set everything up:

# pacstrap /mnt base linux linux-firmware neovim cryptsetup grub
# arch-chroot /mnt
# ln -sf /usr/share/zoneinfo/Europe/Copenhagen /etc/localtime
# hwclock --systohc
# passwd root

Uncomment "en_US.UTF-8" and other needed locales in /etc/locale.gen, and generate them with:

# locale-gen

Create the /etc/locale.conf file and set the LANG variable accordingly:

LANG=en_US.UTF-8

Set the keyboard layout in /etc/vconsole.conf

KEYMAP=dk

Let's also get the base-devel package:

# pacman -S --needed base-devel

I enable SSH in order to be able to log in to the computer booting of the USB stick. This is handy if the computer is a homelab NAS running of the USB stick (otherwise you don't need this):

# systemctl enable sshd

Install additional users and setup the network (you don't need to install additional users if you just want to use the USB stick as a diagnostic tool, however it makes using makepkg a lot easier):

# useradd --create-home foo
# passwd foo

Setup basic networking:

# vi /etc/systemd/network/wired.network
[Match]
Name=en*
Name=eth*
[Network]
DHCP=yes

Make sure that the Name options matches all the networking cards you use. By default udev assigns names to your network interface controllers using Predictable Network Interface Names, which prefixes interfaces names with en (wired/Ethernet), wl (wireless/WLAN), or ww (WWAN). See systemd.net-naming-scheme(7).

Then enable the network:

# systemctl enable systemd-networkd.service
# systemctl enable systemd-resolved

Then setup GRUB:

# grub-install --target=i386-pc /dev/sdc --recheck
# grub-mkconfig -o /boot/grub/grub.cfg

I had some problems with GRUB not being able find the "root device" after using the USB stick on another computer than the one on which I installed the stick. This shouldn't be happening since the system is using UUID, which is a mechanism to give each filesystem a unique identifier. These identifiers are generated by filesystem utilities (e.g. mkfs.) when the device gets formatted and are designed so that collisions are unlikely. However, I found other users with similar problems: https://superuser.com/questions/769047/unable-to-find-root-device-on-a-fresh-archlinux-install

The solution is, as described in the link, to edit /etc/mkinitcpio.conf and change the HOOKS line so that block comes before autodetect.

HOOKS="base udev block autodetect modconf filesystems keyboard fsck"

Then run mkinitcpio to regenerate the initramfs:

# mkinitcpio -p linux

Then it's time to install ZFS.

I am using the zfs-dkms package from Arch Linux AUR and running makepkg as the foo user I setup before:

# pacman -S dkms linux-headers git
# su foo
$ cd /home/foo
$ git clone https://aur.archlinux.org/zfs-utils.git
$ cd zfs-utils
$ makepkg -s
$ cd ..
$ git clone https://aur.archlinux.org/zfs-dkms.git
$ cd zfs-dkms
$ makepkg -s
$ cd ..
$ exit
# pacman -U zfs-utils/zfs-utils-x.x.x-x-x86_64.pkg.tar.xz
# pacman -U zfs-dkms/zfs-dkms-x.x.x-x-x86_64.pkg.tar.xz

If you get the error:

==> Verifying source file signatures with gpg...
    zfs git repo ... FAILED (unknown public key DDF7DB817396A49B2A2723F7403BD972F75D9D76)
==> ERROR: One or more PGP signatures could not be verified!

Then manually import the key as explained in One or more PGP signatures could not be verified and Use a key server

$ gpg --recv-keys DDF7DB817396A49B2A2723F7403BD972F75D9D76

If that doesn't work, use:

# pacman-key -r DDF7DB817396A49B2A2723F7403BD972F75D9D76
# pacman-key --lsign-key DDF7DB817396A49B2A2723F7403BD972F75D9D76

You can add an IgnorePkg entry to pacman.conf to prevent these packages from upgrading when doing a regular update. Since the AUR is unsupported any packages you install are your responsibility to update, not pacman's. If packages in the official repositories are updated, you will need to rebuild any AUR packages that depend on those libraries.

# vi /etc/pacman.conf
IgnorePkg=zfs-dkms zfs-utils

For ZFS to live by its "zero administration" namesake, the zfs daemon must be loaded at startup. A benefit to this is that it is not necessary to mount the zpool in /etc/fstab The zfs daemon can import and mount zfs pools automatically. The daemon mounts the zfs pools reading the file /etc/zfs/zpool.cache. For each pool you want automatically mounted by the zfs daemon execute:

# zpool set cachefile=/etc/zfs/zpool.cache <pool>

Enable the service so it is automatically started at boot time:

# systemctl enable zfs.target

Note: Beginning with ZOL version 0.6.5.8 the ZFS service unit files have been changed so that you need to explicitly enable any ZFS services you want to run. See https://github.com/archzfs/archzfs/issues/72 for more information.

In order to mount zfs pools automatically on boot (you only need that if you use the USB stick as a boot device for a NAS server) you need to enable the following services and targets:

# systemctl enable zfs-import-cache
# systemctl enable zfs-mount
# systemctl enable zfs-import.target

We're done. Exit the chroot and un-mount the USB stick:

# exit
# umount /mnt

You should now have a working USB stick with Arch Linux and ZFS that you can use as a boot device or a diagnostic tool.