Root on ZFS with FreeBSD 9

There are many guides for installing FreeBSD on a ZFS partition floating around on the web, however most of them were outdated or just didn’t quite meet my needs so I used them as a basis to write a new guide.

What makes this guide unique is that FreeBSD is installed inside of a filesystem on the zpool allowing for much more flexibility compared to installing directory to the root of the zpool. For instance, you can make a snapshot of the whole installation with one command, update the kernel and userland, and if for some reason something goes wrong, just revert to the old system with a single command!

Also, swap is kept on the ZFS partition as well, allowing you to easily resize or add swap space as needed.

You will need either the disk1.iso or memstick.img image of FreeBSD for this guide. I tested this guide with FreeBSD 9.0-rc3 but it should work with future versions as well.

Boot from the install disk and choose the “Live CD” option. Login as root.

This guide assumes your root disk is ada0, replace it with your real root disk.

First, we partition the disk. Repeat this process with another disk if you want to use a mirrored root partition.

# Create a GUID partition table on the disk
$ gpart create -s gpt ada0
# Add the boot partition
$ gpart add -s 64k -t freebsd-boot ada0
# Add the ZFS partition and label it as "root"
$ gpart add -t freebsd-zfs -l root ada0
# Write the boot loader to the disk
$ gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

If the first command gives you a gpart: geom 'ada0': File exists error, you need to erase the existing using dd if=/dev/zero of=/dev/ada0 bs=512 count=1.

Now we can create the zpool on the disk.

# Create the pool using the "root" label
# Note: For a mirror use zpool create zroot mirror /dev/gpt/root0 /dev/gpt/root1
$ zpool create zroot /dev/gpt/root
# Export and reimport the pool storing the cache in /tmp
$ zpool export zroot
$ zpool import -o cachefile=/tmp/zpool.cache zroot

Mount and set up the filesystems on the ZFS partition. These are just examples so modify them for your system.

$ zfs set mountpoint=none zroot
$ zfs create -o mountpoint=/mnt zroot/freebsd
$ zfs create zroot/freebsd/usr
$ zfs create zroot/freebsd/usr/home
$ zfs create zroot/freebsd/var
$ zfs create -o compression=on -o setuid=off zroot/freebsd/usr/ports
$ zfs create -o compression=off -o exec=off -o setuid=off zroot/freebsd/usr/ports/distfiles
$ zfs create -o compression=on zroot/freebsd/usr/src
$ zfs create -o compression=on -o exec=on -o setuid=off zroot/freebsd/tmp
$ zfs create -o compression=on -o exec=on -o setuid=off zroot/freebsd/var/tmp

Create a ZFS volume to use as swap.

$ zfs create -V 4G zroot/swap
$ zfs set org.freebsd:swap=on zroot/swap
$ zfs set checksum=off zroot/swap

Set the proper permissions on the tmp directories and add a symlink to the home directory.

$ chmod 1777 /mnt/tmp /mnt/var/tmp
$ cd /mnt; ln -s usr/home home; cd -

Set the boot volume.

$ zpool set bootfs=zroot/freebsd zroot

Now we can install the FreeBSD base system onto our ZFS partition. You can customize which sets you want to install. This may take a while to complete.

$ sh
$ cd /usr/freebsd-dist
$ export DESTDIR=/mnt
$ for file in base.txz doc.txz kernel.txz lib32.txz ports.txz src.txz;
$ > do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

Copy the zpool’s cache file to the new installation.

$ cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache

Enable ZFS to load on boot.

$ echo zfs_load=\"YES\" >> /mnt/boot/loader.conf
$ echo vfs.root.mountfrom=\"zfs:zroot/freebsd\" >> /mnt/boot/loader.conf
$ echo zfs_enable=\"YES\" >> /mnt/etc/rc.conf

FreeBSD will complain about a missing fstab, so make sure to create an empty one.

$ touch /mnt/etc/fstab

Almost done! Now we need to unmount the ZFS filesystems and set the proper mount points.

$ zfs umount -a
$ zfs set mountpoint=legacy zroot/freebsd
$ zfs set mountpoint=/tmp zroot/freebsd/tmp
$ zfs set mountpoint=/usr zroot/freebsd/usr
$ zfs set mountpoint=/var zroot/freebsd/var

Now reboot into your new FreeBSD install!

$ shutdown -r now

Once the system is back up, login as root and finish the installation by setting the root password and timezone.

$ passwd
$ tzsetup