This article is about the boostrapping of NetBSD from source to allow / on ZFS.
To BSD or not to BSD
One of my more radical beliefs is that ideas can be good or truth outside of their time/space. I believe that UNIX is one of those ideas. Even tough most of the world’s infrastructure runs the Linux+Docker/Kubernetes stack, I actually belief the the vast majority of computing use-cases are better served with a basic UNIX system. Because of this belief, I nuked my Kubernetes control plane and deployed NetBSD.
Root on ZFS
Well… I did not just deploy NetBSD.
I have (because of my profession) some experience with ZFS and I really like it.
So I wanted to deploy NetBSD with root on ZFS.
NetBSD does not natively supports root on ZFS, but it does have a ZFS ramdisk (similair to initrd on Linux).
The documentation instructs you to make an FFS “recovery” partition that contains the Kernel, some drivers, and the ZFS ramdisk.
But on machines that have UEFI, you can put all these files in the ESP partition.
Thus I downloaded the NetBSD sources and used build.sh to compile the Kernel, the Solaris and ZFS drivers, and the ZFS ramdisk onto a newly created ZFS pool.
I copied the UEFI bootcode over from /usr/mdec, wrote a custom boot config file, put all of these files on the ESP partition of the SSD.
/
├── EFI
│ ├── BOOT
│ │ └── BOOTX64.EFI
│ └── NetBSD
│ └── boot.cfg
├── netbsd
├── stand
│ └── amd64
│ └── 11.0
│ └── modules
│ ├── solaris
│ │ └── solaris.kmod
│ └── zfs
│ └── zfs.kmod
└── zroot
Figure: The filesystem layout of the ESP partition; the /zroot file is the ZFS ramdisk and the /netbsd file is the NetBSD Kernel.
Having a Kernel with some drivers to boot ZFS is important, but you also need the rest of the system.
So I used build.sh again to compile (and install) the rest of NetBSD, after which I used the configuration menu of the NetBSD installer to do some final tweaks.
Turn me on, boot me up
After a reboot I found myself with a fresh NetBSD deployment that worked. So I copied my SSH keys, installed Neovim as my editor, and started typing away. To recap and to give some information about the deployment.
The computer is a Intel i5 Haswell CPU with 16 GB of memory. It has a 256 GB SATA SSD and a RTL8251 network interface.
start size index contents
0 1 PMBR
1 1 Pri GPT header
2 32 Pri GPT table
34 262144 1 GPT part - ESP
262178 4194304 2 GPT part - swap
4456482 495661677 3 GPT part - rpool
500118159 32 Sec GPT table
500118191 1 Sec GPT header
Figure: The layout of the SSD (/dev/wd0).
rpool/ROOT /
rpool/var /var
rpool/usr /usr
rpool/home /home
rpool/pkg /usr/pkg
Figure: The layout of the ZFS datasets and their mount points.
PID TTY STAT TIME COMMAND
0 ? DKl 0:04.61 [system]
1 ? Is 0:00.00 init
588 ? Ss 0:00.03 /usr/sbin/syslogd -s
992 ? Is 0:00.00 /usr/sbin/powerd
994 ? Ss 0:00.26 /usr/sbin/ntpd -p /var/run/ntpd.pid
1008 ? Is 0:00.03 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
1173 ? Is 0:00.02 /usr/sbin/cron
8582 ? Is 0:00.65 tmux: server (/tmp/tmux-1000/default) (tmux)
10032 ? Is 0:00.02 sshd-session: thijs [priv]
10304 ? Is 0:02.17 nvim --embed content/posts/poweredby.md
10472 ? S 0:01.46 sshd-session: thijs@pts/0 (sshd-session)
10842 ? I 0:00.00 qmgr -l -t unix -u
11042 ? I 0:00.01 pickup -l -t unix -u
11436 ? Is 0:00.00 /usr/libexec/postfix/master -w
11088 pts/0 Ss 0:00.01 -ksh
12078 pts/0 O+ 0:00.00 ps -A
9248 pts/1 Is 0:00.00 -ksh
10614 pts/1 I+ 0:00.18 nvim content/posts/poweredby.md
9099 pts/2 Is 0:00.01 -ksh
11498 pts/2 I+ 0:00.00 make serve
12331 pts/2 Il+ 0:00.08 hugo serve -D --bind 0.0.0.0
1064 ttyE0 Is+ 0:00.03 /usr/libexec/getty Pc constty
Figure: The activity on the system during a ‘regular’ day.
Closing remarks
- To use NetBSD is to live the UNIX philosophy.
- Pipes are amazing and I should explore them on a deeper level then ‘mere’ shell scripts.
- Event programming is cool, and I should be better at it.
- When NetBSD 11.0 is released, I want to do a NetBSD from source on Linux build.