Network boot kickoff.

Step number one for the firmware on any system is sending out a DHCP request, asking the DHCP server for an IP address, the boot server (called "next server" in dhcp terms) and the bootfile.

On success the firmware will contact the boot server, fetch the bootfile and hand over control to the bootfile. Traditional method to serve the bootfile is using tftp (trivial file transfer protocol). Modern systems support http too. I have an article on setting up the dhcp server for virtual machines you might want check out.

What the bootfile is expected to be depends on the system being booted. There are embedded systems -- for example IP phones -- which load the complete system software that way.

When booting UEFI systems the bootfile typically is an EFI binary. That is not the only option though, more on that below.

UEFI network boot with a boot loader.

The traditional way to netboot linux on UEFI systems is using a bootloader. The bootfile location handed out by the DHCP server points to the bootloader and is the first file loaded over the network. Typical choices for the bootloader are grub.efi, snponly.efi (from ipxe project) or syslinux.efi.

Next step is the bootloader fetching the config file. That works the same way the bootloader itself was loaded, using the EFI network driver provided by either the platform firmware (typically the case for onboard NICs) or via PCI option rom (plug-in NICs). The bootloader does not need its own network drivers.

The loaded config file controls how the boot will continue. This can be very simple, three lines asking the bootloader to fetch kernel + initrd from a fixed location, then start the kernel with some command line. This can also be very complex, creating an interactive menu system where the user has dozens of options to choose from (see for example netboot.xyz).

Now the user can -- in case the config file defines menus -- pick what he wants boot.

Final step is the bootloader fetching the kernel and initrd (again using the EFI network driver) and starting the kernel. Voila.

Boot loaders and secure boot.

When using secure boot there is one more intermediate step needed: The first binary needs to be be shim.efi, which in turn will download the actual bootloader. Most distros ship only grub.efi with a secure boot signature, which limits the boot loader choice to that.

Also all components (shim + grub + kernel) must come from the same distribution. shim.efi has the distro secure boot signing certificate embedded, so Fedora shim will only boot grub + kernel with a secure boot signature from Fedora.

Netbooting machines without EFI network driver.

You probably do not have to worry about this. Shipping systems with EFI network driver and UEFI network boot support is standard feature today, snponly.efi should be used for these systems.

When using older hardware network boot support might be missing though. Should that be the case the ipxe project can help because it also features a large collection of firmware network drivers. It ships an all-in-one EFI binary named ipxe.efi which includes the the bootloader and scripting features (which are in snponly.efi too) and additionally all the ipxe hardware drivers.

That way ipxe.efi can boot from the network even if the firmware does not provide a driver. In that case ipxe.efi itself must be loaded from local storage though. You can download the efi binary and ready-to-use ISO/USB images from boot.ipxe.org.

UEFI network boot with an UKI.

A UKI (unified kernel image) is an EFI binary bundle. It contains a linux kernel, an initrd, the command line and a few more optional components not covered here in sections of the EFI binary. Also the systemd efi stub, which handles booting the bundled linux kernel with the bundled initrd.

One advantage is that the secure boot signature of an UKI image will cover all components and not only the kernel itself, which is a big step forward for linux boot security.

Another advantage is that a UKI is self-contained. It does not need a bootloader which knows how to boot linux kernels and handle initrd loading. It is simply an EFI binary which you can start any way you want, for example from the EFI shell prompt.

The later makes UKIs interesting for network booting, because they can be used as bootfile too. The DHCP server hands out the UKI location, the UEFI firmware fetches the UKI and starts it. Done.

Combining the bootloader and UKI approaches is possible too. UEFI bootloaders can load not only linux kernels. EFI binaries (including UKIs) can be loaded too, in case of grub.efi with the chainloader command. So if you want interactive menus to choose an UKI to boot you can do that.

UEFI network boot with an ISO image.

Modern UEFI implementations can netboot ISO images too. Unfortunately there are a few restrictions though:

  1. It is a relatively new feature. It exists for a few years already in edk2, but with the glacial speeds firmware feature updates are delivered (if at all) this means there is hardware in the wild which does not support this.
  2. It is only supported for HTTP boot. Which makes sense given that ISO images can be bulky and the http protocol typically is much faster than the tftp protocol used by PXE boot. Nevertheless you might need additional setup steps because of this.

When the UEFI firmware gets an ISO image as bootfile from the DHCP server it will load the image into a ramdisk, register the ramdisk as block device and try to boot from it.

From that point on booting works the same way booting from a local cdrom device works. The firmware will look for the boot loader on the ramdisk and load it. The bootloader will find the other components needed on the ramdisk, i.e. kernel and initrd in case of linux. All without any network access.

The UEFI firmware will also create ACPI tables for a pseudo nvdimm device. That way the booted linux kernel will find the ramdisk too. You can use the standard Fedora / CentOS / RHEL netinst ISO image, linux will find the images/install.img on the ramdisk and boot up all the way to anaconda. With enough RAM you can even load the DVD with all packages, then do the complete system install from ramdisk.

The big advantage of this approach is that the netboot workflow becomes very simliar to other installation workflows. It's not the odd kid on the block any more where loading kernel and initrd works in a completely different way. The very same ISO image can be:

  1. Burned to a physical cdrom and used to boot a physical machine.
  2. In many cases the ISO images are hypbrid, so they can be flashed to a USB stick too for booting a physical machine.
  3. The ISO image can be attached as virtual device to a virtual machine.
  4. On server grade managed hardware the ISO image can be attached as virtual media using redfish and the BMC.
  5. And finally: The ISO image can be loaded into a ramdisk via UEFI http boot.

Bonus: secure boot support suddenly isn't a headace any more.

The kernel command line.

There is one problem with the fancy new world though. We have lots of places in the linux world which depend on the linux kernel command line for system configuration. For example anaconda expects getting the URL of the install repository and the kickstart file that way.

When using a boot loader that is simple. The kernel command line simply goes into the boot loader config file.

With ISO images it is more complicated, changing the grub config file on a ISO image is a cumbersome process. Also ISO images are not exactly small, so install images with customized grub.cfg need quite some storage space.

UKIs can pass through command line arguments to the linux kernel, but that is only allowed in case secure boot is disabled. When using UKIs with secure boot the best option is to use the UKIs built and signed on distro build infrastructure. Which implies using the kernel command line for customization is not going to work with secure boot enabled.

So, all of the above (and UKIs in general) will work better if we can replace the kernel command line as universal configuration vehicle with something else. Which most likely will not be a single thing but a number of different approaches depending on the use case. Some steps into that direction did happen already. Systemd can autodetect partitions (so booting from disk without root=... on the kernel command line works). And systemd credentials can be used to configure some aspects of a linux system. There is still a loooong way to go though.