Now I finally decided to not try rebuild the server, be it with or without cgit web frontend. I don't feel like taking up the fight with the scrapers in my spare time, I leave that to people who are in a better position to do so. Most repositories had mirrors on one or two of the large gitforges already. Those are the primary repositories now. Go look at gitlab and github.
Last week I've fixed all (I hope) dangeling links to the cgit repsitories to point to the forges instead.
Now I'm down to one self-hosted service, which is the webserver hosting mainly this blog and a few more little things. In 2018 I've migrated the blog from wordpress to jekyll, so it is all static pages. Taking this out by AI scrapers overloading the machine should be next to impossible, and so far this has hold up.
Nevertheless AI scrapers already managed to trigger one outage. Apparently millions of 404 answers where not enough to convince the bots that there is no cgit service (any more). Apache had no problems to deliver those, but the logs have filled up the disk so fast that logrotate didn't manage to keep things under control with the default configuration. Fixed config. Knook wood.
¹ Title inspired by
the 2025
edition of Security Nightmares. Fun watching if you speak
german.
² Most inefficient way to get the complete repo. Just clone it, ok?
IGVM is a file format for firmware images. Microsoft has published a git repository with rust crates to generate and parse igvm files. An C library is available as well. The igvm_defs crate defines all the data structures of the file format.
Main motivation for this new format is confidential computing, even though IGVM supports the 'NATIVE' platform (aka non-confidential VMs) too. When setting up a confidential VM the hypervisor needs to know a number of details, for example which pages should be included in the launch measurement, where are special pages like the SEV-SNP cpuid page, and more. OVMF has defined a metadata block at the end of the firmware image where that kind of information is stored. IGVM provides a standard way to store that information in the firmware image.
IGVM also is alot more flexible than traditional firmware images. Traditional firmware images essentially mimic what physical hardware is doing: A single block of memory is mapped as ROM or flash below 4G into the virtual machine address space. IGVM allows to load data into guest ram at any location (in addition to filling the traditional place below 4G of course).
The virt-firmware-rs project is a collection of rust crates for virtual machine firmware. Some are for the host, some for the guest. Most interesting for this blog article are the igvm-tools.
igvm-inspect is the first tool in that collection. It
will parse igvm files and print the content of the file in a compact
format to the console. Here is an example:
$ igvm-inspect /usr/share/edk2/ovmf/OVMF.igvm
INFO inspecting /usr/share/edk2/ovmf/OVMF.igvm
platform: NATIVE, compat 0x1
platform: SEV_SNP, compat 0x2
SnpVpContext: rm16, cs:ip f000:fff0, compat 0x2 [SEV_SNP]
PageData: gpa 0x800000, type NORMAL, flags 4k, empty, compat 0x2 [SEV_SNP]
PageData: end 0x808fff (8 more pages)
PageData: gpa 0x80a000, type NORMAL, flags 4k, empty, compat 0x2 [SEV_SNP]
PageData: end 0x80cfff (2 more pages)
PageData: gpa 0x80d000, type SECRETS, flags 4k, empty, compat 0x2 [SEV_SNP]
PageData: gpa 0x80e000, type CPUID_DATA, flags 4k, empty, compat 0x2 [SEV_SNP]
PageData: gpa 0x811000, type NORMAL, flags 4k, empty, compat 0x2 [SEV_SNP]
PageData: end 0x82ffff (30 more pages)
PageData: gpa 0xe0000, type NORMAL, flags 4k, compat 0x3 [NATIVE, SEV_SNP]
PageData: end 0xfffff (31 more pages)
PageData: gpa 0xffc84000, type NORMAL, flags 4k, compat 0x3 [NATIVE, SEV_SNP]
PageData: end 0xffffffff (891 more pages)
PageData: gpa 0xffc00000, type NORMAL, flags 4k, compat 0x3 [NATIVE, SEV_SNP]
PageData: end 0xffc83fff (131 more pages)In this listing you can see some features of the IGVM format. A single file can support multiple platforms. Which file directives are needed for which platform is specified with a compatibility bitmask. There are different page types. "SnpVpContext" is the VMSA (aka initial vCPU state) for the firmware.
Next tool is igvm-wrap, which can convert traditional
firmware images into IGVM. The file inspected above has been
created with that tool. The image is simply mapped below 4G (with
the topmost 128k being mirrored below 1M in real mode adress space).
The tool can parse the OVMF metadata section and write that
information into IGVM directives. By default the generated file has
only support for the native platform, but if supported by the
firmware you can turn on snp support. In case you have separate
images for firmware code and efi variable store the latter can be
added to the command line too:
$ igvm-wrap \
--input /usr/share/edk2/ovmf/OVMF_CODE.fd \
--vars /usr/share/edk2/ovmf/OVMF_VARS.fd \
--output ovmf.igvm \
--snp
INFO reading /usr/share/edk2/ovmf/OVMF_CODE.fd
INFO reading /usr/share/edk2/ovmf/OVMF_VARS.fd
INFO writing ovmf.igvmAll of the above works fine without changes to the edk2 code base. But by adding IGVM support in the firmware code we can do more than that.
IGVM has the concept of parameters to pass information about the virtual machine configuration from the hypervisor to the firmware. For example the memory map and the number of vCPUs. These two are parameters are supported now by OVMF (edk2-stable202511 or newer). There is a reserved page at a fixed location for them, and in case the hypevisor has filled the parameters OVMF will used them for memory detection and MP initialization.
There is a new block in the OVMF metadata area declaring the parameter page and the parameters so tools like igvm-wrap can find them and generate the corresponding IGVM parameter declarations. Looks like this (needs virt-firmware-rs version 25.10 or newer):
$ igvm-wrap --input OVMF.fd --output ovmf.igvm
INFO reading OVMF.fd
INFO writing ovmf.igvm
$ igvm-inspect ovmf.igvm
INFO inspecting ovmf.igvm
platform: NATIVE, compat 0x1
ParameterArea: idx 0, bytes 0x1000
Parameter/MemoryMap: idx 0, offset 0x0
Parameter/VpCount: idx 0, offset 0xc0
ParameterInsert: gpa 0x812000, idx 0, compat 0x1 [NATIVE]
PageData: gpa 0xe0000, type NORMAL, flags 4k, compat 0x1 [NATIVE]
[ ... ]OVMF already has a number of different ways to detect memory, depending on the environment it has been booted in (qemu, cloud hypervisor, Xen, ...). So why add another one?
As mentioned above IGVM allows to load data anywhere into guest memory. So we can put additional things into the firmware image. Of course the firmware needs some way to learn about this additional data for this being actually useful. So OVMF got a second special page where it will look for Hand Off Blocks (or HOBs for short), describing where the firmware can find the additional data and what kind of data that is.
So, what "additional things" are supported by OVMF?
Time to introduce the
next virt-firmware-rs
igvm utility: igvm-update. This can add all of the
above to an igvm image (also needs edk2-stable202511 or newer and
virt-firmware-rs 25.10 or newer).
$ igvm-update \
--input ovmf.igvm \
--kernel /boot/vmlinuz-$(uname -r) \
--output ovmf-kernel.igvm
INFO reading ovmf.igvm
INFO reading /boot/vmlinuz-6.16.9-200.fc42.x86_64
INFO add kernel efi binary
INFO add efidata hoblist
INFO writing ovmf-kernel.igvmVoila, now you have an igvm image which boots straight into the included linux kernel.
Why even bother with secure boot in confidential VMs? We have launch measurements, right? Well, problem is, doing the launch measurements is a relatively slow process. So it is desireable to minimize the amount data which must be covered by the launch measurement.
So, the design idea here is to include the firmware itself and the secure boot configuration in the launch measurement. But the UKI containing linux kernel and initrd is not included, this is verified using secure boot instead.
There are two possible ways to handle secure boot verification:
--add-cert-ms) and shim binary (--shim
/path/to/shimx64.efi) to the igvm file.
--add-hash). That way secure boot will allow
that one kernel to be loaded. The concept is very simliar to the
'kernel-hashes=on' logic
in qemu
amd sev support code, except that it uses bog standard secure
boot verification instead of some custom hash verification logic,
and it also expects UKIs are used to make sure secure boot
verification covers initrd and kernel command line too.
That's it. Have fun checking out the new features.
]]>Lets start with some tech background and history, which is helpful to understand the chain of events leading to CVE-2025-2296.
The x86 linux kernel has a 'setup' area at the start of the binary, and the traditional role for that area is to hold information needed by the linux kernel to boot, for example command line and initrd location. The boot loader patches the setup header before handing over control to the kernel, which allows the linux kernel to find the command line you have typed into the boot loader prompt. Booting in BIOS mode still works that way, and will most likely continue to do so until the BIOS mode days are numbered.
In the early days of UEFI support the boot process in UEFI mode worked quite simliar. It's known as 'EFI handover protocol'. It turned out to have a number of disadvantages though, for example passing additional information requires updating both linux kernel and the boot loader. The latter is true for BIOS mode too, but new development there are very rare with the world moving towards UEFI.
Enter 'EFI stub'. With this the linux kernel is simply an EFI binary. Execution starts in EFI mode, so the early kernel code can do EFI calls and gather all information needed to boot the kernel without depending on the boot loader to do this for the kernel. Version dependencies are gone. Additional bonus is that no kernel-specific knowledge is needed any more. Anything which is able to start efi binaries -- for example efi shell -- can be used to launch the kernel.
Qemu offers the option to launch linux kernels directly, using
the -kernel command line switch. What actually happens
behind the scenes is that qemu exposes the linux kernel to the guest
using the firmware config interface (fw_cfg for short). The virtual
machine firmware (SeaBIOS or OVMF) will fetch the kernel from qemu,
place it in guest memory and launch it.
OVMF supports both 'EFI stub' and 'EFI handover protocol' boot methods. It will try the modern 'EFI stub' way first, which actually is just 'try start as EFI binary'. Which btw. implies that you can load any EFI binary that way, this is not limited to linux kernels.
If starting the kernel as EFI binary fails OVMF will try to fallback to the old 'EFI handover protocol' method. OVMF names the latter 'legacy linux kernel loader' in messages printed to the screen.
So, what is the problem with secure boot? Well, there isn't only one problem, we had multiple issues:
Secure boot bypass sounds scary, but is it really?
First, the bypass is restricted to exactly one binary, which is the linux kernel the firmware fetches from qemu. This issue does not allow to run arbitrary code inside the guest, for example some unsigned efi binary an attacker places on the ESP after breaking into the virtual machine.
Second, the guest has to trust the virtualization host to not do evil things. The host has full control over the environment the guest is running in. Providing the linux kernel image for direct kernel boot is only one little thing out of many. If an evil host wants attack/disturb the guest there are plenty of ways to do so. The host does not need some exploit for that.
Third, many typical virtual machine configurations do not use direct kernel boot. The kernel is loaded from the guest disk instead.
So, the actual impact is quite limited.
There is no quick and easy fix available. Luckily it is also not super urgent and critical. Time to play the long game ...
Fix #1: qemu exposes an unmodified kernel image to the guest now (additionally to the traditional setup/rest split which is kept for BIOS mode and backward compatibility). Fixes the first issue.
Fix #2: qemu can expose the shim binary to the guest, using the
new -shim command line switch. Fixes the third issue.
Both qemu changes are available in qemu version 10.0 (released April
2025) and newer.
Fix #3: OVMF companion changes for fixes #1 + #2, use the new fw_cfg items exposed by qemu. Available in edk2-stable202502 and newer. Both qemu and OVMF must be updated for this to work.
Fix #4: Add a config option to disable the legacy 'EFI handover protocol' loader. Leave it enabled by default because of the low security impact and the existing use cases, but print warnings to the console in case the legacy loader is used. Also present in edk2-stable202502 and newer. Addresses the second issue.
With all that in place it is possible to plug the CVE-2025-2296 hole, by flipping the new config switch to disabled.
Fix #5: Do not use the legacy loader in confidential VMs. Present in edk2-stable202511 and newer.
Roughly one year has been passed since the first changes have been committed to qemu and edk2. What happened? libvirt also got support for passing shim to the guest for direct kernel boot (version 11.2.0 and newer). The new versions have found their way into the major linux distributions. debian, ubuntu and fedora should all be ready for the next step now.
Fix #6: Flip the default value for the legacy loader config option to disabled. This update just landed in the edk2 git repository and will be in edk2-stable202602.
What is left to do? The final cleanup. Purge the legacy loader from the edk2 code base. Will probably happen a year or two down the road.
The edk2 changes are in X86QemuLoadImageLib.
The qemu changes are in hw/i386/x86-common.c.
]]>
On the x86 platform qemu provides a isa-debugcon
device. That is the simplest possible logging device, featuring a
single ioport. Reading from the ioport returns a fixed value,
which can be used to detect whenever the device is present or not.
Writing to the ioport sends the character written to the chardev
backend linked to device.
By convention the qemu firmware -- both seabios and OVMF -- uses the ioport address 0x402. So getting the firmware log printed on your terminal works this way:
qemu-system-x86_64 \
-chardev stdio,id=fw \
-device isa-debugcon,iobase=0x402,chardev=fw
When using libvirt you can use this snippet (in
the <devices> section) to send the firmware log
to file:
<serial type='null'>
<log file='/path/to/firmware.log' append='off'/>
<target type='isa-debug' port='1'>
<model name='isa-debugcon'/>
</target>
<address type='isa' iobase='0x402'/>
</serial>
Note that virsh console will connect to the first
serial device specified in the libvirt xml, so this should be
inserted after other serial devices to avoid breaking your
serial console setup.
On the arm virt platform there is no special device for the firmware log. The logs are sent to the serial port instead. That is inconvinient when using a serial console though, so linux distros typically ship two variants of the arm firmware image. One with logging turned on, and one with logging turned off (on RHEL and Fedora the latter have 'silent' in the filename). By default libvirt uses the silent builds, so in case you need the debug log you have to change the VM configuration to use the other variant.
Recently (end of 2023) the arm builds learned a new trick. In case two serial ports are present the output will be split. The first serial port is used for the console, the second port for the debug log. With that the console is actually usable with the verbose firmware builds. To enable that you simply need two serial devices in your libvirt config:
<serial type='pty'>
<log file='/path/to/console.log' append='off'/>
<target type='system-serial' port='0'>
<model name='pl011'/>
</target>
</serial>
<serial type='null'>
<log file='/path/to/firmware.log' append='off'/>
<target type='system-serial' port='1'>
<model name='pl011'/>
</target>
</serial>
Starting with the edk2-stable202508 tag OVMF supports
logging to a memory buffer. The feature is disabled by default and
must be turned on at compile time using the -D
DEBUG_TO_MEM=TRUE option when building the firmware.
There are multiple ways to access the log memory buffer. First is
a small efi application which can print the log to the efi console
(source
code,
x64
binary). Pass -p or pager as argument
on the command line to enable the build-in pager.
Second way is a recent linux kernel, version 6.17 got a new bool
config option: OVMF_DEBUG_LOG. When enabled the linux
kernel will make the firmware log available via sysfs. If supported
by both kernel and firmware the log will show up in
the /sys/firmware/efi directory with the
filename ovmf_debug_log.
Third option is a qemu monitor command. The changes just landed in
qemu master branch and will be available in the next release qemu
(10.2) expected later this year. Both a qmp command
(query-firmware-log) and a hmp command (info
firmware-log) are availablke. This is useful to diagnose
firmware failures happen early enough in boot that the other options
can not be used.
Step number one for the firmware on any system is sending out a DHCP request, asking the DHCP server for an IP address, the boot server (called "next server" in dhcp terms) and the bootfile.
On success the firmware will contact the boot server, fetch the bootfile and hand over control to the bootfile. Traditional method to serve the bootfile is using tftp (trivial file transfer protocol). Modern systems support http too. I have an article on setting up the dhcp server for virtual machines you might want check out.
What the bootfile is expected to be depends on the system being booted. There are embedded systems -- for example IP phones -- which load the complete system software that way.
When booting UEFI systems the bootfile typically is an EFI binary. That is not the only option though, more on that below.
The traditional way to netboot linux on UEFI systems is using a
bootloader. The bootfile location handed out by the DHCP server
points to the bootloader and is the first file loaded over the
network. Typical choices for the bootloader
are grub.efi, snponly.efi
(from ipxe project)
or syslinux.efi.
Next step is the bootloader fetching the config file. That works the same way the bootloader itself was loaded, using the EFI network driver provided by either the platform firmware (typically the case for onboard NICs) or via PCI option rom (plug-in NICs). The bootloader does not need its own network drivers.
The loaded config file controls how the boot will continue. This can be very simple, three lines asking the bootloader to fetch kernel + initrd from a fixed location, then start the kernel with some command line. This can also be very complex, creating an interactive menu system where the user has dozens of options to choose from (see for example netboot.xyz).
Now the user can -- in case the config file defines menus -- pick what he wants boot.
Final step is the bootloader fetching the kernel and initrd (again using the EFI network driver) and starting the kernel. Voila.
When using secure boot there is one more intermediate step needed:
The first binary needs to be be shim.efi, which in turn
will download the actual bootloader. Most distros ship
only grub.efi with a secure boot signature, which
limits the boot loader choice to that.
Also all components (shim + grub + kernel) must come from the same
distribution. shim.efi has the distro secure boot
signing certificate embedded, so Fedora shim will only boot
grub + kernel with a secure boot signature from Fedora.
You probably do not have to worry about this. Shipping systems with
EFI network driver and UEFI network boot support is standard feature
today, snponly.efi should be used for these systems.
When using older hardware network boot support might be missing
though. Should that be the case
the ipxe project can help because it
also features a large collection of firmware network drivers. It
ships an all-in-one EFI binary named ipxe.efi which
includes the the bootloader and scripting features (which are
in snponly.efi too) and additionally all the ipxe
hardware drivers.
That way ipxe.efi can boot from the network even if the
firmware does not provide a driver. In that
case ipxe.efi itself must be loaded from local storage
though. You can download the efi binary and ready-to-use ISO/USB
images from boot.ipxe.org.
A UKI (unified kernel image) is an EFI binary bundle. It contains a linux kernel, an initrd, the command line and a few more optional components not covered here in sections of the EFI binary. Also the systemd efi stub, which handles booting the bundled linux kernel with the bundled initrd.
One advantage is that the secure boot signature of an UKI image will cover all components and not only the kernel itself, which is a big step forward for linux boot security.
Another advantage is that a UKI is self-contained. It does not need a bootloader which knows how to boot linux kernels and handle initrd loading. It is simply an EFI binary which you can start any way you want, for example from the EFI shell prompt.
The later makes UKIs interesting for network booting, because they can be used as bootfile too. The DHCP server hands out the UKI location, the UEFI firmware fetches the UKI and starts it. Done.
Combining the bootloader and UKI approaches is possible too. UEFI
bootloaders can load not only linux kernels. EFI binaries
(including UKIs) can be loaded too, in case of grub.efi
with the chainloader command. So if you want
interactive menus to choose an UKI to boot you can do that.
Modern UEFI implementations can netboot ISO images too. Unfortunately there are a few restrictions though:
When the UEFI firmware gets an ISO image as bootfile from the DHCP server it will load the image into a ramdisk, register the ramdisk as block device and try to boot from it.
From that point on booting works the same way booting from a local cdrom device works. The firmware will look for the boot loader on the ramdisk and load it. The bootloader will find the other components needed on the ramdisk, i.e. kernel and initrd in case of linux. All without any network access.
The UEFI firmware will also create ACPI tables for a pseudo nvdimm
device. That way the booted linux kernel will find the ramdisk too.
You can use the standard Fedora / CentOS / RHEL netinst ISO image,
linux will find the images/install.img on the ramdisk
and boot up all the way to anaconda. With enough RAM you can even
load the DVD with all packages, then do the complete system install
from ramdisk.
The big advantage of this approach is that the netboot workflow becomes very simliar to other installation workflows. It's not the odd kid on the block any more where loading kernel and initrd works in a completely different way. The very same ISO image can be:
Bonus: secure boot support suddenly isn't a headace any more.
There is one problem with the fancy new world though. We have lots of places in the linux world which depend on the linux kernel command line for system configuration. For example anaconda expects getting the URL of the install repository and the kickstart file that way.
When using a boot loader that is simple. The kernel command line simply goes into the boot loader config file.
With ISO images it is more complicated, changing the grub config file on a ISO image is a cumbersome process. Also ISO images are not exactly small, so install images with customized grub.cfg need quite some storage space.
UKIs can pass through command line arguments to the linux kernel, but that is only allowed in case secure boot is disabled. When using UKIs with secure boot the best option is to use the UKIs built and signed on distro build infrastructure. Which implies using the kernel command line for customization is not going to work with secure boot enabled.
So, all of the above (and UKIs in general) will work better if we
can replace the kernel command line as universal configuration
vehicle with something else. Which most likely will not be a single
thing but a number of different approaches depending on the use
case. Some steps into that direction did happen already. Systemd
can autodetect
partitions (so booting from disk without root=...
on the kernel command line works).
And systemd
credentials can be used to configure some aspects of a linux
system. There is still a loooong way to go though.
If this sounds familiar to you, it probably is. It means that memory should be either writable ("W", typically data), or executeable ("X", typically code), but not both. Elsewhere in the software industry this is standard security practice since ages. Now it starts to take off for UEFI firmware too.
This is a deep dive into recent changes, in both code (firmware) and administration (secure boot signing), the consequences this has for the linux, and the current state of affairs.
All UEFI memory allocations carry a memory type
(EFI_MEMORY_TYPE). UEFI tracks since day one whenever
a memory allocation is meant for code or data, among a bunch of
other properties such as boot service vs. runtime service memory.
For a long time it didn't matter much in practice. The concept of virtual memory does not exist for UEFI. IA32 builds even run with paging disabled (and this is unlikely to change until the architecture disappears into irrelevance). Other architectures use identity mappings.
While UEFI does not use address translation, nowdays it can use page
tables to enforce memory attributes, including (but not limited to)
write and execute permissions. When configured to do so it will set
code pages to R-X and data pages to RW-
instead of using RWX everywhere, so code using memory
types incorrectly will trigger page faults.
New in the UEFI spec (added in version 2.10) is
the EFI_MEMORY_ATTRIBUTE_PROTOCOL. Sometimes
properties of memory regions need to change, and this protocol can
be used to do so. One example is a self-uncompressing binary, where
the memory region the binary gets unpacked to initially must be
writable. Later (parts of) the memory region must be flipped from
writable to executeable.
As of today (Dec 2023) edk2 has
a EFI_MEMORY_ATTRIBUTE_PROTOCOL implementation for the
ARM and AARCH64 architectures, so this is present in the ArmVirt
firmware builds but not in the OVMF builds.
In an effort to improve firmware security in general and especially for secure boot Microsoft changed the requirements for binaries they are willing to sign with their UEFI CA key.
One key requirement added is that the binary layout must allow to enforce memory attributes with page tables, i.e. PE binary sections must be aligned to page size (4k). Sections also can't be both writable and executable. And the application must be able to deal with data section being mapped as not executable (NX_COMPAT).
These requirements apply to the binary itself
(i.e. shim.efi for linux systems) and everything loaded
by the binary (i.e. grub.efi, fwupd.efi
and the linux kernel).
We had and party still have a bunch of problems in all components
involved in the linux boot process,
i.e. shim.efi, grub.efi and the efi stub
of the linux kernel.
Some are old bugs such as memory types not being used correctly, which start to cause problems due to the firmware becoming more strict. Some are new problems due to Microsoft raising the bar for PE binaries, typically sections not being page-aligned. The latter are easily fixed in most cases, often it is just a matter of adding alignment to the right places in the linker scripts.
Lets have closer look at the components one by one:
shim.efi
shim added code to use the new EFI_MEMORY_ATTRIBUTE_PROTOCOL
before it was actually implemented by any firmware. Then this
was released completely untested. Did not work out very
well, we got a nice time bomb, and edk2 implementing
EFI_MEMORY_ATTRIBUTE_PROTOCOL for arm triggered it
...
Fixed in main branch, no release yet.
Getting new shim.efi binaries signed by Microsoft depends on the complete boot chain being compilant with the new requirements, which prevents shim bugfixes being shipped to users right now.
That should be solved soon though, see the kernel section below.
grub.efigrub.efi used to use memory types incorrectly.
Fixed upstream years ago, case closed.
Well, in theory. Upstream grub development goes at glacial speeds, so all distros carry a big stack of downstream patches. Not surprisingly that leads to upstream fixes being absorbed slowly and also to bugs getting reintroduced.
So, in practice we still have buggy grub versions in the wild. It is getting better though.
The linux kernel efi stub had it's fair share of bugs too. On non-x86 architectures (arm, riscv, ...) all issues have been fixed a few releases ago. They all share much of the efi stub code base and also use the same self-decompressing method (CONFIG_EFI_ZBOOT=y).
On x86 this all took a bit longer to sort out. For historical reasons x86 can't use the zboot approach used by the other architectures. At least as long as we need hybrid BIOS/UEFI kernels, which most likely will be a number of years still.
The final x86 patch series has been merged during the 6.7 merge window. So we should have a fixed stable kernel in early January 2024, and distros picking up the new kernel in the following weeks or months. Which in turn should finally unblock shim updates.
There should be enough time to get everything sorted for the spring distro releases (Fedora 40, Ubuntu 24.04).
edk2 has a bunch of config options to fine tune the firmware behavior, both compile time and runtime. The relevant ones for the problems listed above are:
PcdDxeNxMemoryProtectionPolicy
Compile time option. Use the --pcd switch for the
edk2 build script to set these. It's bitmask, with
one bit for each memory type, specifying whenever the firmware
shoud apply memory protections for that particular memory type,
by setting the flags in the page tables accordingly.
Strict configuration is PcdDxeNxMemoryProtectionPolicy =
0xC000000000007FD5. This is also the default for ArmVirt
builds.
Bug compatible configuration
is PcdDxeNxMemoryProtectionPolicy =
0xC000000000007FD1. This excludes
the EfiLoaderData memory type from memory
protections, so using EfiLoaderData allocations for
code will not trigger page faults. Which is an very common
pattern seen in boot loader bugs.
PcdUninstallMemAttrProtocol
Compile time options, for ArmVirt only. Brand
new, committed
to the edk2 repo this week (Dec 12th 2023). When set to TRUE
the EFI_MEMORY_ATTRIBUTE_PROTOCOL will be
unistalled.
Default is FALSE.
Setting this to TRUE will work around the shim bug.
opt/org.tianocore/UninstallMemAttrProtocol
Runtime option, for ArmVirt only. Also new. Can be set using
-fw_cfg on the qemu command line: -fw_cfg
name=opt/org.tianocore/UninstallMemAttrProtocol,string=y|n.
This is a runtime override for PcdUninstallMemAttrProtocol.
Works for both enabling and disabling the shim bug workaround.
In the future PcdDxeNxMemoryProtectionPolicy will
probably disappear in favor of memory profiles, which will allow to
configure the same settings (plus a few more) at runtime.
The default builds in the edk2-ovmf
and edk2-aarch64 packages are configured to be bug
compatible, so VMs should boot fine even in case the guests are
using a buggy boot chain.
While this is great for end users it doesn't help much for
bootloader development and testing, so there are alternatives.
The edk2-experimental package comes with a collection
of builds better suited for that use case, configured with strict
memory protections and (on
aarch64) EFI_MEMORY_ATTRIBUTE_PROTOCOL enabled, so you
can see buggy builds actually crash and burn. 🔥
For AARCH64 this
is /usr/share/edk2/experimental/QEMU_EFI-strictnx-pflash.raw.
The magic words for libvirt are:
<domain type='kvm'>
[ ... ]
<os>
<type arch='aarch64' machine='virt'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2/experimental/QEMU_EFI-strictnx-pflash.raw</loader>
<nvram template='/usr/share/edk2/aarch64/vars-template-pflash.raw'/>
</os>
[ ... ]If a page fault happens you will get this line ...
Synchronous Exception at 0x00000001367E6578
... on the serial console, followed by a stack trace and register dump.
For X64 this
is /usr/share/edk2/experimental/OVMF_CODE_4M.secboot.strictnx.qcow2.
Needs edk2-20231122-12.fc39 or newer. The magic words
for libvirt are:
<domain type='kvm'>
[ ... ]
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<loader readonly='yes' secure='yes' type='pflash' format='qcow2'>/usr/share/edk2/experimental/OVMF_CODE_4M.secboot.strictnx.qcow2</loader>
<nvram template='/usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2' format='qcow2'/>
</os>
[ ... ]It is also a good idea to add a debug console to capture the firmware log:
<serial type='null'>
<log file='/path/to/firmware.log' append='off'/>
<target type='isa-debug' port='1'>
<model name='isa-debugcon'/>
</target>
<address type='isa' iobase='0x402'/>
</serial>If you are lucky the page fault is logged there, also with an register dump. If you are not so lucky the VM will just reset and reboot.
The virt-firmware
project is a collection of python modules and scripts for working
with efi variables, efi varstores and also pe binaries. In case
your distro hasn't packages you can install it
using pip like most python packages.
The virt-fw-vars utility can work with efi varstores.
For example it is used to create the OVMF_VARS*secboot*
files, enrolling the secure boot certificates into the efi security
databases.
The simplest operation is to print the variable store:
virt-fw-vars --input /usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2 \
--print --verbose | less
When updating edk2 varstores virt-fw-vars always needs
both input and output files. If you want change an existing
variable store both input and output can point to the same file.
For example you can turn on shim logging for an existing libvirt
guest this way:
virt-fw-vars --input /var/lib/libvirt/qemu/nvram/${guest}_VARS.qcow2 \
--output /var/lib/libvirt/qemu/nvram/${guest}_VARS.qcow2 \
--set-shim-verbose
The next virt-firmware version will get a new --inplace
switch to avoid listing the file twice on the command line for this
use case.
If you want start from scratch you can use an empty variable store
from /usr/share/edk2 as input. For example when
creating a new variable store template with the test CA certificate
(shipped with pesign.rpm) enrolled additionally:
dnf install -y pesign
certutil -L -d /etc/pki/pesign-rh-test -n "Red Hat Test CA" -a \
| openssl x509 -text > rh-test-ca.txt
virt-fw-vars --input /usr/share/edk2/ovmf/OVMF_VARS_4M.qcow2 \
--output OVMF_VARS_4M.secboot.rhtest.qcow2 \
--enroll-redhat --secure-boot \
--add-db OvmfEnrollDefaultKeys rh-test-ca.txtThe test CA will be used by all Fedora, CentOS Stream and RHEL build infrastructure to sign unofficial builds, for example when doing scratch builds in koji or when building rpms locally on your developer workstation. If you want test such builds in a VM, with secure boot enabled, this is a convenient way to do it.
Useful for having a look at EFI binaries is pe-inspect.
If this isn't present try pe-listsigs. Initially the
utility only listed the signatures, but was extended over time to
show more information, so I added the pe-inspect alias
later on.
Below is the output for an 6.6 x86 kernel, you can see it does not have the patches to page-align the sections:
# file: /boot/vmlinuz-6.6.4-200.fc39.x86_64 # section: file 0x00000200 +0x00003dc0 virt 0x00000200 +0x00003dc0 r-x (.setup) # section: file 0x00003fc0 +0x00000020 virt 0x00003fc0 +0x00000020 r-- (.reloc) # section: file 0x00003fe0 +0x00000020 virt 0x00003fe0 +0x00000020 r-- (.compat) # section: file 0x00004000 +0x00df6cc0 virt 0x00004000 +0x05047000 r-x (.text) # sigdata: addr 0x00dfacc0 +0x00000d48 # signature: len 0x5da, type 0x2 # certificate # subject CN: Fedora Secure Boot Signer # issuer CN: Fedora Secure Boot CA # signature: len 0x762, type 0x2 # certificate # subject CN: kernel-signer # issuer CN: fedoraca
pe-inspect also knows the names for a number of special
sections and supports decoding and pretty-printing them, for example
here:
# file: /usr/lib/systemd/boot/efi/systemd-bootx64.efi # section: file 0x00000400 +0x00011a00 virt 0x00001000 +0x0001191f r-x (.text) # section: file 0x00011e00 +0x00003a00 virt 0x00013000 +0x00003906 r-- (.rodata) # section: file 0x00015800 +0x00000400 virt 0x00017000 +0x00000329 rw- (.data) # section: file 0x00015c00 +0x00000200 virt 0x00018000 +0x00000030 r-- (.sdmagic) # #### LoaderInfo: systemd-boot 254.7-1.fc39 #### # section: file 0x00015e00 +0x00000200 virt 0x00019000 +0x00000049 r-- (.osrel) # section: file 0x00016000 +0x00000200 virt 0x0001a000 +0x000000de r-- (.sbat) # sbat,1,SBAT Version,sbat,1,https://github.com/rhboot/shim/blob/main/SBAT.md # systemd,1,The systemd Developers,systemd,254,https://systemd.io/ # systemd.fedora,1,Fedora Linux,systemd,254.7-1.fc39,https://bugzilla.redhat.com/ # section: file 0x00016200 +0x00000200 virt 0x0001b000 +0x00000084 r-- (.reloc)
The last utility I want introduce is virt-fw-sigdb,
which can create, parse and modify signature databases. The
signature database format is used by the firmware to store
certificates and hashes in EFI variables. But sometimes the format
used for files too. virt-firmware has the functionality anyway, so
I've added a small frontend utility to work with those files.
One file in signature database format
is /etc/pki/ca-trust/extracted/edk2/cacerts.bin which
contains the list of of trusted CAs in sigature database format.
Can be used to pass the CA list to the VM firmware for TLS
connections (https network boot).
Shim also uses that format when compiling multiple certificates into the built-in VENDOR_DB or VENDOR_DBX databases.
Thats it for today folks. Hope you find this useful.
]]>
On your linux machine you can use lscpu to see the size
of the physical address space:
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
^^^^^^^^^^^^^^^^
[ ... ]
In /proc/iomem you can see how the address space is
used. Note that the actual addresses are only shown to root.
The very first x86_64 processor (AMD Opteron) shipped with a physical address space of 40 bits (aka one TeraByte). So when qemu added support for the (back then) new architecture the qemu vcpu likewise got 40 bits of physical address space, probably assuming that this would be a safe baseline. It is still the default in qemu (version 8.1 as of today) for backward compatibility reasons.
Enter Intel. The first 64-bit processors shipped by Intel featured only 36 bits of physical address space. More recent Intel processors have 39, 42 or more physical address bits. Problem is this limit applies not only to the real physical addess space, but also to Extended Page Tables (EPT). Which means the physical address space of virtual machines is limited too.
So, the problem is the virtual machine firmware does not know how much physical address space it actually has. When checking CPUID it gets back 40 bits, but it could very well be it actually has only 36 bits.
To address that problem the virtual machine firmware was very conservative with address space usage, to avoid crossing the unknown limit.
OVMF used to have a MMIO window with fixed size (32GB), which was based on the first multiple of 32GB after normal RAM. So a typical, smallish virtual machine had 0 -> 32GB for RAM and 32GB -> 64GB for IO, staying below the limit for 36 bits of physical address space (which equals 64GB).
VMs having more than 30GB of RAM will need address space above 32GB for RAM, which pushes the IO window above the 64GB limit. The assumtion that hosts which have enough physical memory to run such big virtual machines also have a physical address space larger than 64GB seems to have worked good enough.
Nevertheless the fixed 32G-sized IO window became increasingly problematic. Memory sizes are growing, not only for main memory, but also for device memory. GPUs have gigabytes of memory these days.
Qemu has tree -cpu options to control physical address
space advertized to the guest, for quite a while already.
off (except for -cpu
host where it is on).
on
by default.
host-phys-bits=on. Can be used to
reduce the number of physical address space bits communicated to
the guest. Useful for live migration compatibility in case your
machine cluster has machines with different physical address space
sizes.
host-phys-bits=off. Can be used to
set the number of physical address space bits to any value you
want, including non-working values. Use only if you know what you
are doing, it's easy to shot yourself into the foot with this one.
Recent OVMF versions (edk2-stable202211 and newer) try to figure the
size of the physical address space using a heuristic: In case the
physical address space bits value received via CPUID is 40 or below
it is checked against known-good values, which are 36 and 39 for
Intel processors and 40 for AMD processors. If that check passes or
the number of bits is 41 or higher OVMF assumes qemu is configured
with host-phys-bits=on and the value can be trusted.
In case there is no trustworthy phys-bits value OVMF will continue with the traditional behavior described above.
In case OVMF trusts the phys-bits value it will apply some OVMF-specific limitations before actually using it:
The final phys-bits value will be used to calculate the size of the physical address space available. The 64-bit IO window will be placed as high as possibe, i.e. at the end of the physical address space. The size of the IO window and also the size of the PCI bridge windows (for prefetchable 64-bit bars) will be scaled up with the physical address space, i.e. on machines with a larger physical address space you will also get larger IO windows.
Starting with version 1.16.3 SeaBIOS uses a heuristic simliar to OVMF to figure whenever there is a trustworthy phys-bits value.
If that is the case SeaBIOS will enable the 64-bit IO window by default and place it at the end of the address space like OVMF does. SeaBIOS will also scale the size of the IO window with the size of the address space.
Although the overall behavior is simliar there are some noteworthy differences:
Starting with release 8.2 the firmware images bundled with upstream qemu are new enough to include the OVMF and SeaBIOS changes described above.
The new firmware behavior triggered a few bugs elsewhere ...
When doing live migration the vcpu configuration on source and target host must be identical. That includes the size of the physical address space.
libvirt can calculate the cpu baseline for a given cluster, i.e. create a vcpu configuration which is compatible with all cluster hosts. That calculation did not include the size of the physical address space though.
With the traditional, very conservative firmware behavior this bug did not cause problems in practice, but with OVMF starting to use the full physical address space live migrations in heterogeneous clusters started to fail because of that.
In libvirt 9.5.0 and newer this has been fixed.
In general, it is a good idea to set the qemu
config option host-phys-bits=on.
In case guests can't deal with PCI bars being mapped at high
addresses the host-phys-bits-limit=bits option
can be used to limit the address space usage. I'd suggest to stick
to values seen in actual processors, so 40 for AMD and 39 for Intel
are good candidates.
In case you are running 32-bit guests with alot of memory (which btw
isn't a good idea performance-wise) you might need turn off long
mode support to force the PCI bars being mapped below 4G. This can
be done by simply using qemu-system-i386 instead
of qemu-system-x86_64, or by explicitly
setting lm=off in the -cpu options.
Some people already noticed and asked questions. So guess I better write things down in my blog so I don't have to answer the questions over and over again, and I hope to also clarify some things on distro firmware builds.
So, yes, the jenkins autobuilder creating the firmware repository at https://www.kraxel.org/repos/jenkins/ has been shutdown yesterday (Jul 19th 2022). The repository will stay online for the time being, so your establish workflows will not instantly break. But the repository will not get updates any more, so it is wise to start looking for alternatives now.
The obvious primary choice would be to just use the firmware builds provided by your distribution. I'll cover edk2 only, which seems to be the by far most popular use, even thought here are also builds for other firmware projects.
Given I'm quite familier with the RHEL / Fedora world I can give
some advise here. The edk2-ovmf package comes with
multiple images for the firmware code and the varstore template
which allow for various combinations. The most important ones are:
OVMF_CODE.secboot.fd and OVMF_VARS.secboot.fdOVMF_CODE.secboot.fd and OVMF_VARS.fdOVMF_CODE.fd and OVMF_VARS.fdThe classic way to setup this in libvirt looks like this:
<domain type='kvm'>
[ ... ]
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
<nvram template='/usr/share/OVMF/OVMF_VARS.fd'/>
</os>
To make this easier the firmware builds come with json files
describing the capabilities and requirements. You can find these
files in /usr/share/qemu/firmware/. libvirt can use
them to automatically find suitable firmware images, so you don't
have to write the firmware image paths into the domain
configuration. You can simply use this instead:
<domain type='kvm'>
[ ... ]
<os firmware='efi'>
<type arch='x86_64' machine='q35'>hvm</type>
</os>libvirt also allows to ask for specific firmware features. If you don't want use secure boot for example you can ask for the blank varstore template (no secure boot keys enrolled) this way:
<domain type='kvm'>
[ ... ]
<os firmware='efi'>
<type arch='x86_64' machine='q35'>hvm</type>
<firmware>
<feature name='enrolled-keys' enabled='no' />
</firmware>
</os>
In case you change the configuration of an existing virtual machine
you might (depending on the kind of change) have to run virsh
start --reset-nvram domain once to to start over with
a fresh copy of the varstore template.
The world has moved forward. UEFI isn't a niche use case any more. Linux distributions all provide good packages theys days. The edk2 project got good CI coverage (years ago it was my autobuilder raising the flag when a commit broke the gcc build). The edk2 project got a regular release process distros can (and do) follow.
All in all the effort to maintain the autobuilder doesn't look justified any more.
]]>
To build edk2 you need to have a bunch of tools installed. An
compiler and the make are required of course, but also
iasl, nasm and libuuid. So
install them first (package names are for centos/fedora).
dnf install -y make gcc binutils iasl nasm libuuid-develIf you want cross-build arm firmware on a x86 machine you also need cross compilers. While being at also set the environment variables needed to make the build system use the cross compilers:
dnf install -y gcc-aarch64-linux-gnu gcc-arm-linux-gnu
export GCC5_AARCH64_PREFIX="aarch64-linux-gnu-"
export GCC5_ARM_PREFIX="arm-linux-gnu-"Next clone the tiaocore/edk2 repository and also fetch the git submodules.
git clone https://github.com/tianocore/edk2.git
cd edk2
git submodule update --init
The edksetup script will prepare the build environment
for you. The script must be sourced because it sets some
environment variables (WORKSPACE being the most
important one). This must be done only once (as long as you keep
the shell with the configured environment variables open).
source edksetup.shNext step is building the BaseTools (also needed only once):
make -C BaseToolsNote: Currently (April 2022) BaseTools are being rewritten in Python, so most likely this step will not be needed any more at some point in the future.
Finally the build (for x64 qemu) can be kicked off:
build -t GCC5 -a X64 -p OvmfPkg/OvmfPkgX64.dsc
The firmware volumes built can be found
in Build/OvmfX64/DEBUG_GCC5/FV.
Building the aarch64 firmware instead:
build -t GCC5 -a AARCH64 -p ArmVirtPkg/ArmVirtQemu.dsc
The build results land
in Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV.
Qemu expects the aarch64 firmware images being 64M im size. The firmware images can't be used as-is because of that, some padding is needed to create an image which can be used for pflash:
dd of="QEMU_EFI-pflash.raw" if="/dev/zero" bs=1M count=64
dd of="QEMU_EFI-pflash.raw" if="QEMU_EFI.fd" conv=notrunc
dd of="QEMU_VARS-pflash.raw" if="/dev/zero" bs=1M count=64
dd of="QEMU_VARS-pflash.raw" if="QEMU_VARS.fd" conv=notrunc
There are a bunch of compile time options, typically enabled
using -D NAME or -D NAME=TRUE. Options
which are enabled by default can be turned off using -D
NAME=FALSE. Available options are defined in
the *.dsc files referenced by the build
command. So a feature-complete build looks more like this:
build -t GCC5 -a X64 -p OvmfPkg/OvmfPkgX64.dsc \
-D FD_SIZE_4MB \
-D NETWORK_IP6_ENABLE \
-D NETWORK_HTTP_BOOT_ENABLE \
-D NETWORK_TLS_ENABLE \
-D TPM2_ENABLESecure boot support (on x64) requires SMM mode. Well, it builds and works without SMM, but it's not secure then. Without SMM nothing prevents the guest OS writing directly to flash, bypassing the firmware, so protected UEFI variables are not actually protected.
Also suspend (S3) support works with enabled SMM only in case parts of the firmware (PEI specifically, see below for details) run in 32bit mode. So the secure boot variant must be compiled this way:
build -t GCC5 -a IA32 -a X64 -p OvmfPkg/OvmfPkgIa32X64.dsc \
-D FD_SIZE_4MB \
-D SECURE_BOOT_ENABLE \
-D SMM_REQUIRE \
[ ... add network + tpm + other options as needed ... ]
The FD_SIZE_4MB option creates a larger firmware image,
being 4MB instead of 2MB (default) in size, offering more space for
both code and vars. The RHEL/CentOS builds use that. The Fedora
builds are 2MB in size, for historical reasons.
If you need 32-bit firmware builds for some reason, here is how to
do it:
build -t GCC5 -a ARM -p ArmVirtPkg/ArmVirtQemu.dsc
build -t GCC5 -a IA32 -p OvmfPkg/OvmfPkgIa32.dsc
The build results will be in
in Build/ArmVirtQemu-ARM/DEBUG_GCC5/FV and
Build/OvmfIa32/DEBUG_GCC5/FV
The x86 firmware builds create three different images:
OVMF_VARS.fd/var/lib/libvirt/qemu/nvram.
OVMF_CODE.fdOVMF.fdCODE
and VARS. This can be loaded as ROM
using -bios, with two drawbacks: (a) UEFI variables
are not persistent, and (b) it does not work
for SMM_REQUIRE=TRUE builds.
qemu handles pflash storage as block devices, so we have to create block devices for the firmware images:
CODE=${WORKSPACE}/Build/OvmfX64/DEBUG_GCC5/FV/OVMF_CODE.fd
VARS=${WORKSPACE}/Build/OvmfX64/DEBUG_GCC5/FV/OVMF_VARS.fd
qemu-system-x86_64 \
-drive if=none,id=code,format=raw,file=${CODE},readonly=on \
-drive if=none,id=vars,format=raw,file=${VARS},snapshot=on \
-machine q35,pflash0=code,pflash1=vars \
[ ... ]
Here is the arm version of that (using the padded files created
using dd, see above):
CODE=${WORKSPACE}/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI-pflash.raw
VARS=${WORKSPACE}/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS-pflash.raw
qemu-system-aarch64 \
-drive if=none,id=code,format=raw,file=${CODE},readonly=on \
-drive if=none,id=vars,format=raw,file=${VARS},snapshot=on \
-machine virt,pflash0=code,pflash1=vars \
[ ... ]The core edk2 repo holds a number of packages, each package has its own toplevel directory. Here are the most interesting ones:
The firmware modules in the edk2 repo often named after the boot
phase they are running in. Most drivers are
named SomeThingDxe for example.
pip3
install ovmfctl. The project is hosted
at gitlab.
Usage: ovmfctl --input file.fd.
It's a debugging tool which just prints the structure and content of firmware volumes.
This is a tool to print and modify variable store volumes. Main focus has been on certificate handling so far.
Enrolling certificates for secure boot support in virtual machines has been a rather painfull process. It's handled by EnrollDefaultKeys.efi which needs to be started inside a virtual machine to enroll the certificates and enable secure boot mode.
With ovmfctl it is dead simple:
ovmfctl --input /usr/share/edk2/ovmf/OVMF_VARS.fd \
--enroll-redhat \
--secure-boot \
--output file.fd
This enrolls the Red Hat Secure Boot certificate which is used by
Fedora, CentOS and RHEL as platform key. The usual Microsoft
certificates are added to the certificate database too, so windows
guests and shim.efi work as expected.
If you want more fine-grained control you can use
the --set-pk, --add-kek, --add-db
and --add-mok switches instead.
The --enroll-redhat switch above is actually just a shortcut for:
--set-pk a0baa8a3-041d-48a8-bc87-c36d121b5e3d RedHatSecureBootPKKEKkey1.pem \
--add-kek a0baa8a3-041d-48a8-bc87-c36d121b5e3d RedHatSecureBootPKKEKkey1.pem \
--add-kek 77fa9abd-0359-4d32-bd60-28f4e78f784b MicrosoftCorporationKEKCA2011.pem \
--add-db 77fa9abd-0359-4d32-bd60-28f4e78f784b MicrosoftWindowsProductionPCA2011.pem \
--add-db 77fa9abd-0359-4d32-bd60-28f4e78f784b MicrosoftCorporationUEFICA2011.pem \
If you just want the variable store be printed use ovmfctl
--input file.fd --print. Add --hexdump
for more details.
Extract all certificates: ovmfctl --input file.fd
--extract-certs.
Try ovmfctl --help for a complete list of command line
switches. Note that Input and output file can be indentical for
inplace updates.
That's it. Enjoy!
]]>