Fallback mechanism

Introduction

This mechanism permits to test and recover from certain non-booting coreboot images.

This works by having two coreboot images in the same flash chip:

• One fallback/ image: The working image.
• One normal/ image: The image to be tested.

This feature is not widely tested on all boards. It also requires it to have a reboot_bits exported in the cmos layout. This also doesn't protect against human errors when using such feature, or bugs in the code responsible for switching between the two images.

Uses cases

• Test new images way faster: if the image doesn't boot it will fallback on the old known-working image and save a long reflashing procedure
• Test new images more safely: Despite of the recommendations of having a way to externally reflash, many new user don't. Assuming that the user don't screw up the fallback/ procedure (which adds a layer of complexity) he can test new images more safely because it will fallback on the known good image.
• More compact testing setup: Since reflashing tools are not mandatory anymore, the tests can be done with less voluminous hardware, which means that the test setup is easier to bring with you while travelling.
• Faster bisecting of commit which broke the boot, assuming it broke after the inclusion of that fallback mecanism.

Enabling the fallback switch

The Fallback switch behaviour is governed by the BOOTBLOCK_SIMPLE and BOOTBLOCK_NORMAL compilation options.

They are selectable in "Bootblock behaviour" in make menuconfig.

Currently only two choices are available:

• BOOTBLOCK_SIMPLE: "(X) Always load fallback"
• BOOTBLOCK_NORMAL: "(X) Switch to normal if CMOS says so"

If BOOTBLOCK_SIMPLE is chosen, then fallback/ , then no siwtch will ever be done and fallback/ will always be chosen, since this is the default we have to change it to get it to work. If BOOTBLOCK_NORMAL is chosen, then the switch will be able to work.

If the BOOTBLOCK_NORMAL is chosen, the functionality is disabled but can be enabled later if needed.

Make sure that in "General setup --->" you have:

(fallback) CBFS prefix to use

How it works (summary)

Coreboot will switch to fallback/ if the boot count is higher than CONFIG_MAX_REBOOT_CNT (or if normal/ isn't present).

Coreboot increments the reboot count at each boot.

Here, clearing the boot count is delegated to what is run after coreboot.

To get the maximum safety out of it, clearing the boot count after the last step of the boot is advised.

Example of use

For instance once the system is fully booted, a systemd unit can reset the boot count.

That way if the coreboot changes makes it impossible to boot a linux kernel or even if GNU/Linux can't fully boot, the boot count won't be reset.

Then the user will power off the computer, and at the next boot CONFIG_MAX_REBOOT_CNT will hopefully be reached. Then coreboot will boot on the good known working image and the boot will complete.

At that point the user is expected to reflash a good image in order not to go in normal/ again at the next boot.

Current limitations

• scripts exist only for the systemd init system, but they are easy to adapt to other init systems
• suspend/resume systemd scripts not written yet
• some issues can arrise when the nvram layout is not the same between normal/ and fallback/
• The number of failed boot is 3 by default (for all boards that don't set CONFIG_MAX_REBOOT_CNT)
• In order to fully boot, some boards do reboot once during the boot procedure. The issue is that it reboot conditionally, and no code has been written yet to take that into account.
• Payloads can have non-configurable default locations when loading things from cbfs:
  • When using grub as a payload, grub.cfg is at etc/grub.cfg by default, so if you want to test grub as a payload, remember to change grub.cfg's path not to interfer with the fallback's grub configuration.
  • Changing the path of what SeaBIOS loads from cbfs is probably configurable with SeaBIOS cbfs symlinks but not yet tested/documented with the use of the fallback mecanism
• Complexity for the user:
  • Once the normal/ image has been tested, if the user wants to flash it to fallback/ he will have to make sure that the normal/ image was running when he tested it, and that it was not the fallback/ (that could happen due to an error of the user for instance), cbmem -c is a good way to do it.
  • The user has to check if the coreboot image with fallback/ that he is adding normal/ to, had CONFIG_BOOTBLOCK_NORMAL enabled. Else it will probably keep booting on normal/

Using it

Prerequisites

• Make sure that your fallback/ image has the fallback switch mecanism enabled
• Build the fallback image as you would build an image usually

Building the coreboot.rom image

Building normal/ (normal.sh)

This scrpit takes an existing coreboot image path as argument.

#!/bin/sh
# In the cases where this work is copyrightable, it falls under the GPLv2
# or later license that is available here:
# https://www.gnu.org/licenses/gpl-2.0.txt

image="$1"
if [ $# -ne 1 ] ; then
	echo "Usage $0 <image>"
	exit 1
fi

die()
{
  echo "$1 Failed"
  exit 1
}

cbfs_add()
{
  name=$1
  file=$2
  cbfs_remove ${name}
  ./util/cbfstool/cbfstool ./build/coreboot.rom add -n ${name} -t raw -f ${file}
}

cbfs_remove()
{
  name=$1
  ./util/cbfstool/cbfstool ./build/coreboot.rom remove -n ${name}
}

cbfs_reuse_payload()
{
  ./util/cbfstool/cbfstool ./build/coreboot.rom extract -f ./build/payload.elf -n fallback/payload
  ./util/cbfstool/cbfstool ./build/coreboot.rom add -f ./build/payload.elf -n normal/payload -t payload
}

check_config()
{
  grep "^CONFIG_CBFS_PREFIX=\"normal\"$" .config > /dev/null || die "Not using normal cbfs prefix"
  grep "^CONFIG_UPDATE_IMAGE=y$" .config > /dev/null || die "Not using CONFIG_UPDATE_IMAGE"
  grep "^CONFIG_SKIP_MAX_REBOOT_CNT_CLEAR=y" .config > /dev/null || die "Not using CONFIG_SKIP_MAX_REBOOT_CNT_CLEAR"
}

check_config
make oldconfig || die "make oldconfig"
make clean || die "clean"
mkdir build/ || die "mkdir build"

cp ${image} ./build/coreboot.rom || die "cp"

cbfs_remove normal/romstage
cbfs_remove normal/ramstage
cbfs_remove normal/payload
cbfs_remove normal/dsdt.aml
cbfs_remove config
cbfs_remove revision

# it now adds it automatically
cbfs_remove etc/ps2-keyboard-spinup
 
make || die "make"

# uncomment if you want to reuse fallback's payload
# cbfs_reuse_payload

./util/cbfstool/cbfstool ./build/coreboot.rom print

OS configuration examples

The configurations below assume that the user wants to keep booting on normal/ if the boot doesn't fail.

Example scripts

The most simple way to do it is to run some nvramtool commands, they are described in the scripts below. set-normal-0.sh has to be run:

• After the boot is completed and is declared a success.
• After the resuming is completed.

The way to make them run at boot and after suspend is not described here yet.

set-fallback-1.sh

#!/bin/sh
nvramtool -w boot_option=Fallback
nvramtool -w last_boot=Fallback
nvramtool -w reboot_bits=1

set-normal-0.sh

#!/bin/sh
nvramtool -w boot_option=Normal
nvramtool -w last_boot=Normal
nvramtool -w reboot_bits=0

get-nvram.sh

#!/bin/sh
nvramtool -a | grep -e boot_option -e last_boot -e reboot_bits

With systemd

Systemd setup

Requirements:

• nvramtool has to be in the path.

Limitations:

• This setup doesn't needs to run that systemd unit when resuming from suspend to ram, but it's not described yet here.

The unit file below has to be activated with:

systemctl enable coreboot-booted-ok
systemctl start coreboot-booted-ok

/etc/systemd/system/coreboot-booted-ok.service:

#  This file is not part of systemd.
#
#  this file is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Tell coreboot that the computer booted fine.
DefaultDependencies=no
Wants=display-manager.service
After=display-manager.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/nvramtool -w boot_option=Normal
ExecStart=/usr/local/sbin/nvramtool -w last_boot=Normal
ExecStart=/usr/local/sbin/nvramtool -w reboot_bits=0

[Install]
WantedBy=multi-user.target