arm64 Linux targetsReply-to:
<will@kernel.org>Revision history:
* THIS IS A WORK-IN-PROGRESS AND IS CURRENTLY INCOMPLETE! *
This guide assumes that you have some familiarity with Linux systems, an
arm64 development board and a desktop PC running a Debian derivative on the
same local network.
Effectively testing a large, complex, collaborative project such as the Linux kernel is an exercise fraught with difficulties: Which tree should be the testing focus? How can code coverage be guaranteed across the configuration space? Is there a reliance upon implicit behaviours between subsystems? Does the code run as expected across multiple machines or architectures? Is there the potential for undefined behaviour?
Consequently, ensuring that even a seemingly trivial piece of kernel code works as intended relies heavily on bug reports from users and results from targeted tests. This approach clearly has its limits and inevitably tends to focus on maintaining stability of "common-case" functionality so that routine operations on mainstream architectures rarely suffer from visible regressions between major releases of the kernel. Outside of this scope, however, regressions and bugs are just as significant when it comes to establishing security and portability guarantees of the kernel. Fuzzing is a largely automated technique that can be used to explore these unusual corners of the kernel methodically and has been adopted by the hugely successful syzkaller project to find hundreds of bugs in the mainline kernel, many of which turned out not to be so esoteric after all.
This document is a work-in-progress guide for setting up a syzkaller system
targetting the arm64 architecture (AArch64 if you're fancy) based on my own
experience trying to do this, and finding it a bit more difficult than I would
have liked.
Although it's probably possible to run syzkaller entirely on the machine being
tested, it also wouldn't be much use because if when the kernel crashes,
syzkaller itself will die and you'll be stuck without much in the way of
debugging information required to identify the cause of the problem.
A much better option is to run the syzkaller tests inside a virtual machine
(VM) so that the damage is hopefully contained to that VM instance. This also
means you can run the sucker as root without it accidentally trashing your data
or going crazy and uploading your SSH keys to the web. To accomplish this,
syzkaller provides a tool called the syz-manager, which is responsible for
interacting with target VMs by sending them binaries to execute and collecting
their output. The syz-manager process also reports the progress of each
target via a local webserver and, on detecting a crash, works to generate a
reproducer binary for analysis. This is a fairly intensive task, so I would
recommend running the syz-manager on your desktop machine, communicating with
the targets over the local network.
To summarise, the components we're going to configure are as follows:
x86 machine for the heavy lifting. If you're fortunate
enough to have access to one of those fabled 'ARM servers', you could
probably use that, but I'll assume x86 because it means we're going to do a
spot of cross compilation. Hostname: manager.arm64 board running Linux and KVM. Hostname: target-host.target-guest.Syzkaller is written in Go and requires a toolchain version of 1.11+.
The build packaged by recent distributions (e.g. Debian buster) should be
sufficient to install using:
manager:~# apt-get install golang-go
Failing that, you can try an official prebuilt binary. Once it's installed, you should be able to run:
manager:~$ go version
go version go1.13.4 linux/amd64
Before we can build syzkaller, we need to get ourselves an arm64
cross-compiler. Thankfully, we can grab pre-built binaries from arm.com,
although I have little faith in the link remaining stable. Make sure you grab
the one named 'x86_64 Linux hosted cross compiler for the AArch64 GNU/Linux target
(aarch64-linux-gnu)':
manager:~$ wget "https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz"
manager:~$ tar -xJf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz
You will then need to ensure that the contents of
gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu/bin/ are visible on your
$PATH. Alternatively, you might be able to use the host clang, but I
haven't tried that myself. Once successful, you should be able to run something
like:
manager:~$ aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.3.0
Copyright © 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Then it's time to pull down the syzkaller sources and build 'em. As it turns
out, go already knows how to do this for us:
manager:~$ go get -u -d github.com/google/syzkaller/...
manager:~$ cd ~/go/src/github.com/google/syzkaller/
manager:~/go/src/github.com/google/syzkaller$ make TARGETARCH=arm64
Once that's done, you should see a mixture of x86 and arm64 binaries for the manager and the target respectively:
manager:~/go/src/github.com/google/syzkaller$ file bin/syz-manager
bin/syz-manager: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
manager:~/go/src/github.com/google/syzkaller$ file bin/linux_arm64/syz-executor
bin/linux_arm64/syz-executor: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=0740fa499330983c0fddcbfb477ca6f90e601510, with debug_info, not stripped
Eventually, the static binaries will be sent over to the target and executed there. Cool, huh?
It's unfortunately difficult to write a generic guide for configuring an
arbitrary arm64 board to act as a host machine because so many important
details vary wildly between different embedded systems. Even silly things like
the serial port are designed like it's the wild west, with incompatible
modifications all over the place and vendors competing for the highest baud
rate as though it's some sort of pissing contest. Instead, I'll list some
of the requirements you need to satisfy and then I'll explain how I configured
the vastly underpowered La Frite board which I received as a freebie
from the thoroughly excellent Kernel Recipes conference. If you're ever
given the opportunity to attend, then you definitely should.
The target (host) system needs to satisfy at least all of the following:
arm64 CPU booting in 64-bit mode.sshd up and running.EL2. Look for a line
in the kernel dmesg that reads 'CPU: All CPU(s) started at EL2'. If you
can't find it, hurl the thing in the bin and organise a protest.From this point on, I'm going to assume the target (host) is running Debian and that you have access to a root shell. If you're already in that state, then skip ahead to configuring KVMtool.
'La Frite' is a low-powered, low-cost development board based on the S805x SoC from Amlogic. It just about ticks the boxes above but, more
importantly, I had a couple kicking around to sacrifice to the syzkaller gods.
If you feel to the need to purchase one, they appear to be on sale.
Nope, not assembly language but actual assembly of the board and its cheap aluminium chassis instead. If you have trouble then your best bet is probably to look at the images I've linked to in this section. The process requires mounting the eMMC on the PCB before applying the thermal heatpad to the SoC itself and clamping shut with the two outer plates. The final touches are some delightful rubber feet that don't seem to serve any real purpose. Looking down at the completed enclosure, the 40-pin GPIO header should be visible with the 'u-boot button' accessible in the far corner near the USB ports. Make sure you have your questionably-standards-compliant USB A-to-A cable handy, since you'll need it in the next step.
Word of warning! Rumour has it that the eMMC spacers are directional and, if mounted incorrectly, could lead to intermittent eMMC disconnects and/or probe failures during boot. See what you think but I couldn't spot the directionality although my eyesight is, frankly, appalling. However, I did experience some issues with eMMC probing and so ended up wrapping it in tape (why not?) which seems to have helped a bit, but hasn't completely solved the problem. Hmm. Did I mention the board was free?
On the cables front, you're going to need:
Link the serial adapter to the UART pins exposed on the GPIO header as follows:
The silkscreen has a little arrow to identify pin 1, with pins 39 and 40 being numbered at the other end so you can figure out how the sequence works. The baud rate is an impressively modest 115200. Don't hook up the other cables just yet.
Although the board comes with some pre-installed firmware, there were some eMMC issues when running with earlier versions so it's best just to nuke it with the latest and greatest before going any further. Before we can do that, we need to grab a copy of the bespoke flashing tool:
manager:~$ git clone https://github.com/libre-computer-project/pyamlboot.git
manager:~$ cd ~/pyamlboot
manager:~/pyamlboot$ git checkout gxl
You'll also need the Python USB libraries installed:
manager:~/pyamlboot# apt-get install python3-usb
Yet another warning! If you haven't guessed already, we're about to run a
random python script from the internet as root. I encourage you to read the
thing before doing so.
Now, connect one end of the dodgy USB A-to-A cable to your computer, and with the u-boot button held down connect the other end to the USB port closest to the 40-pin GPIO header on the board. I found that you didn't need extra power: it should light up like a Christmas tree without the micro-USB connected. If you have the serial console up, you might see some junk:
GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:0;RCY:0;USB:0;
If it stops here, then that's Fritese for "Waiting for firmware" and we can finally run that script that you've audited:
manager:~/pyamlboot# ./flash-firmware.sh aml-s805x-ac
I like the part where it downloads the random temporary file best. Anywho, the serial console should be littered with messages, hopefully finishing with something to indicate that the flashing was either successful or not needed. If it failed, maybe you can try again.
Disconnect and immediately reconnect the dodgy USB cable: you'll see the
firmware booting on the serial console. Hammer Escape until it drops you into
a menu entitled *** U-Boot Boot Menu ***. So good they named it twice.
From this menu, select the eMMC USB Drive Mode option. You should then see
the eMMC show up as a USB mass storage device on your desktop machine:
usb 2-2: Manufacturer: Libre Computer
usb-storage 2-2:1.0: USB Mass Storage device detected
scsi host4: usb-storage 2-2:1.0
scsi 4:0:0:0: Direct-Access Linux UMS disk 0 ffff PQ: 0 ANSI: 2
sd 4:0:0:0: Attached scsi generic sg1 type 0
sd 4:0:0:0: [sdb] 15269888 512-byte logical blocks: (7.82 GB/7.28 GiB)
sd 4:0:0:0: [sdb] Attached SCSI removable disk
In my case, it's sdb, so when running the next few commands take care to
substitute that with whatever you ended up with.
Grab a pre-baked Debian image for flashing:
manager:~$ wget http://share.loverpi.com/board/libre-computer-project/libre-computer-board/image/debian/libre-computer-aml-s805x-ac-debian-buster-headless-4.19.64%2B-2019-08-05.zip
manager:~$ unzip libre-computer-aml-s805x-ac-debian-buster-headless-4.19.64+-2019-08-05.zip
manager:~# dd if=libre-computer-aml-s805x-ac-debian-buster-headless-4.19.64+-2019-08-05.img of=/dev/sdb bs=4M
This can take a little while, so put the kettle on and come back later (it takes about 2 mins).
When it's completed, disconnect the USB cable and discard it.
Connect the ethernet and mini-USB power supply. After a few seconds, you should
see the Linux kernel booting at last. You'll get dropped to a login after
systemd has done its thing: the credentials are libre:computer .
Although this works as a basic setup, I tweaked it slightly to make it a bit more friendly. I recommend you do the same:
#### Change the default user and group names ####
libre-computer:~$ sudo su
libre-computer:~# echo ttyAML0 >> /etc/securetty
libre-computer:~# passwd
# <Set whatever root password you like>
libre-computer:~# exit
libre-computer:~$ exit
# <Log back in as root>
# You should customise this unless you're also called Will
libre-computer:~# usermod -l will -d /home/will -c "" -m libre
libre-computer:~# groupmod -n will libre
libre-computer:~# exit
# <Log back in as you>
libre-computer:~$ passwd
# <Set whatever password you like for yourself>
#### Update the system ####
libre-computer:~# apt-get update
libre-computer:~# apt-get dist-upgrade
# <You may as well SSH in now to avoid the pain of the serial console>
#### Change the hostname ####
libre-computer:~# echo "target-host" > /etc/hostname
libre-computer:~# sed -i 's/libre-computer/target-host/' /etc/hosts
#### Avoid silly timeout when mounting swap ####
libre-computer:~# vim /etc/fstab
# <Change the 'x-systemd.device-timeout=1' entry in the line for the swap partition to have a larger value. 10 works for me.>
#### Avoid reboot taking ages thanks to the NIC not shutting down properly ####
libre-computer:~# vim /etc/systemd/system.conf
# <Uncomment the 'DefaultTimeoutStopSec=90s' line and change it to something smaller. Again, 10 works for me.>
# Note that this won't take effect until after a reboot
#### Enable the GRUB menu during boot ####
libre-computer:~# vim /etc/default/grub
# <Uncomment the 'GRUB_TERMINAL=console' line>
libre-computer:~# update-grub
#### Reboot the system ####
libre-computer:~# reboot
This isn't required by syzkaller, but I thought I'd include it here because it's fairly straightforward once you've got this far and it might save you from being stuck forever on the 4.19 kernel shipped with the Debian filesystem.
Take a copy of the kernel sources on your desktop:
manager:~$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
manager:~$ cd linux
I've had success with v5.4, but later releases should work too:
manager:~/linux$ git reset --hard v5.4
Then simply build a defconfig Debian kernel package:
manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j $(nproc) bindeb-pkg
For some reason, this puts all of the output in the parent directory, so it will make a mess there. You might decide to clean it up later on.
I remember getting annoyed in the early days of ACPI on arm64 when the
enterprise folks would poke fun at the community for continuously breaking the
devicetree bindings used by the kernel. Well, it turns out they were right, so
we need to update the kernel and the devicetree blob at the same time. We'll
scp them over to the target from the build machine:
manager:~/linux$ scp arch/arm64/boot/dts/amlogic/meson-gxl-s805x-libretech-ac.dtb ../linux-image-5.4.0_5.4.0-1_arm64.deb 10.0.0.251:~/
Then on the target:
target-host:~# cp meson-gxl-s805x-libretech-ac.dtb /boot/efi/dtb/libre-computer/aml-s805x-ac/platform.dtb
target-host:~# dpkg -i linux-image-5.4.0_5.4.0-1_arm64.deb
Say a short prayer, then reboot. If all goes well, you'll boot back into the
mainline kernel and everything should work. There's a big [FAILED] entry in
the systemd log for an LED Trigger, but I haven't bothered to figure out
what's going on there because I don't care and it still glows too much for my
liking even without whatever is broken.
I am by no means an expert on this board, I just wasted a weekend beating it
into submission. The real experts are reachable via IRC at #librecomputer and
#linux-amlogic on freenode. I'm grateful for the help they gave me
when I was about to reach for the hammer. There is also a tonne of information
in the dedicated forums over at loverpi.
You could use QEMU here instead if you prefer, but I found with the limited eMMC space on my target, it was just a little large when installing the version packaged with Debian.
Before we go any further, we need to grab a bunch of essential utilities and dependencies:
target-host:~$ apt-get install gcc git libfdt-dev make
Then grab the sources:
target-host:~$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git
target-host:~$ cd kvmtool
target-host:~/kvmtool$ make -j4
Before we give our new binary a go, we'll ensure that we're in the right group:
target-host:~$ sudo usermod -a -G kvm,netdev $(whoami)
then log out and back in again.
If you built your own mainline kernel as described earlier, then you can give it a spin along the lines of:
target-host:~/kvmtool$ ./lkvm run -p "earlycon" <(zcat /boot/vmlinuz-5.4.0)
Which should boot to a basic guest shell.
The manager is going to want to ssh into both the guest (for running tests)
and the host (for observing a crashed guest). This means we need to set up a
network bridge on the host so that the two are independently addressable. We
can do this by bodging a new network interface using macvtap:
target-host:~# cat << EOF > /etc/network/interfaces.d/kvmtap0
auto kvmtap0
iface kvmtap0 inet manual
pre-up ip link add link eth0 name kvmtap0 type macvtap mode bridge
post-down ip link del kvmtap0
EOF
We also need to persuade udev to make the /dev/tap* nodes accessible to the
netdev group:
target-host:~# cat << EOF > /etc/udev/rules.d/99-tap.rules
KERNEL=="tap[0-9]*",GROUP="netdev",MODE="0660"
EOF
The interface should appear magically following a reboot, but we can also just raise it now:
target-host:~# service udev restart
target-host:~# ifup kvmtap0
Note: Although this will allow the manager machine to communicate with both
the target host and the target guest on the local network, due to the way that
macvtap works, the guest will not be visible to the host.
The guest doesn't need a lot, other than a specially-configured kernel and a filesystem with an SSH daemon.
If you already have a kernel source tree after building a mainline kernel for 'La Frite' earlier on, then we can reuse that. Otherwise, you'll want to pull them down onto your desktop box:
manager:~$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
manager:~$ cd linux
This is the kernel that we'll be fuzzing, so now is the time to apply whatever
untested rubbish you have on top. We'll use defconfig as a base:
manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
Syzkaller requires some additional kernel configuration options to operate usefully:
# Enable KCOV
manager:~/linux$ ./scripts/config -e KCOV -e KCOV_INSTRUMENT_ALL -e KCOV_ENABLE_COMPARISONS
# Enable KASAN
manager:~/linux$ ./scripts/config -e KASAN -e KASAN_INLINE
# Enable fault injection
manager:~/linux$ ./scripts/config -e FAULT_INJECTION -e FAULT_INJECTION_DEBUG_FS -e FAILSLAB -e FAIL_PAGE_ALLOC -e FAIL_MAKE_REQUEST -e FAIL_IO_TIMEOUT -e FAIL_FUTEX
# Enable kernel debugging options
manager:~/linux$ ./scripts/config -e LOCKDEP -e PROVE_LOCKING -e DEBUG_ATOMIC_SLEEP -e PROVE_RCU -e DEBUG_VM -e FORTIFY_SOURCE -e HARDENED_USERCOPY -e LOCKUP_DETECTOR -e SOFTLOCKUP_DETECTOR -e HARDLOCKUP_DETECTOR -e BOOTPARAM_HARDLOCKUP_PANIC -e DETECT_HUNG_TASK -e WQ_WATCHDOG
# Enable virtio-rng
manager:~/linux$ ./scripts/config -e HW_RANDOM -e HW_RANDOM_VIRTIO
At which point we can build a kernel fit for fuzzing:
manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig
manager:~/linux$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j $(nproc) Image
Then transfer the Image to the arm64 host:
manager:~/linux$ scp arch/arm64/boot/Image target-host:~/
Although you could probably build something manually with busybox, I find it
easiest just to use Debian once again. You can build the image directly on the
target, but first we need an empty image with a filesystem:
target-host:~$ truncate -s 4G debian-buster-arm64.img
target-host:~$ /sbin/mkfs.ext4 debian-buster-arm64.img
Building the image this way also means we don't waste disk space, since
truncate creates a sparse file. We can mount the image as follows:
target-host:~# losetup /dev/loop0 debian-buster-arm64.img
target-host:~# mount /dev/loop0 /mnt
At which point we're ready to create our new filesystem:
target-host:~# apt-get install debootstrap
target-host:~# debootstrap buster /mnt
Once that's done (you might want to grab another cup of tea), we just need to change the root password and hostname so that we can log in:
target-host:~# chroot /mnt
target-host:/# passwd
# <Set root password for guest>
target-host:/# echo "target-guest" > /etc/hostname
target-host:/# sed -i 's/localhost/localhost target-guest/' /etc/hosts
target-host:/# exit
Finally, we can clean up:
target-host:~# umount /mnt
target-host:~# losetup -d /dev/loop0
At this point, we should be able to boot our shiny new guest environment:
target-host:~# ./kvmtool/lkvm run -n mode=tap,tapif=/dev/tap$(cat /sys/class/net/kvmtap0/ifindex),guest_mac=$(cat /sys/class/net/kvmtap0/address) --rng -d debian-buster-arm64.img -k Image
You can tweak the amount of guest memory and number of virtual CPUs that it has
by passing -m and -c respectively.
Once the guest has booted, you can log in on the serial console as root, using the password you chose when creating the filesystem. From there, let's get the network up and running:
target-guest:~# echo << EOF > /etc/network/interfaces.d/eth0
auto eth0
iface eth0 inet dhcp
EOF
target-guest:~# ifup eth0
Unless you're very good at typing passwords, having syzkaller use key-based
authentication for its SSH sessions is essential. We'll get sshd going with
public key authentication for the root user, which is how syz-manager will
connect to the guest later on. From inside the guest:
target-guest:~# apt-get install openssh-server
target-guest:~# echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
target-guest:~# service ssh restart
Then back on the x86 manager machine:
manager:~$ ssh-keygen -q -N "" -f syzkaller
This will generate a public/private key pair that can be used for authentication. We just need to install the public key on the target (host and guest), which we can do from the comfort of the manager!
manager:~$ ssh-copy-id -i syzkaller target-host
manager:~$ ssh-copy-id -i syzkaller root@target-guest
For each invocation, it should report something along the lines of:
Number of key(s) added: 1
Now you can connect to the guest and remove the line we added to sshd_config
earlier on:
manager:~$ ssh -i syzkaller root@target-guest "sed -i '\$d' /etc/ssh/sshd_config"
TODO: Teach syzkaller how to deal with this setup! We're in a position where the manager can ssh to the target using key authentication, spawn a guest and then ssh into the guest.
arm64 kernel