qemu + syzkalling the Linux kernel

This post details how to fuzz Linux kernel using syzkaller, qemu and uclibc buildroot.

See Google internals¹ document if you are interested how syzkaller works.

Introduction

The Linux kernel is certainly a piece of software that is exposed to untrusted user input, so it is an important target for fuzzing. The kernel is also sufficiently high-profile that it has been worth writing specific, template-based fuzzers for different areas of it, such as the filesystem or the perf_event subsystem. For the system call interface in general, the Trinity² fuzz tester is commonly used. It fuzzes the kernel in an intelligent way that is driven by per-system call templates.

Vyukov and a team from Google have brought coverage-guided fuzz testing to the kernel with syzkaller, which uses a hybrid approach. As with Trinity, syzkaller relies on templates that indicate the argument domains for each system call, but it also uses feedback from code coverage information to guide the fuzzing.

The need for instrumentation does make syzkaller more complicated to set up than Trinity. To start with, the compiler option to generate the needed coverage data has only recently been added to gcc (as -fsanitize-coverage=trace-pc), so the kernel needs to be built with a fresh-from-tip version of gcc.

Test setup

The test was conducted on a Ubuntu 16.04 with gcc version 9.1.0.

I recommend using the latest stable version of gcc.

Linux kernel

Begin with building the Linux kernel from source³.

x86-64

Appy the KCOV patch for x86 if the CONFIG_KCOV is not present in the kernel⁴.

make x86_64_defconfig
make kvmconfig

Update .config to detect enabled syscalls and kernel bitness, memory leak detection, network injection, use-after-free and out-of-bounds detection (KASAN), fault injection testing, and to disable KASLR, etc.

./scripts/config \
  -e KCOV -e KCOV_INSTRUMENT_ALL -e KCOV_ENABLE_COMPARISONS \
  -e DEBUG_FS -e DEBUG_INFO \
  -e KALLSYMS -e KALLSYMS_ALL \
  -e DEBUG_KMEMLEAK --set-val DEBUG_KMEMLEAK_EARLY_LOG_SIZE 16000 \
  -e TUN \
  -e KASAN -e KASAN_EXTRA -e KASAN_INLINE \
  -e FAULT_INJECTION -e FAULT_INJECTION_DEBUG_FS \
  -e FAILSLAB -e FAIL_PAGE_ALLOC -e FAIL_MAKE_REQUEST \
  -e FAIL_IO_TIMEOUT -e FAIL_FUTEX -e FAIL_FUNCTION \
  -e GDB_SCRIPTS \
  -d RANDOMIZE_BASE \
  -e LOCKDEP -e PROVE_LOCKING -e DEBUG_ATOMIC_SLEEP -e PROVE_RCU \
  -e DEBUG_VM -e REFCOUNT_FULL -e FORTIFY_SOURCE -e HARDENED_USERCOPY \
  -e LOCKUP_DETECTOR -e SOFTLOCKUP_DETECTOR -e HARDLOCKUP_DETECTOR \
  -e BOOTPARAM_HARDLOCKUP_PANIC \
  -e DETECT_HUNG_TASK -e WQ_WATCHDOG \
  --set-val DEFAULT_HUNG_TASK_TIMEOUT 140 \
  --set-val RCU_CPU_STALL_TIMEOUT 100

Update config and build kernel with:

make olddefconfig
make -j$(nproc)

aarch64

Apply the KCOV patch for ARM if the CONFIG_KCOV is not present in the kernel⁵.

make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- defconfig

Add the configuration in chapter 2.1 and the following to .config:

./scripts/config \
  -e NET_9P -e NET_9P_VIRTIO \
  --set-val CROSS_COMPILE "aarch64-none-linux-gnu-" \
  --set-val CMDLINE "console=ttyAMA0"

Update config and build kernel with:

make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- olddefconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- -j$(nproc)

Remember to handle every configuration conflict marked with warning: override: reassigning to symbol.

uclibc buildroot

Build uclibc buildroot from source⁶.

x86-64

make x86_64_defconfig
make menuconfig

Choose the following options:

Select:

Target packages
    Networking applications
        [*] dhcpcd
        [*] iproute2
        [*] openssh
Filesystem images
        exact size - 1g

Unselect:

Kernel
    Linux Kernel

Build buildroot with:

make -j$(nproc)

aarch64

make menuconfig

Target options
    Target Architecture - Aarch64 (little endian)
Toolchain type
    External toolchain - Linaro AArch64
System Configuration
[*] Enable root login with password
        ( ) Root password = set your password using this option
[*] Run a getty (login prompt) after boot  --->
    TTY port - ttyAMA0
Target packages
    [*]   Show packages that are also provided by busybox
    Networking applications
        [*] dhcpcd
        [*] iproute2
        [*] openssh
    Miscellaneous
        [*] haveged
Filesystem images
    [*] ext2/3/4 root filesystem
        ext2/3/4 variant - ext3
        exact size in blocks - 6000000
    [*] tar the root filesystem

Build buildroot with:

make -j$(nproc)

configure

Add to output/target/etc/fstab:

debugfs	/sys/kernel/debug	debugfs	defaults	0	0

Add to output/target/etc/ssh/sshd_config:

PermitRootLogin yes
PasswordAuthentication yes
PermitEmptyPasswords yes

Run make again.

qemu

x86-64

qemu-system-x86_64 \
  -kernel $HOME/git/linux/arch/x86/boot/bzImage \
  -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
  -hda $HOME/git/buildroot/output/images/rootfs.ext2 \
  -net user,hostfwd=tcp::10022-:22 -net nic \
  -enable-kvm \
  -nographic \
  -m 2G \
  -smp 2 \
  -pidfile vm.pid

aarch64

qemu-system-aarch64 \
  -machine virt \
  -cpu cortex-a57 \
  -nographic -smp 1 \
  -hda $HOME/git/buildroot/output/images/rootfs.ext3 \
  -kernel $HOME/git/linux/arch/arm64/boot/Image \
  -append "console=ttyAMA0 root=/dev/vda oops=panic panic_on_warn=1 panic=-1 ftrace_dump_on_oops=orig_cpu debug earlyprintk=serial slub_debug=UZ" \
  -m 2048 \
  -net user,hostfwd=tcp::10022-:22 -net nic

Test if you can ssh to the device: ssh -p 10022 root@localhost.

syzkaller

build

Download golang version 1.12⁷ and update environment variables:

export GOROOT=$HOME/git/go
export PATH=$GOROOT/bin:$PATH

Build syzkaller:

go get -u -d github.com/google/syzkaller
cd $HOME/go/src/github.com/google/syzkaller

I used syzkaller commit 426631ddb41a12ad156d0254fea375a9dfa607fc.

x86-64:

make

aarch64:

make TARGETARCH=arm64

run

x86.cfg:

{
	"name": "x86",
	"target": "linux/amd64",
	"http": ":56700",
	"workdir": "/home/john/syzkaller/x86",
	"kernel_obj": "/home/john/git/linux",
	"syzkaller": "/home/john/go/src/github.com/google/syzkaller",
	"image": "/home/john/git/buildroot/output/images/rootfs.ext2",
	"sandbox": "none",
	"reproduce": false,
	"procs": 1,
	"type": "qemu",
	"vm": {
		"count": 1,
		"kernel": "/home/john/git/linux/arch/x86/boot/bzImage",
		"cpu": 2,
		"mem": 1024
	}
}

Run with syz-manager -config=x86.cfg.

aarch64.cfg:

{
	"name": "aarch64",
	"target": "linux/arm64",
	"http": ":56700",
	"workdir": "/home/john/syzkaller/aarch64",
	"kernel_obj": "/home/john/git/linux",
	"syzkaller": "/home/john/go/src/github.com/google/syzkaller",
	"image": "/home/john/git/buildroot/output/images/rootfs.ext3",
	"sandbox": "none",
	"reproduce": false,
	"procs": 1,
	"type": "qemu",
	"vm": {
		"count": 1,
		"kernel": "/home/john/git/linux/arch/arm64/boot/Image",
		"cpu": 2,
		"mem": 1024
	}
}

Run with syz-manager -config=aarch64.cfg.

kcsan

Kernel Concurrency Sanitizer⁸ (KCSAN) is a dynamic data-race detector for kernel space. KCSAN is a sampling watchpoint-based data-race detector, unlike Kernel Thread Sanitizer (KTSAN), which is a happens-before data-race detector.

I will look further how to test using KCSAN.

References