Wednesday, 24 October 2018

Linux Kernel Cross Compilation

Introduction

There are several reasons to cross-compile the Linux kernel:
  1. Compiling on a native CPU and hardware requires a wide range of devices, which is not so practical. Furthermore, the actual hardware is often not suitable for the workload of kernel compilation due to a slow CPU, small memory, etc.
  2. Hardware emulation (e.g., with qemu) can be a viable substitution if the slowness can be tolerated. However, how to setup the compilation environment is another challenge, as it requires at least a preferably Linux-flavored OS, the bash and make ecosystem, and most importantly, the toolchain.
  3. Cross-compiling on a powerful host enables quick detection of kernel compile errors and config errors, as shown in the stable queue builds project. Coupled with emulation, testing on non-native architectures becomes easier as well.
For my personal use, I would like to see the kernel build process on each architecture, capture the compilation flags of each files, collect their linking information, and finally see if I can mine some insights out of it. In this case, given the complexity of parsing the Kconfig and Makefile, probably the best way is to actually build the kernel and dump the verbose version of the build log.
Fortunately, with a modern Linux release (like Ubuntu or Fedora), all we need to cross-compile the kernel is a toolchain that can produce binaries for another CPU architecture.

Toolchain

On cross-compiling the kernel, we only need two things: binutils, and gcc, and there are generally two ways to obtain them:
  1. install pre-compiled packages
  2. build from source
Installing pre-compiled packages can be a hassle-free way if you don't want to get your hands dirty. In fact, for the following architectures: arm, aarch64, hppa, hppa64, m68k, mips, mips64, powerpc, powerpc64, s390x, sh4, sparc64, you can directly obtain them via apt-get from the official repositories. For example, 
apt-get install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu
Toolchains for other architectures such as blackfin, c6x, and tile can be obtained from the Fedora repo and converted to .deb packages as shown in the tutorial in 2013.
However, in case you want to compile the toolchain from source, for whatever reasons you might have, such as using the latest version or a customized version of gcc, you might follow the following steps.
As a head up, I am running a standard Ubuntu 16.04.3 LTS release with kernel version 4.4.0. The host gcc and binutils (i.e., the gcc and binutils used to build the toolchain) are the default ones with the Ubuntu release, which is in version 5.4.0 and 2.27 respectively. However, I don't see major obstacles in applying it to other combinations of OS, gcc, and binutils versions as long as they are recent enough to build the toolchain.
Before diving into the building process, let's setup some environment variables. Part of them are for convenience reasons, while others are necessary in the build process.
export TARGET=aarch64-unknown-linux-gnu     # replace with your intended target
export PREFIX="$HOME/opt/cross/aarch64"     # replace with your intended path
export PATH="$PREFIX/bin:$PATH"
The TARGET variable should be a target triplet which is used by the autoconf system. Their valid values can be found in the config.guess script.
The last step is necessary. By adding the installation prefix to the the PATH of the current shell session, we ensure the gcc is able to detect our new binutils once we have built them.

binutils

The source code for binutils can be obtained from the GNU servers:
# for stable releases
BINUTILS_VERSION=2.29.1
wget ftp://ftp.gnu.org/gnu/binutils/binutils-$BINUTILS_VERSION.tar.xz

# for development branch
git clone git://sourceware.org/git/binutils-gdb.git
After that, modify the source code as you wish and build it with the following steps.
cd $BINUTILS_BUILD

$BINUTILS_SOURCE/configure \
  --target=$TARGET \
  --prefix=$PREFIX \
  --with-sysroot \
  --disable-nls \
  --disable-werror

make
make install
Since we have multiple targets to build, it is better to have a separate build directory for each target.
--disable-nls tells binutils not to include native language support. This is optional, but reduces dependencies and compile time.
--with-sysroot tells binutils to enable sysroot support in the cross-compiler by pointing it to a default empty directory. By default the linker refuses to use sysroots for no good technical reason, while gcc is able to handle both cases at runtime.
--disable-werror behaves exactly as the name suggests, do not add -Werror in the flag.
After installation, the binaries aarch64-unknown-linux-gnu-{as/ar/ld/...} should exist in $PATH.
These instructions apply to binutils version 2.29.1, which is the latest release at the time of writing.

gcc

Similar to binutils. the source code for gcc can be obtained from the GNU servers too:
# for stable releases
GCC_VERSION=7.2.0
wget ftp://ftp.gnu.org/gnu/gcc/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.xz

# for development branch
git clone git://gcc.gnu.org/git/gcc.git
After that, modify the source code as you wish and build it with the following steps.
cd $GCC_SOURCE
./contrib/download_prerequisites

cd $GCC_BUILD
$GCC_SOURCE/configure \
  --target=$TARGET \
  --prefix=$PREFIX \
  --enable-languages=c,c++ \
  --without-headers \
  --disable-nls \
  --disable-shared \
  --disable-decimal-float \
  --disable-threads \
  --disable-libmudflap \
  --disable-libssp \
  --disable-libgomp \
  --disable-libquadmath \
  --disable-libatomic \
  --disable-libmpx \
  --disable-libcc1

make all-gcc
make install-gcc
Downloading the pre-requisites (gmp, mpfr, mpc, isl) areis necessary as gcc needs these packages for compilation. This is handled seamlessly with the download_prerequisites script.
--enable-languages=c,c++ tells gcc not to build frontends for other languages like Fortran, Java, etc.
--without-headers tells gcc not to rely on any C library being present for the target.
--disable-<package> tells gcc not to build those packages as they will not be needed in kernel compilation.
More importantly, we should not simply make all as that would build way too much (for example, the libgcc, libc, libstdc++, etc), which are not needed at all. All we need is the compiler itself which can be built by make all-gcc.
Another important note is that in order for gcc to lookup the correct set of binutils, both $TARGET and $PREFIX must be exactly the same when configuring binutils and gcc. For example, aarch64-unknown-linux-gnu-gcc will lookup aarch64-unknown-linux-gnu-as in the same directory: merely putting aarch64-unknown-linux-gnu-as in $PATHis not enough.
The instructions apply to gcc version 7.2.0, which is the latest release at the time of writing.
After the whole process, the following files should present in the $PREFIX/bin directory, and also in the $PATH.
aarch64-unknown-linux-gnu-addr2line
aarch64-unknown-linux-gnu-ar
aarch64-unknown-linux-gnu-as
aarch64-unknown-linux-gnu-c++
aarch64-unknown-linux-gnu-c++filt
aarch64-unknown-linux-gnu-cpp
aarch64-unknown-linux-gnu-elfedit
aarch64-unknown-linux-gnu-g++
aarch64-unknown-linux-gnu-gcc
aarch64-unknown-linux-gnu-gcc-7.2.0
aarch64-unknown-linux-gnu-gcc-ar
aarch64-unknown-linux-gnu-gcc-nm
aarch64-unknown-linux-gnu-gcc-ranlib
aarch64-unknown-linux-gnu-gcov
aarch64-unknown-linux-gnu-gcov-dump
aarch64-unknown-linux-gnu-gcov-tool
aarch64-unknown-linux-gnu-gprof
aarch64-unknown-linux-gnu-ld
aarch64-unknown-linux-gnu-ld.bfd
aarch64-unknown-linux-gnu-nm
aarch64-unknown-linux-gnu-objcopy
aarch64-unknown-linux-gnu-objdump
aarch64-unknown-linux-gnu-ranlib
aarch64-unknown-linux-gnu-readelf
aarch64-unknown-linux-gnu-size
aarch64-unknown-linux-gnu-strings
aarch64-unknown-linux-gnu-strip

Kernel Build

With the toolchain ready, cross-compiling the kernel involves two extra steps:
  1. Find the architecture name ($KERNEL_ARCH) in kernel source tree
    • they are typically Located in arch/* in the kernel source tree.
  2. Find the configuration ($KERNEL_CONF) for each sub-arch (e.g., 32-bit vs 64-bit, big endian vs little endian, etc)
    • make ARCH=$KERNEL_ARCH help will show some hints.
    • if there is no specific machine config, the defconfig should work.
    • if there are available machine configs, choosing from an existing config seems to be more hassle free compared with doing a manual menuconfig.
Once we find the values for $KERNEL_ARCH and $KERNEL_CONF, cross-compiling the kernel is as easy as the following:
cd $KERNEL_SOURCE
make ARCH=$KERNEL_ARCH O=$KERNEL_BUILD $KERNEL_CONF

cd $KERNEL_BUILD
make ARCH=$KERNEL_ARCH CROSS_COMPILE=$TARGET- V=1 vmlinux modules
For example, in the case of building the aarch64 kernel, the command should look like the following:
cd $KERNEL_SOURCE
make ARCH=arm64 O=$KERNEL_BUILD defconfig

cd $KERNEL_BUILD
make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- V=1 vmlinux modules
Of course we can always speed up the build process with the make -j flags.
The instructions apply to Linux kernel version 4.13.5, which is the latest release at the time of writing.

Results

In total, I have currently cross-compiled the kernel for 7 architectures and 12 sub-archs, with the procedure described above. The result is summarized in the following table:
Architecture Name $TARGET $KERNEL_ARCH $KERNEL_CONF
x86 (32-bit) i386 i386-pc-linux-gnu x86 i386_defconfig
x86 (64-bit) x86_64 x86_64-pc-linux-gnu x86 x86_64_defconfig
arm (32-bit) arm armv7-unknown-linux-gnueabi arm multi_v7_defconfig
arm (64-bit) aarch64 aarch64-unknown-linux-gnu arm64 defconfig
powerpc (32-bit) ppc powerpcle-unknown-linux-gnu powerpc pmac32_defconfig
powerpc (64-bit) ppc64 powerpc64le-unknown-linux-gnu powerpc ppc64le_defconfig
sparc (32-bit) sparc sparc-unknown-linux-gnu sparc sparc32_defconfig
sparc (64-bit) sparc64 sparc64-unknown-linux-gnu sparc sparc64_defconfig
mips (32-bit) mips mips-unknown-linux-gnu mips 32r6_defconfig
mips (64-bit) mips64 mips64-unknown-linux-gnu mips 64r6_defconfig
s390x (64-bit) s390x s390x-ibm-linux-gnu s390 default_defconfig
ia64 (64-bit) ia64 ia64-unknown-linux-gnu ia64 generic_defconfig
I have also tried building an allyesconfig kernel and succeeded in 7 architectures, as shown in the following table. Unfortunately, ia64 failed miserably with error message Error: Operand 2 of 'adds' should be a 14-bit integer (-8192-8191), and the error seems to have been there for more than 7 months.
Architecture Name $TARGET $KERNEL_ARCH $KERNEL_CONF
x86 (64-bit) x86_64 x86_64-pc-linux-gnu x86 allyesconfig
arm (32-bit) arm armv7-unknown-linux-gnueabi arm allyesconfig
arm (64-bit) aarch64 aarch64-unknown-linux-gnu arm64 allyesconfig
powerpc (64-bit) ppc64 powerpc64-unknown-linux-gnu powerpc allyesconfig
sparc (64-bit) sparc64 sparc64-unknown-linux-gnu sparc allyesconfig
mips (64-bit) mips64 mips64-unknown-linux-gnu mips allyesconfig
s390x (64-bit) s390x s390x-ibm-linux-gnu s390 allyesconfig
In case you want to try out, you can directly feed the corresponding values for $TARGET, $KERNEL_ARCH, and $KERNEL_CONF into the scripts above and test.
Posted by at No comments:
Subscribe to: Posts (Atom)