1. The problem

This document is targeted at people cross-compiling lots of large Open Source software projects and who want to scale and generalize that effort.

Usually people understand cross-compilation as compiling a program for a CPU architecture different from the one used on the machine where the compilation is done. This is accomplished using a cross-compiler toolchain and cross-compiled libraries and specifying these in the Makefiles that build the software.

This is easy with the low-level software and compilation tools because they are required to support cross-compilation. It's a very different matter with Open Source software in general, especially with end-user applications and e.g. GUI toolkits.

Most of the Open Source software projects use a 'configure' script to configure their software for compilation. This script is produced by tools called 'autoconf' and 'automake', which process 'm4' macros written by the application developer. The script will generate the Makefiles that are used for building the software and a 'config.h' header file containing defines for features found on the build system.

Configure is meant to ease configuring the software for compilation and its default assumption is that the software will be run in the same environment in which it was compiled in and run from the place where it was installed to.

2. Configuration and installation processes

Autoconf provides application developers certain macros to check out features in the system (see MacKenzie, David & Elliston, Ben: Creating Automatic Configuration Scripts ). The problematic ones are:

AC_TRY_RUN

Tries to compile, link and run given test code for some feature. Test programs return zero for success.

AC_TRY_LINK

Tries to compile and link test code for library function existence.

AC_CHECK_LIB

Tries to compile and link test code using certain library.

AC_SEARCH_LIBS

This does the same as AC_TRY_LINK, but 'configure' tries to do linking from all of the system library paths, not just from ones given in CPPFLAGS, CFLAGS and CXXFLAGS environment variables.

Higher level software is traditionally compiled natively on Unix systems. Therefore, most of the application developers haven't taken into account cross-compilation when using these macros, so 'configure' ends up cross-compiling test code and trying to run it on build host which breaks the configuring. Configure can also find libraries, headers and versions of those that are only present on the build host, not on the target, which will fail either compilation of the program or running it on the target.

The worst thing is that you don't have any idea whether 'configure' found the correct values or not, unless you go manually through tens of thousands lines of output it produces (or the binary fails when it's run). Even if you've checked the standard things, application developers can and have written also their own m4 test macros and test programs for 'configure' to run.

When software is configured, the installation path is often written into sources. This installation path is often included also into numerous configuration files, dynamically loaded libraries etc. And the software doesn't work if it doesn't find it's configuration files in the specified place!

You usually need 'root' user rights to be able to install software into same path on the build host that it will be run on the target. Then you have in your build host system directories a program that doesn't run on that machine, and to test and run the program on the target device, you have to first track all of its files.

3. Summary and current solution

Configure itself is not a problem. It solves very well the problem it was designed for; configuring software for system of which the application developer (who made the software) doesn't know anything, but which the configure script can find out information about. The problem is how to limit and be sure about what it finds, how to let it test whether the found functionality actually works as expected and to make it install the software into correct place, both for the build host testing and the target device.

Current solution in many projects using already existing Open Source software is to configure and compile it natively. On real embedded development this solution is not optimal. It does not scale and embedded devices, even developer editions, are usually much slower than developer's x86 workstations and they have space constraints.

1. Build Host

All the targets inside the SDK sandbox use the same host-compiled utilities (needed by 'configure' and other configuration utilities) which are run only on the build host. The compiler toolchain and libraries are specific to each of the sandbox target CPUs / C-library combination.

As the binaries produced by the PC target x86-toolchain can be run on your x86-desktop, we call this target the native target.

2. Target

Target device will have two target systems:

• Root image and packages produced as the result of the cross-compilation and removing of unnecessary files. This is the system that will be put on target devices when they are ready. We call it the product target system.

• One used by the cross-compilation sandbox on the build host. This is used only in the product development, inside the sandbox. We call it the development target system.

Ideally you would have two target devices. One is used for developing the software and unit testing it with the development target system and second one is where you do all the integration and IPC tests for the final system using the produced product target system.

Development target system is the one you use for CPU transparency.

1. Test system

Our build host machine is a x86 Linux desktop machine running RedHat 8 with glibc2 as the shared C-library and comes with all the normal desktop Linux libraries and utilities installed.

We're tried both uClibc and glibc as the shared C-libraries and dynamically linked Busybox for the command line tools. Target CPU is ARM.

2. Problems

1. First problem of sandboxing 'configure' is that it needs a lot of (POSIX standard) software, which won't be available on the embedded target environment.

2. Second problem is that many 'configure' scripts test for features by compiling programs (with the supplied cross-compiler tool chain) and then running them. This doesn't work if program is compiled for a different CPU architecture than on which 'configure' tries to run it.

1. Strace

A utility which traces for given program what kernel calls it does and what are their arguments and return values. This works almost on any CPU that Linux kernel work on.

2. GDB

A symbolic source code debugger. You need to recompile your program to include debug information to fully benefit from GDB. For each CPU, you need to compile GDB server (containing network debug stubs) and normal x86-version of GDB which understands that CPU . Server is run inside the CPU specific sandbox or on the target, and GDB on the build host. This way you can have those huge debug binaries only on the build host and the GDB server on the target can use stripped versions of them. Additional benefit is that you can use any of the GUI GDB front-ends on the build host to run GDB.

3. Valgrind

A dynamic library for testing all kinds of programming errors by running the program on a simulated x86-CPU. For example, Valgrind can detect stack, heap and kernel call memory access errors, memory leaks, threading, and bad x86-CPU cache utilization. It is only for the x86 target as CPU simulation slows the program execution significantly and consumes more memory.

1. User Mode Linux

UML, the User Mode Linux is a Linux kernel that can be run inside another Linux instance. You point it to an image file containing the root filesystem and it boots from there. It's a great way to test new kernel versions and debug drivers. In the latest 2.6 Linux kernel UML is a standard compilation option.

In our case UML with the root image containing the above described "Target system" will provide the sandbox. Sandbox will also need to access the source files created and maintained outside it, this can be done by exporting the source directory with NFS from the build host, and mounting this inside the UML.

Currently UML has some minor problems and it's not included in the stable kernel. As UML is included in the 2.6 kernel, these issues should be fixed pretty soon. UML and NFS overhead will make compilation significantly slower.

2. Chroot

Chroot is a kernel facility and user-space utility to limit program into given directory in the filesystem. Chrooted programs can't access any files outside this directory, this includes devices and dynamic libraries. Using symbolic links doesn't work either because program can't access the file pointed to by the link.

If you don't need or want to use UML, you can use 'chroot' to limit the configure environment to what will be on the target device. Just 'chroot' the shell which you use to do the compilation. Using chroot is faster than using UML and SDK tools can be on a normal build host directory instead of a separate root image.

If you want to do testing also in the chroot environment, you need to be able to use device files. To accomplish this, you can either:

• 'mknod ' the device files your target system requires into /dev/ -directory in the sandbox. For the device major and minor numbers, look into the /dev/ - directory of your build host, they are same on all the Linux versions. Note that several programs (e.g. ssh) will act funny if they can't access certain "system" devices like /dev/zero .

• Or just copy your whole /dev/ directory inside the sandbox. In Scratchbox /dev/ is copied inside the SDK and that copy is then mounted with -bind option inside to each SDK users' own Scratchbox sandbox.

If you want regular users to be able to use 'chroot' after you've setup the sandbox, install the chroot-uid.c program as suid root. Note that users can then use this program to fool other suid root programs. It's mainly intended for users who already have root privileges, but would rather do development as a normal user. In the long run UML is a better alternative, once it is complete.

1. User Mode Linux

UML offers network interface to the host machine. You can use X11 through the network. The downside is that you can't use shared memory extension which will slow significantly image operations. See tutorial .

UML offers also access to devices in case you have full X11 inside your sandbox. For this the sandbox user will need access to the required devices.

2. Chroot

With chroot you don't necessarily need to go through the network for service access, you just need to get the server Unix domain socket available inside the sandbox. Most of the servers put their socket into /tmp/ directory.

In Scratchbox we've mounted the system /tmp/ directory with the -bind option to each users own Scratchbox sandbox.

1. Introduction

If you have a network enabled device with CPU compatible to your target device and enough memory, Linux provides a mechanism which can be used to make the CPU architecture transparent to the software we're configuring.

2. Miscellaneous binary formats

Linux kernel includes support for running miscellaneous binary formats transparently to the user and user-space programs. This support is in a module called binfmt_misc which can also be compiled statically to the kernel. You configure it by echoing following values to the '/proc/sys/fs/binfmt_misc/register ' file:

• Name of the binary format

• How to identify the format

• "Interpreter" that should be invoked to run the binary

See ARM identification for binfmt_misc for an example.

In our case the "interpreter" will be a program that executes the binary on the target device and forwards the standard input, output and error streams, environment variables and execution error code between the build host and target. This program has to be compiled so that it works in the sandbox also when it's configured for the target CPU.

'ssh' can be used on build host and 'sshd' on the target for doing the above. SSH keys required for automating this can be on the NFS directory and these keys shouldn't be used for anything else. Unfortunately, with ssh you will need to transfer the environment variables separately.

To automate all this, we've developed 'sbrsh / sbrshd ' client-server software that automates all the required interaction (NFS mounting, environment variable transfer, I/O redirection, error returning, chrooting) between build host and target device.

You can get the binary CPU architecture signature for binfmt_misc configuration from the '/usr/share/magic ' file used by the 'file ' utility.

3. Target device setup

You need to have run-time environment similar to the sandbox on the target device (or device having the same CPU as the target). This includes the shell and dynamic libraries, not the utilities needed by the 'configure'.

This requires networking as you need to NFS-mount the directory where you're doing the source configuration on the build host, into same place (in directory hierarchy) on the target device as it's on the build host sandbox. This way target will have the same cross-compiled programs as build host, and any files that the executed program writes on the target device, will be in the same place on the build host sandbox.

This setup is called the "Development target environment".

4. Fooling configure

How the above works:

• 'configure' tests the system and won't find any libraries and include files located outside the sandbox.

• 'configure' creates and cross-compiles a test program and tries to run it.

• Kernel binfmt_misc module will note a registered binary format being 'executed', and runs it using the given interpreter.

• "interpreter" will run the binary on the target device from NFS mounted development directory and redirect all of it's input and output between target and build host.

• Result is that proper files are updated through NFS and program input and output forwarded correctly so that everything works for 'configure' as it were doing things natively.

5. Summary

This works both with the chroot and User Mode Linux and solves the second problem.

Because CPU is transparent to the sandbox, testing the cross-compiled binaries can be done in the pre-configured target CPU sandbox without manually logging to the target and copying the files. This requires NFS to be set up correctly (including the firewall settings of the host machine). When doing the tests, it's better check that you're doing them using the correct CPU target environment. Note that you can do only tests that don't require processes to communicate with each other this way. For integration tests you need the target product system environment.

A single development target device can be enough for a whole developer group for the target configuration / compilation and testing phase.

1. Introduction

You can do development for the target environment with SDK sandbox containing only the x86 target toolchain.

You need the target device and cross-compilation support only for testing CPU architecture specific features and of course making a target specific binary release of your software. For this it's enough that you have a device with the correct CPU, network interface supported by Linux and enough RAM. It doesn't need to be the exact device for which you're doing the development for, the same CPU (family) is enough (e.g. iPAQ for StrongArm target).

Some of the target specific kernel device driver testing can be done using x86 emulated device drivers in User Mode Linux and a x86-sandbox. This should be much easier than native testing when your final target has limited resources and/or no network.

2. Target compilation

If you don't have a target device with networking capabilities or are otherwise lacking a target device, you can still do something with the software 'configure' scripts. Having the above described SDK will deal with the issues of 'configure' finding wrong libraries, versions, header files etc.

Then you have to fix just everything on configure scripts that tries to run cross-compiled binaries. Some possible ways of dealing with this:

• Ifdeffing (the m4 way) those lines in the scripts and providing the required values otherwise e.g. by using configure cache values and environment values.

• For certain things you can e.g. compile the programs on x86 and make configure to use your pre-compiled version of the test program. You still have to check the output for CPU architecture specific differences, with which our solution above deals with.

• Make the binfmt_misc solution to catch all the cross-compiled program runs and return a correct value.

Here are a few examples of things you have to do to Glib2 to get it to cross-compile on a normal system: patch for cross-compiling glib2 , Testing GLib cross compilation .

In our opinion this is suitable way for fixing only a small number of software build systems.

3. CPU emulators

We tested two ARM emulators to see whether they could be used instead of a real device with target CPU.

Neither of these emulators could be used with the sandbox, either because their emulation of the target CPU was incomplete, they were too slow (several orders of magnitude) or they needed an environment which would have made the interaction with the sandbox environment too difficult.

For other CPUs (e.g. Motorola 68xxx family) there are other emulators, but we would assume the problems to be similar and in most cases our solution to be most convenient.

1. Dynamic vs. static linking

First we though that by compiling SDK tools statically we would avoid the problem that dynamic C-library inside the SDK is different and that it can be for different CPU architecture than the one which SDK tools are compiled against.

However, that didn't work because glibc compiled programs can't be compiled completely statically, nss (which resolves the user names and groups for 'ls' for instance) and pam (checks user password and other information in /etc/passwd) components get their functionality from runtime loaded dynamic libraries which in turn depend on dynamic glibc library. Recompiling glibc would have had the disadvantage that we don't know what name service and security module functionality developer would need in compiled into glibc and then SDK tool compilation would have included also glibc compilation which we would like to avoid.

Finally we solved this by forcing host linker to hardcode the dynamic library load path into SDK tool binaries and copying all the dynamic library dependencies (+ nss and pam modules) inside SDK into that hardcoded library location from the SDK build host. That way SDK tools don't find dynamic libraries intended for the target target software and target software configure scripts don't find host libraries.

2. Gcc cross-dependency mess

Building of the base C and C++ libraries in integrated into GCC build. This is very awkward from the cross-compilation point of view because then dynamic applications compiled with gcc will depend from the C-library against which gcc itself was compiled with.

E.g. gcc v3.2 will compile stdlibc++ library (containing STL etc.) along with the g++ compiler. So, this library will now depend from the C-library with which g++ was compiled with and C++ applications compiled against that library will then also depend from C-library with which g++ was compiled with. Because libstdc++ is over 1MB, using it statically is not an option as libgcc.a is for gcc. Why can't they release gcc base C-libraries as a separate package which is always released with the gcc package?

Because of above, you can't build compilers like this: compile gcc / g++ for each target CPU, compile C-libraries, compile C++ libraries, compile applications.

Instead you have to do it like this: compile first-phase gcc, compile C-library with that gcc version, re-compile whole gcc toolchain with the resulting C-library, then compile applications.

The result is that you need a separate toolchain for each CPU / C-library combination. It would have been nice to keep gcc toolchains as part of the SDK and only C / C++ -libraries target specific.

3. ARM identification for binfmt_misc

ARM identification for binfmt_misc

#!/bin/sh # register arm_runner to run ARM ELF binaries echo :arm_runner:M::\x7fELF\ \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x28:\ \xff\xff\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ \xff:/bin/arm_runner: > /proc/sys/fs/binfmt_misc/register