Chapter 36: Intro to Cross-Compilation: Why and How

Chapter Objectives

Upon completing this chapter, you will be able to:

• Understand the fundamental concepts of cross-compilation and its importance in embedded systems development.
• Differentiate between native compilation and cross-compilation, and identify scenarios where each is appropriate.
• Set up a complete cross-compilation toolchain for the Raspberry Pi 5 on a host development machine.
• Write, compile, and deploy a simple C application to the Raspberry Pi 5 using a cross-compiler.
• Debug common issues related to cross-compilation, such as library mismatches and incorrect architecture settings.
• Appreciate the role of cross-compilation in professional embedded Linux workflows, including the use of build systems like Yocto and Buildroot.

Introduction

Welcome to the world of embedded Linux development, where the devices we build are often vastly different from the powerful machines we use to create them. In previous chapters, we have explored the Raspberry Pi 5 as a versatile and powerful platform for embedded projects. We have learned how to interact with its operating system, write simple scripts, and control its GPIO pins. However, as we move towards more complex and resource-intensive applications, we encounter a fundamental challenge: the limitations of the target device itself.

This is where cross-compilation becomes an indispensable skill. Imagine trying to build a large, complex software project, like a custom robotics control system or a real-time video processing application, directly on the Raspberry Pi 5. While possible, it would be a slow and inefficient process, consuming the Pi’s limited CPU, RAM, and storage resources. The compilation process could take hours, or even days, and the Pi would be largely unusable for other tasks during this time.

Cross-compilation provides an elegant solution to this problem. Instead of compiling our code on the target device (the Raspberry Pi), we compile it on a powerful host machine, such as a desktop or laptop computer. This process creates an executable file that is specifically designed to run on the target’s architecture, even though it was built on a different one. This allows us to leverage the superior processing power and resources of our development machines, dramatically reducing compilation times and streamlining the development workflow.

In this chapter, we will delve into the “why” and “how” of cross-compilation. We will explore the underlying concepts, understand the benefits it offers, and walk through the practical steps of setting up a cross-compilation toolchain for the Raspberry Pi 5. By the end of this chapter, you will not only grasp the theoretical importance of cross-compilation but also possess the hands-on skills to apply it to your own embedded Linux projects. This knowledge will form a critical foundation for more advanced topics, such as building custom Linux distributions and developing professional-grade embedded applications.

Technical Background

The Tale of Two Architectures: Host vs. Target

At the heart of cross-compilation lies the distinction between the host and the target. The host is the machine where you write, build, and compile your code. This is typically a powerful desktop or laptop computer running a standard operating system like Linux, macOS, or Windows. The target, on the other hand, is the embedded device where the compiled code will ultimately run. In our case, the target is the Raspberry Pi 5.

The key difference between the host and target is their processor architecture. Most desktop computers use the x86-64 architecture, while the Raspberry Pi 5 uses an ARM-based architecture (specifically, a 64-bit ARM Cortex-A76). These architectures have different instruction sets, memory layouts, and application binary interfaces (ABIs). This means that a program compiled for an x86-64 machine will not run on an ARM machine, and vice versa. It’s like trying to play a Blu-ray disc in a DVD player – the hardware simply doesn’t understand the format.

This is where the “cross” in cross-compilation comes in. A cross-compiler is a special type of compiler that runs on the host machine but generates machine code for the target’s architecture. It acts as a translator, taking your human-readable source code (written in C, C++, or another language) and converting it into a binary executable that the target’s processor can understand and execute.

The Cross-Compilation Toolchain: More Than Just a Compiler

A cross-compiler is not a single, monolithic program. It is part of a larger collection of tools known as a toolchain. A complete cross-compilation toolchain includes everything you need to build software for the target device, including:

• The Compiler (GCC): The GNU Compiler Collection (GCC) is the most widely used compiler for C, C++, and other languages in the Linux world. The cross-compiler is a version of GCC that is configured to generate code for the target architecture.
• The Binary Utilities (Binutils): This is a collection of essential tools for working with binary files, including:
  • as (the assembler): Translates assembly language code into machine code.
  • ld (the linker): Combines multiple object files and libraries into a single executable file.
  • objcopy: Copies and translates object files.
  • objdump: Displays information about object files.
  • strip: Removes symbol table information from an executable, reducing its size.
• The C Library (glibc, uClibc, musl): The C library provides the standard set of functions that C programs rely on, such as printf, malloc, and strcpy. The cross-compilation toolchain must include a version of the C library that has been compiled for the target architecture. The choice of C library can have a significant impact on the size and performance of your embedded system.
  • glibc (GNU C Library): The most common C library in the Linux world. It is feature-rich and provides a high degree of compatibility, but it is also relatively large.
  • uClibc: A smaller, lightweight C library designed for embedded systems. It provides a subset of the functionality of glibc, but with a much smaller footprint.
  • musl: Another lightweight C library that is gaining popularity in the embedded world. It is known for its clean design, static linking capabilities, and focus on correctness and standards compliance.
• The Kernel Headers: These are the header files from the Linux kernel that are needed to compile programs that interact with the kernel, such as device drivers and low-level system utilities. The kernel headers in the toolchain must match the version of the kernel running on the target device.
• The Debugger (GDB): The GNU Debugger (GDB) is a powerful tool for debugging programs. A cross-compilation toolchain includes a version of GDB that can be used to debug programs running on the target device from the host machine. This is known as remote debugging.

graph TD
    subgraph "Cross-Compilation Toolchain for Target (e.g., ARM)"
        direction TB
        
        A["<b>Cross-Compiler</b><br><i>(e.g., aarch64-none-linux-gnu-gcc)</i>"]
        B["<b>Binary Utilities (Binutils)</b><br>Tools for manipulating binary files"]
        C["<b>C Library (libc)</b><br>Standard functions for applications"]
        D[<b>Kernel Headers</b><br>Interfaces to the Linux Kernel]
        E["<b>Debugger</b><br><i>(e.g., GDB)</i> for remote debugging"]

        A --- B
        A --- C
        A --- D
        A --- E
    end

    subgraph "Binutils Details"
        direction LR
        B --> B1[ld<br><i>Linker</i>]
        B --> B2[as<br><i>Assembler</i>]
        B --> B3[objcopy<br><i>File Converter</i>]
        B --> B4[strip<br><i>Symbol Remover</i>]
    end

    subgraph "C Library Options"
        direction LR
        C --> C1[glibc<br><i>Feature-rich, large</i>]
        C --> C2[uClibc<br><i>Lightweight</i>]
        C --> C3[musl<br><i>Static-focused, clean</i>]
    end

    classDef main fill:#1e3a8a,stroke:#1e3a8a,stroke-width:2px,color:#ffffff;
    classDef process fill:#0d9488,stroke:#0d9488,stroke-width:1px,color:#ffffff;
    classDef system fill:#8b5cf6,stroke:#8b5cf6,stroke-width:1px,color:#ffffff;
    classDef decision fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff;

    class A,B,C,D,E process;
    class B1,B2,B3,B4 system;
    class C1,C2,C3 decision;

The “Sysroot”: A Virtual Root Filesystem

When you compile a program, the compiler and linker need to know where to find the necessary header files and libraries. In a native compilation environment, these files are typically located in standard system directories like /usr/include and /usr/lib. However, in a cross-compilation environment, using the host’s system directories would be a disaster. The compiler would try to link against the host’s x86-64 libraries, resulting in an executable that would not run on the ARM-based Raspberry Pi.

To solve this problem, cross-compilation toolchains use a concept called the sysroot. The sysroot is a directory on the host machine that contains a minimal version of the target’s root filesystem. It includes all the necessary headers, libraries, and other files that are needed to build software for the target. When you invoke the cross-compiler, you tell it to use the sysroot as the root of its search path for system files. This ensures that the compiler and linker only use the files that are appropriate for the target architecture.

The sysroot is a critical component of the cross-compilation process. It provides a clean, isolated environment for building software for the target, preventing any “contamination” from the host’s system files.

The Build Process: From Source Code to Target Executable

Let’s walk through the process of building a simple C program using a cross-compiler. The process can be broken down into four main stages:

1. Preprocessing: The C preprocessor (cpp) processes the source code, handling directives like #include and #define. It expands macros, includes the contents of header files, and produces a single, preprocessed source file.
2. Compilation: The compiler (gcc) takes the preprocessed source file and translates it into assembly language code for the target architecture.
3. Assembly: The assembler (as) takes the assembly language code and translates it into machine code, creating an object file. The object file contains the raw binary instructions for the program, but it is not yet a complete executable.
4. Linking: The linker (ld) takes one or more object files and combines them with the necessary libraries from the sysroot to create a final, executable file. The linker resolves any references to external functions and variables, and creates a file that can be loaded and executed by the target’s operating system.

graph TD
    subgraph "Host Machine (x86-64)"
        direction LR
        
        Source[hello_pi.c<br><i>Source Code</i>]
        
        subgraph "Cross-Toolchain Stages"
            direction TB
            
            Stage1(<b>1. Preprocessing</b><br><i>cpp</i>)
            Stage2(<b>2. Compilation</b><br><i>gcc -S</i>)
            Stage3(<b>3. Assembly</b><br><i>as</i>)
            Stage4(<b>4. Linking</b><br><i>ld</i>)
        end

        Sysroot["Sysroot<br><i>(Target ARM Libraries & Headers)</i>"]
        
        Source -- "Input" --> Stage1
        Stage1 -- "Preprocessed Code<br><i>hello_pi.i</i>" --> Stage2
        Stage2 -- "Assembly Code (ARM)<br><i>hello_pi.s</i>" --> Stage3
        Stage3 -- "Object File (ARM)<br><i>hello_pi.o</i>" --> Stage4
        Sysroot -- "Links against target libs" --> Stage4
        
        Stage4 -- "Final Executable" --> Executable{hello_pi<br><i>ARM Binary</i>}
    end

    subgraph "Target Device (Raspberry Pi)"
        direction TB
        Executable -- "Deploy (scp)" --> Deployed[hello_pi]
        Deployed -- "Run: ./hello_pi" --> Output(Output on Pi's Terminal)
    end

    classDef start fill:#1e3a8a,stroke:#1e3a8a,stroke-width:2px,color:#ffffff;
    classDef process fill:#0d9488,stroke:#0d9488,stroke-width:1px,color:#ffffff;
    classDef system fill:#8b5cf6,stroke:#8b5cf6,stroke-width:1px,color:#ffffff;
    classDef endNode fill:#10b981,stroke:#10b981,stroke-width:2px,color:#ffffff;
    classDef io fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff;

    class Source,Executable,Deployed,Output io;
    class Stage1,Stage2,Stage3,Stage4 process;
    class Sysroot system;

The Importance of a Coherent Toolchain

It is crucial that all the components of the cross-compilation toolchain are compatible with each other. The compiler, binutils, C library, and kernel headers must all be built for the same target architecture and configured to work together seamlessly. A mismatched toolchain can lead to a host of subtle and difficult-to-diagnose problems, such as segmentation faults, incorrect program behavior, and a complete failure to boot the target device.

This is why it is generally recommended to use a pre-built, tested toolchain from a reputable source, rather than trying to build one from scratch. Building a cross-compilation toolchain is a complex and error-prone process, and it is easy to make mistakes that can have far-reaching consequences. Companies like Linaro and ARM provide high-quality, pre-built toolchains for a variety of ARM-based targets, including the Raspberry Pi.

Cross-Compilation in the Real World: Build Systems

While it is possible to invoke the cross-compiler directly from the command line, this approach quickly becomes unwieldy for larger projects. Real-world embedded Linux development relies on sophisticated build systems like Yocto and Buildroot to automate the process of building a complete, custom Linux distribution for the target device.

These build systems take care of all the low-level details of cross-compilation, including:

• Downloading and patching the source code for the kernel, bootloader, and user-space applications.
• Configuring and building the cross-compilation toolchain.
• Building all the software packages for the target.
• Creating a final, bootable image that can be flashed to the target device.

While a deep dive into Yocto and Buildroot is beyond the scope of this chapter, it is important to understand that they are the industry-standard tools for professional embedded Linux development. The manual cross-compilation techniques we will learn in this chapter provide a valuable foundation for understanding how these powerful build systems work under the hood.

Practical Examples

Now that we have a solid theoretical understanding of cross-compilation, it’s time to get our hands dirty. In this section, we will walk through the practical steps of setting up a cross-compilation toolchain for the Raspberry Pi 5 and using it to build and deploy a simple C application.

Setting Up the Host Environment

Our host machine will be a computer running a modern Linux distribution, such as Ubuntu 22.04 LTS. If you are using Windows or macOS, you can use a virtual machine or the Windows Subsystem for Linux (WSL) to create a suitable development environment.

First, we need to install some essential packages on our host machine:

sudo apt-get update
sudo apt-get install build-essential git bison flex libssl-dev

These packages provide the basic tools and libraries needed for software development.

Obtaining the Cross-Compilation Toolchain

As we discussed earlier, it is best to use a pre-built toolchain. For the Raspberry Pi 5, which uses a 64-bit ARM architecture (AArch64), we can download a toolchain from ARM’s developer website.

wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

Tip: The version number of the toolchain may change over time. Be sure to check the ARM developer website for the latest version.

Once the download is complete, we need to extract the toolchain to a suitable location, such as /opt.

sudo mkdir -p /opt/toolchains
sudo tar -xvf gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz -C /opt/toolchains

This will create a directory named gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu in /opt/toolchains. To make it easier to use, let’s create a symbolic link to this directory:

sudo ln -s /opt/toolchains/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu /opt/toolchains/aarch64-rpi5

Now, we need to add the toolchain’s bin directory to our PATH environment variable. This will allow us to invoke the cross-compiler and other tools from anywhere in the filesystem.

export PATH=/opt/toolchains/aarch64-rpi5/bin:$PATH

To make this change permanent, you can add this line to your ~/.bashrc or ~/.zshrc file.

To verify that the toolchain is installed correctly, we can ask the cross-compiler for its version:

aarch64-none-linux-gnu-gcc --version

You should see output similar to this:

aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 10.3-2021.07) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Writing a Simple “Hello, World!” Application

Now that our toolchain is set up, let’s write a simple C program to test it. Create a new file named hello_pi.c and add the following code:

#include <stdio.h>
#include <unistd.h>

int main() {
    printf("Hello from the Raspberry Pi 5!\n");
    printf("This program was cross-compiled on a host machine.\n");

    // Get the hostname of the Raspberry Pi
    char hostname[256];
    if (gethostname(hostname, sizeof(hostname)) == 0) {
        printf("Running on host: %s\n", hostname);
    } else {
        perror("gethostname");
    }

    return 0;
}

This is a standard “Hello, World!” program with a small addition: it uses the gethostname function to print the hostname of the device it is running on. This will help us confirm that the program is indeed running on the Raspberry Pi.

Cross-Compiling the Application

Now, let’s compile our program using the cross-compiler. In the same directory as hello_pi.c, run the following command:

aarch64-none-linux-gnu-gcc -o hello_pi hello_pi.c

Let’s break down this command:

• aarch64-none-linux-gnu-gcc: This is the name of our cross-compiler. The name follows a standard convention: arch-vendor-kernel-os.
  • aarch64: The target architecture (64-bit ARM).
  • none: The vendor (in this case, none).
  • linux: The target kernel.
  • gnu: The target C library (glibc).
• -o hello_pi: This specifies the name of the output file.
• hello_pi.c: This is the name of our source file.

If the compilation is successful, you will have a new file named hello_pi in your directory. Let’s examine this file using the file command:

file hello_pi

You should see output like this:

hello_pi: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Notice that the output clearly states that this is an “ARM aarch64” executable. If you try to run this file on your x86-64 host machine, you will get an error:

./hello_pi

./hello_pi: cannot execute binary file: Exec format error

This is exactly what we expect. The executable is not compatible with our host’s architecture.

Deploying and Running the Application on the Raspberry Pi 5

Now, we need to get our compiled program onto the Raspberry Pi. There are several ways to do this, but one of the easiest is to use the scp (secure copy) command. You will need to know the IP address of your Raspberry Pi.

scp hello_pi pi@<raspberry_pi_ip_address>:~/

Replace <raspberry_pi_ip_address> with the actual IP address of your Pi. You will be prompted for the password for the pi user.

Once the file has been copied, open a terminal on your Raspberry Pi (either directly or via SSH) and navigate to the home directory. You should see the hello_pi file. Make it executable and run it:

chmod +x hello_pi
./hello_pi

You should see the following output:

Hello from the Raspberry Pi 5!
This program was cross-compiled on a host machine.
Running on host: raspberrypi

Success! We have successfully written, cross-compiled, and deployed a C application to the Raspberry Pi 5.

Hardware Integration: Blinking an LED

Let’s take this a step further and write a program that interacts with the Raspberry Pi’s hardware. We will write a simple program to blink an LED connected to one of the GPIO pins.

For this example, you will need:

• A Raspberry Pi 5
• A breadboard
• An LED
• A 330Ω resistor
• Jumper wires

Connect the components as follows:

• Connect the longer leg (anode) of the LED to GPIO 17 (pin 11) on the Raspberry Pi.
• Connect the shorter leg (cathode) of the LED to one end of the 330Ω resistor.
• Connect the other end of the resistor to a ground pin (such as pin 9) on the Raspberry Pi.

Now, let’s write the C code to blink the LED. We will use the libgpiod library, which is the modern, recommended way to interact with GPIO pins in Linux.

First, we need to get the libgpiod library and its header files for our sysroot. The easiest way to do this is to copy them from the Raspberry Pi itself. On your Raspberry Pi, find the location of the libgpiod library and header files:

find / -name "libgpiod.so*"
find / -name "gpiod.h"<br>

This will likely be in /usr/lib/aarch64-linux-gnu/ and /usr/include/. Copy these files to the corresponding directories in your toolchain’s sysroot on your host machine.

Warning: This is a simplified approach for demonstration purposes. In a professional workflow, you would use a build system like Yocto or Buildroot to properly manage the sysroot and its libraries.

Now, create a new file named blink.c on your host machine and add the following code:

#include <gpiod.h>
#include <stdio.h>
#include <unistd.h>

int main() {
    const char *chipname = "gpiochip4";
    unsigned int line_num = 17; // GPIO 17
    struct gpiod_chip *chip;
    struct gpiod_line *line;
    int ret;

    chip = gpiod_chip_open_by_name(chipname);
    if (!chip) {
        perror("gpiod_chip_open_by_name");
        return 1;
    }

    line = gpiod_chip_get_line(chip, line_num);
    if (!line) {
        perror("gpiod_chip_get_line");
        gpiod_chip_close(chip);
        return 1;
    }

    ret = gpiod_line_request_output(line, "blink", 0);
    if (ret < 0) {
        perror("gpiod_line_request_output");
        gpiod_line_release(line);
        gpiod_chip_close(chip);
        return 1;
    }

    printf("Blinking GPIO %d on %s\n", line_num, chipname);

    while (1) {
        gpiod_line_set_value(line, 1);
        usleep(500000);
        gpiod_line_set_value(line, 0);
        usleep(500000);
    }

    gpiod_line_release(line);
    gpiod_chip_close(chip);

    return 0;
}

Now, let’s cross-compile this program. We need to tell the compiler where to find the libgpiod library.

aarch64-none-linux-gnu-gcc -o blink blink.c -lgpiod

The -lgpiod flag tells the linker to link against the libgpiod library.

Now, deploy the blink executable to your Raspberry Pi, make it executable, and run it:

scp blink pi@<raspberry_pi_ip_address>:~/
ssh pi@<raspberry_pi_ip_address>
chmod +x blink
sudo ./blink

Warning: You may need to run this program with sudo to get access to the GPIO pins.

You should now see the LED connected to GPIO 17 blinking on and off. Congratulations! You have successfully cross-compiled a program that interacts with the Raspberry Pi’s hardware.

Common Mistakes & Troubleshooting

Cross-compilation can be a tricky process, and it’s easy to run into problems. Here are some of the most common mistakes and how to troubleshoot them:

Exercises

1. Modify the “Hello, World!” application: Modify the hello_pi.c program to also print the version of the Linux kernel running on the Raspberry Pi. You can do this by reading the contents of the /proc/version file. Cross-compile, deploy, and run the modified program.
2. Control two LEDs: Expand the blink.c program to control two LEDs connected to two different GPIO pins. Make them blink in an alternating pattern (i.e., when one is on, the other is off).
3. Create a Makefile: For larger projects, it is useful to use a Makefile to automate the build process. Create a Makefile for the blink project that can compile the code, deploy it to the Raspberry Pi, and clean up the build artifacts. The Makefile should use variables for the cross-compiler, target IP address, and other settings.

Summary

• Cross-compilation is the process of building software on a host machine for a different target architecture.
• It is essential for embedded Linux development, as it allows us to leverage the power of our development machines and avoid the limitations of the target device.
• A cross-compilation toolchain includes a compiler, binutils, C library, kernel headers, and a debugger.
• The sysroot is a critical component of the toolchain that provides a virtual root filesystem for the target.
• Build systems like Yocto and Buildroot automate the cross-compilation process for professional embedded development.
• We have learned how to set up a cross-compilation toolchain for the Raspberry Pi 5, write and compile C applications, and deploy them to the target.

Further Reading

1. ARM Developer – GNU Toolchain: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain
2. The Yocto Project: https://www.yoctoproject.org/
3. Buildroot: https://buildroot.org/
4. Raspberry Pi Documentation: https://www.raspberrypi.com/documentation/
5. libgpiod Documentation: https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/about/
6. “Mastering Embedded Linux Programming” by Chris Simmonds: A comprehensive guide to embedded Linux development, with detailed coverage of cross-compilation and build systems.
7. “Embedded Linux Systems with the Yocto Project” by Rudolf J. Streif: A practical, hands-on guide to using the Yocto Project to build custom Linux distributions for embedded systems.