Chapter 103: Kernel Subsystems: The Linux Networking Stack

Chapter Objectives

By the end of this chapter, you will be able to:

• Analyze the architecture of the Linux kernel networking stack, including the flow of data from the hardware interface to user space.
• Explain the role and structure of the socket buffer (sk_buff) as the fundamental unit of data transmission within the kernel.
• Configure the Linux kernel build system (menuconfig) to support specific networking protocols, hardware drivers, and firewalling features relevant to the Raspberry Pi 5.
• Implement a basic kernel module that utilizes Netfilter hooks to inspect and analyze network packets in transit.
• Debug common networking issues at the driver and protocol level using standard Linux tools and kernel introspection methods.
• Evaluate the trade-offs between interrupt-driven and polling-based (NAPI) network interface processing in embedded systems.

Introduction

In the modern landscape of embedded engineering, a device that cannot communicate is often a device of limited utility. The “Internet of Things” (IoT) is not merely a marketing term; it is the operational reality of contemporary embedded systems design. Whether you are designing a remote telemetry unit for an industrial pipeline, a smart home controller, or an autonomous drone, the ability to transmit and receive data reliably is paramount. At the heart of this capability lies the Linux kernel networking stack, a subsystem renowned for its stability, scalability, and strict adherence to global standards.

For students and professionals working with platforms like the Raspberry Pi 5, understanding the networking stack is more than just knowing how to set an IP address. It requires a journey into the kernel’s interior to understand how electrical signals on a wire—or radio waves in the air—are transformed into meaningful application data. In previous chapters, we explored the intricacies of toolchains and the general construction of the kernel image. We established how to build a bootable system. Now, we must refine that build to handle the complex, asynchronous nature of network traffic.

This chapter bridges the gap between the physical network hardware of the Raspberry Pi 5—specifically its Gigabit Ethernet controller and WiFi capabilities—and the user-space applications that rely on them. We will move beyond simple usage to explore the architectural decisions that make Linux the operating system of choice for the world’s internet infrastructure. We will dissect the life of a packet, explore the critical data structures that define connectivity, and write code that interacts directly with the flow of information. Mastering this subsystem is essential for optimizing throughput, minimizing latency, and ensuring the security of your embedded deployment.

Technical Background

The Linux networking stack is a layered architecture that strictly adheres to, yet pragmatically implements, the theoretical OSI (Open Systems Interconnection) model. Unlike a monolithic block of code, the stack is a modular, hierarchical system designed to abstract the complexity of hardware from protocols, and protocols from applications. To understand this system, one must visualize it not as a static structure, but as a dynamic pipeline where data is continuously encapsulated, decapsulated, routed, and transformed.

The Core Architecture: A Layered Approach

At the lowest level of the software stack sits the Device Interface Layer. This is where the kernel interacts directly with the hardware via device drivers. On the Raspberry Pi 5, this involves specific drivers for the RP1 I/O controller which manages the Ethernet MAC. The driver’s primary responsibility is to facilitate the transfer of data between the physical medium and the system memory. This layer abstracts the specific register-level operations of the hardware into a standardized interface known as net_device. The kernel does not care if the underlying hardware is a fiber optic card, a copper Ethernet port, or a loopback virtual device; via the net_device abstraction, they all appear functionally identical to the upper layers.

Above the device drivers lies the Network Layer, dominated in most modern systems by the Internet Protocol (IP). This layer is responsible for routing and logical addressing. When a packet arrives, the network layer must decide its fate: is this packet destined for the local system, or should it be forwarded to another host? This decision logic, known as the routing subsystem, is highly complex and integrated with the Netfilter framework, which allows for packet filtering (firewalling) and Network Address Translation (NAT). It is here that the kernel makes critical decisions based on routing tables and policy rules, determining the path data takes through the complex web of interconnected networks.

Ascending further, we reach the Transport Layer. While the network layer handles the “hop-by-hop” movement of data, the transport layer manages “end-to-end” communication. The two dominant protocols here are the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). TCP is a connection-oriented protocol that guarantees delivery, ordering, and data integrity. The kernel maintains a complex state machine for every active TCP connection, handling handshakes, retransmissions, and congestion control algorithms that adapt to network conditions. Conversely, UDP provides a lightweight, connectionless service, favoring speed over reliability, often used in real-time embedded applications like video streaming.

Finally, at the top of the kernel stack is the Socket Layer. This serves as the system call interface between the kernel and user-space applications. When a programmer writes C code using socket(), bind(), or connect(), they are interacting with this layer. The socket layer abstracts the complexities of the transport and network layers into a file-descriptor-based interface, allowing applications to treat network connections much like files on a disk—reading and writing streams of bytes without worrying about checksums, sequence numbers, or routing tables.

The Currency of the Kernel: The Socket Buffer (sk_buff)

If the networking stack is a factory, the Socket Buffer, or sk_buff, is the standardized container that moves along the conveyor belt. It is arguably the most important data structure in the Linux networking subsystem. Understanding sk_buff is the litmus test for any kernel network developer.

In a naive implementation of a network stack, one might copy data repeatedly as it moves up or down the layers. For example, the IP layer might copy the payload to add its header, and then the Ethernet layer might copy it again to add the MAC header. This copying is computationally expensive and disastrous for performance. The Linux kernel solves this via the sk_buff design. The structure contains pointers to a block of memory that holds the packet data. Instead of copying the data, the kernel simply manipulates pointers.

Imagine the sk_buff as a sophisticated envelope. When an application sends data, the kernel allocates a buffer and places the data in the middle. As the packet descends through the stack, the TCP layer adjusts a pointer to reserve space at the front of the data and writes its header. The IP layer does the same, “prepending” its header by moving a pointer, not the memory. Finally, the Ethernet driver adds the MAC header. The actual data payload never moves in memory; only the metadata describing where the packet starts and ends is updated. This zero-copy behavior is critical for the Raspberry Pi 5 to achieve Gigabit speeds without saturating its CPU cores.

Interrupts, Polling, and NAPI

In embedded systems, how the processor becomes aware of incoming data is a critical design choice. Historically, network interfaces were purely interrupt-driven. When a packet arrived, the hardware asserted an interrupt line, the CPU stopped what it was doing, ran an Interrupt Service Routine (ISR), and processed the packet. This works perfectly for low traffic. However, under high load—such as a denial-of-service attack or a burst of high-speed sensor data—this model collapses. The CPU spends all its time handling interrupts (context switching) and has no cycles left to actually process the data or run applications. This phenomenon is known as “receive livelock.”

To combat this, Linux introduced the New API (NAPI). NAPI creates a hybrid architecture combining interrupts and polling. When the first packet arrives, the network card triggers an interrupt. The kernel disables further interrupts from that card and switches to a polling mode, where it periodically checks the card’s buffer and processes a batch of packets (a “budget”) in a soft-interrupt context. Once the buffer is empty, the kernel re-enables interrupts and goes back to sleep. This approach ensures that the system remains responsive even under heavy network load, a crucial feature for the Raspberry Pi 5 when acting as a gateway or edge processor.

flowchart TD
    Start([Packet Arrives at NIC]) --> IntDrive["Interrupt Triggered<br>(CPU context switch)"]

    subgraph Transition [NAPI Hybrid Switch]
        IntDrive --> DisableInt["Disable NIC Interrupts<br>Reduce context switch overhead"]
        DisableInt --> SchedulePoll["Schedule NAPI Poll<br>Add to softirq poll list"]
    end

    SchedulePoll --> PollLoop{NAPI Polling Loop}

    subgraph Processing [In SoftIRQ Context]
        PollLoop -- "Fetch packet batch<br>(Up to budget)" --> ProcessBatch["Process sk_buffs<br>Push to IP Layer"]
        ProcessBatch --> CheckWork{Queue Empty?}
    end

    CheckWork -- "No" --> BudgetCheck{Budget Exhausted?}
    BudgetCheck -- "Yes" --> Yield["Yield CPU<br>Stay in Polling Mode"]
    Yield --> PollLoop

    BudgetCheck -- "No" --> PollLoop

    CheckWork -- "Yes" --> ReEnable["Re-enable Interrupts"]
    ReEnable --> Sleep([Go back to Sleep])

    %% Styling
    style Start fill:#1e3a8a,stroke:#1e3a8a,stroke-width:2px,color:#ffffff
    style IntDrive fill:#ef4444,stroke:#ef4444,stroke-width:1px,color:#ffffff
    style DisableInt fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff
    style SchedulePoll fill:#0d9488,stroke:#0d9488,stroke-width:1px,color:#ffffff
    style PollLoop fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff
    style ProcessBatch fill:#0d9488,stroke:#0d9488,stroke-width:1px,color:#ffffff
    style CheckWork fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff
    style BudgetCheck fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff
    style ReEnable fill:#10b981,stroke:#10b981,stroke-width:2px,color:#ffffff
    style Sleep fill:#1e3a8a,stroke:#1e3a8a,stroke-width:2px,color:#ffffff
    style Yield fill:#eab308,stroke:#eab308,stroke-width:1px,color:#1f2937

The Control Plane: Netlink

While the data plane moves packets, the control plane manages configuration. How does the ip command in user space tell the kernel to assign an address or bring an interface up? It uses Netlink. Netlink is a special socket family used for Inter-Process Communication (IPC) between the kernel and user space. Unlike ioctl, which was the historical method, Netlink is asynchronous and multicast-capable. When you plug a network cable into the Raspberry Pi, the kernel publishes a message over a Netlink multicast group. Background daemons (like NetworkManager or systemd-networkd) listen to these messages and react instantly. This event-driven architecture allows embedded Linux systems to be dynamic and self-configuring.

Hardware Integration on the Raspberry Pi 5

The Raspberry Pi 5 introduces a significant architectural shift compared to its predecessors. Previous models often relied on an internal USB bridge for Ethernet, which shared bandwidth with other peripherals and introduced latency. The Pi 5 utilizes the RP1 I/O controller, a custom piece of silicon designed by Raspberry Pi. The Gigabit Ethernet MAC is integrated into this southbridge and connects to the main BCM2712 SoC via a high-speed PCIe link.

This implies that the driver structure is closer to what one would find in a desktop PC networking card than a traditional low-speed embedded microcontroller. The kernel driver must manage Direct Memory Access (DMA) rings directly over PCIe. When a packet arrives, the RP1 writes it directly into the main system RAM via DMA and then signals the CPU. This decoupling of data movement from CPU intervention is what allows the Pi 5 to sustain high throughput with relatively low CPU usage, making it an excellent candidate for tasks like software-defined networking (SDN) or acting as a high-speed firewall.

Practical Examples

In this section, we will transition from theory to practice. We will verify the kernel configuration for a Raspberry Pi 5, explore the file system representation of network devices, and write a kernel module to hook into the networking stack.

Build and Configuration Steps

To enable the networking features we discussed, we must ensure our kernel is configured correctly. If you are building a custom kernel using Buildroot or Yocto, or compiling the upstream kernel for the Pi 5, you will interact with make menuconfig.

The networking subsystem is vast. To begin, navigate to the networking support section. The hierarchy is essential to understand dependencies.

# Navigate to the kernel source directory
cd linux/

# Open the configuration menu
make menuconfig

You must ensure the following critical options are enabled. Note that some may be compiled as modules (<M>) or built-in (<*>). For an embedded system that requires network boot or immediate connectivity, built-in is often preferred for the core drivers.

1. Networking Support:[*] Networking supportThis is the master switch CONFIG_NET. Without this, the kernel compiles without a stack.
2. Networking Options:Networking support -> Networking optionsHere you select the protocols.
   • <*> Packet socket: Essential for tools like tcpdump and Wireshark (CONFIG_PACKET).
   • <*> Unix domain sockets: Required for local IPC (CONFIG_UNIX).
   • <*> TCP/IP networking: The core stack (CONFIG_INET).
   • <*> IP: kernel level autoconfiguration: Useful for diskless boots via DHCP (CONFIG_IP_PNP).
3. Network Device Support:Device Drivers -> Network device supportThis is where hardware meets software. For the Raspberry Pi 5:
   • Ethernet driver support -> Broadcom devices
   • Ensure the specific genet/bcm2711/RP1 related drivers are selected.

After saving your configuration, you would proceed to build the kernel and modules:

# Compile the kernel, modules, and device tree blobs
make -j4 Image modules dtbs

# Install modules to a temporary location for deployment
make modules_install INSTALL_MOD_PATH=./output/

Hardware Integration: The Device Tree

On the Raspberry Pi 5, the hardware description is handled by the Device Tree. We don’t hardcode memory addresses in the C code; instead, the driver reads the .dts file at boot. Let’s look at a conceptual snippet of how the Ethernet controller is defined. You can find this in arch/arm64/boot/dts/broadcom/.

/* Conceptual Device Tree Fragment for RPi Ethernet */
&ethernet0 {
    compatible = "brcm,bcm2711-genet-v5";
    reg = <0x0 0xfd580000 0x0 0x10000>; /* Physical address of registers */
    interrupts = <GIC_SPI 157 IRQ_TYPE_LEVEL_HIGH>;
    phy-mode = "rgmii";
    phy-handle = <&phy1>;
    status = "okay";

    mdio {
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        
        phy1: ethernet-phy@1 {
            reg = <0x1>; /* MDIO address of the PHY chip */
        };
    };
};

Interpretation:

• compatible: Tells the kernel which driver code to load (Generic Network Controller).
• reg: The memory-mapped I/O address where the CPU can talk to the Ethernet hardware.
• phy-mode: Defines the electrical interface between the MAC (controller) and the PHY (physical transceiver). “rgmii” is common for Gigabit.

Code Snippet: The Netfilter Hook

To truly understand the stack, we will write a kernel module that intercepts packets. We will uses the Netfilter framework. We will register a hook function that gets called for every packet arriving at the IP layer.

Create a file named packet_logger.c.

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/netfilter.h>
#include <linux/netfilter_ipv4.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/udp.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Embedded Systems Instructor");
MODULE_DESCRIPTION("A simple packet logger using Netfilter");

static struct nf_hook_ops nfho;

// The function that will be called for every packet
unsigned int hook_func(void *priv, struct sk_buff *skb, const struct nf_hook_state *state)
{
    struct iphdr *iph;
    
    // Safety check: ensure socket buffer is valid
    if (!skb) return NF_ACCEPT;

    // Access the IP header
    // ip_hdr() is a helper function to retrieve the IP header from skb
    iph = ip_hdr(skb);
    
    if (!iph) return NF_ACCEPT;

    // Log packets destined for a specific port (e.g., SSH on 22) purely as an example
    // We use %pI4 to print IP addresses in dotted-decimal format
    if (iph->protocol == IPPROTO_TCP) {
        printk(KERN_INFO "Packet Logger: TCP packet from %pI4 to %pI4\n",
               &iph->saddr, &iph->daddr);
    }

    // NF_ACCEPT allows the packet to continue through the stack
    // NF_DROP would silently discard it
    return NF_ACCEPT; 
}

static int __init packet_logger_init(void)
{
    printk(KERN_INFO "Packet Logger: Initializing...\n");

    // Fill in the hook structure
    nfho.hook = hook_func;              // Function to call
    nfho.hooknum = NF_INET_PRE_ROUTING; // Where to hook (Before routing decision)
    nfho.pf = PF_INET;                  // Protocol Family (IPv4)
    nfho.priority = NF_IP_PRI_FIRST;    // Priority (Highest)

    // Register the hook
    nf_register_net_hook(&init_net, &nfho);
    
    return 0;
}

static void __exit packet_logger_exit(void)
{
    // Unregister the hook to prevent crashes on unload
    nf_unregister_net_hook(&init_net, &nfho);
    printk(KERN_INFO "Packet Logger: Exiting...\n");
}

module_init(packet_logger_init);
module_exit(packet_logger_exit);

Explanation:

This code demonstrates the power of kernel access. We define a hook_func. By registering this at NF_INET_PRE_ROUTING, our function sees every IPv4 packet entering the device before the kernel decides where to send it. We use the sk_buff structure (passed as skb) to extract the IP header. The macro %pI4 is a special kernel format specifier for printing IP addresses. Note the return value NF_ACCEPT; if we returned NF_DROP, we would effectively create a black-hole firewall.

Makefile for Compilation

To build this module, you need the kernel headers installed on your Raspberry Pi (or the cross-compilation environment set up).

obj-m += packet_logger.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Flash and Test Procedures

1. Compile: Run make in the directory containing your C file and Makefile.
2. Load: Use sudo insmod packet_logger.ko to load the module.
3. Verify: Check the kernel ring buffer using dmesg -w.
4. Generate Traffic: Ping the Raspberry Pi from another computer.
5. Observe: You should see “Packet Logger: TCP packet…” (if you initiate a TCP connection, like SSH) or modify the code to log ICMP for ping.
6. Unload: Always remove the module cleanly with sudo rmmod packet_logger before modifying code.

Common Mistakes & Troubleshooting

Working with the networking stack is notoriously difficult because errors often manifest as “silence”—packets simply don’t arrive. Here are the most frequent pitfalls developers encounter:

Exercises

These exercises are designed to progress from userspace observation to kernel-space modification.

1. Userspace Trace Analysis:Use tcpdump on the Raspberry Pi 5 to capture a full handshake of an SSH connection. Save the capture to a file. Then, use Wireshark (on your host PC) to open the file. Identify the Ethernet header, IP header, and TCP header. Correlate the hexadecimal values in the “Raw View” with the fields discussed in the Technical Background. Specifically, identify the source and destination MAC addresses and IP addresses manually.
2. Kernel Configuration Minimization:Starting with the default Raspberry Pi kernel configuration (bcm2711_defconfig), use menuconfig to create a “minimal” networking kernel. Disable IPv6, Wireless support (802.11), and Bluetooth. Compile this kernel and boot the Raspberry Pi. Verify using ip addr that only the Ethernet interface and Loopback are present. This exercise reinforces understanding of granular build configuration.
3. The Drop Module:Modify the packet_logger.c example provided in the chapter. Instead of just logging packets, change the logic to drop all incoming ICMP (Ping) packets (IPPROTO_ICMP) while allowing all other traffic. Load the module and attempt to ping the Pi from your PC. The ping should fail (timeout), but you should still be able to SSH into the device. This demonstrates the mechanics of a basic firewall.
4. Device Tree Exploration:Locate the live Device Tree on your running Raspberry Pi at /proc/device-tree/. Navigate through the directories to find the Ethernet controller node (usually under scb or soc). Use the hexdump or cat command to read the compatible and reg properties. Compare these values with the source code .dts files in the Linux kernel source tree. This validates how the static source code translates into the runtime hardware description.

Summary

• The Linux Networking Stack is a modular, layered implementation of the OSI model, abstracting hardware complexities from user applications.
• Socket Buffers (sk_buff) are the core data structure, utilizing pointer manipulation rather than data copying to ensure high performance and low latency.
• NAPI (New API) combines interrupts and polling to prevent receive livelocks and effectively manage high-throughput traffic on embedded devices like the Raspberry Pi 5.
• Netfilter provides hooks into the stack, enabling powerful packet manipulation, firewalling, and NAT capabilities directly within the kernel.
• Device Trees are essential for defining the hardware parameters of network interfaces on ARM-based systems, decoupling driver code from board-specific details.
• Kernel configuration via menuconfig allows developers to tailor the networking subsystem, stripping unnecessary protocols to reduce footprint or enabling advanced features for specific use cases.

Further Reading

1. The Linux Foundation – Linux Kernel Networking Documentation.official documentation within the kernel source tree, specifically Documentation/networking/.
2. Benvenuti, C. (2006). Understanding Linux Network Internals. O’Reilly Media.Though older, this text remains the definitive reference for the architectural logic of the stack.
3. Corbet, J., Rubini, A., & Kroah-Hartman, G. (2005). Linux Device Drivers. O’Reilly Media.The standard reference for writing drivers, including network interface drivers.
4. Rosen, R. (2013). Linux Kernel Networking: Implementation and Theory. Apress.Provides a modern look at the stack, including newer features like NAPI and Netlink details.
5. Raspberry Pi Foundation – Linux Kernel Documentation.Specifics on the BCM2712 and RP1 architecture updates for the Raspberry Pi 5.
6. LWN.net – Networking Namespace and Architecture Articles.Authoritative, technical articles on kernel development updates and architectural changes.