Chapter 46: Understanding the Linux Filesystem Hierarchy Standard (FHS)

Chapter Objectives

By the end of this chapter, you will be able to:

• Understand the historical context and rationale behind the Filesystem Hierarchy Standard (FHS).
• Identify and describe the purpose of all major top-level directories in a standard Linux filesystem.
• Navigate and analyze the filesystem on an embedded device like the Raspberry Pi 5 to locate critical system files.
• Explain the distinction between static and variable data, and where each should be located.
• Apply FHS principles to correctly install custom software and manage system configuration files.
• Debug common filesystem layout issues in an embedded Linux environment.

Introduction

Every well-organized workshop has a system. Tools are in specific drawers, materials are on designated shelves, and project plans are filed in a cabinet. This organization isn’t arbitrary; it allows anyone familiar with the system to find what they need, work efficiently, and avoid chaos. A Linux system’s filesystem is no different. The Filesystem Hierarchy Standard (FHS) is the blueprint for this organization, defining a standardized layout of directories and files. For an embedded systems developer, mastering the FHS is not an academic exercise—it is a fundamental skill for building robust, maintainable, and portable systems.

The FHS provides a crucial framework in the world of embedded Linux, where resources are often constrained and systems must be predictable. When you are debugging a device that won’t boot, configuring a peripheral, or deploying a new application, knowing precisely where to look for the kernel image, device drivers, configuration files, or system logs can be the difference between a quick fix and hours of frustration. The Raspberry Pi 5, our development platform, adheres to this standard, making it an excellent environment to explore these concepts. This chapter will move beyond simply listing directories; it will delve into the why behind the structure, exploring the role each part of the filesystem plays in the life of a running system. You will learn to see the filesystem not as a static container, but as a dynamic, living part of the operating system’s architecture.

Technical Background

The structure of the Linux filesystem is a legacy that stretches back to the earliest days of UNIX in the 1970s. In those formative years, there was no formal standard, leading to variations between different UNIX implementations. This divergence created significant challenges for software developers and system administrators who needed their programs and scripts to work across different systems. The Filesystem Hierarchy Standard (FHS) was born out of this necessity for interoperability. It was created to define a common layout that software could depend on, ensuring that a program could reliably find the libraries, configuration files, and executables it needed to run.

The FHS is maintained by the Linux Foundation and provides a set of requirements and guidelines for file and directory placement. Its core philosophy is to organize the filesystem based on the nature of the data. It distinguishes between shareable and unshareable files, and between static and variable files. Shareable files, like user applications, can be accessed across a network, while unshareable files, like device locks, are specific to a single host. Static files, such as binaries and libraries, do not change without administrator intervention, whereas variable files, like logs and spool directories, change continuously. This logical separation is the key to understanding the entire hierarchy.

%%{ init: { 'theme': 'base', 'themeVariables': { 'primaryColor': '#f8fafc', 'edgeLabelBackground':'#f8fafc', 'clusterBkg': '#f8fafc' } } }%%
graph TD
    Root["/"]:::primary

    subgraph Essential["Essential for Boot"]
        direction LR
        Bin["/bin<br/><i>User Binaries</i>"]:::process
        SBin["/sbin<br/><i>System Binaries</i>"]:::process
        Etc["/etc<br/><i>Configuration</i>"]:::process
        Dev["/dev<br/><i>Device Files</i>"]:::system
        Boot["/boot<br/><i>Bootloader Files</i>"]:::system
    end

    subgraph Runtime["Runtime & User Data"]
        direction LR
        Usr["/usr<br/><i>User Programs</i>"]:::process
        Var["/var<br/><i>Variable Data</i>"]:::process
        Home["/home<br/><i>User Directories</i>"]:::process
        Tmp["/tmp<br/><i>Temporary Files</i>"]:::check
    end

    subgraph Virtual["Kernel Virtual FS"]
        direction LR
        Proc["/proc<br/><i>Process Info</i>"]:::system
        Sys["/sys<br/><i>System Bus</i>"]:::system
    end

    Root --> Essential
    Root --> Runtime
    Root --> Virtual

    Usr --> UsrBin["/usr/bin"]:::process
    Usr --> UsrSBin["/usr/sbin"]:::process
    Usr --> UsrLib["/usr/lib"]:::process
    Usr --> UsrLocal["/usr/local"]:::process

    Var --> VarLog["/var/log"]:::process
    Var --> VarLock["/var/lock"]:::check
    Var --> VarSpool["/var/spool"]:::process

    classDef primary fill:#1e3a8a,stroke:#1e3a8a,stroke-width:2px,color:#ffffff
    classDef process fill:#0d9488,stroke:#0d9488,stroke-width:1px,color:#ffffff
    classDef system fill:#8b5cf6,stroke:#8b5cf6,stroke-width:1px,color:#ffffff
    classDef check fill:#ef4444,stroke:#ef4444,stroke-width:1px,color:#ffffff

The Root Directory: The Beginning of Everything

The entire filesystem structure begins at the root directory, denoted by a single forward slash (/). Every single file and directory on a Linux system is located under this root directory, regardless of which physical storage device it resides on. For an embedded system, the root filesystem contains the most critical components required to boot the system and bring it to a functional state where other filesystems can be mounted. Mounting is the process by which the operating system makes files and directories on a storage device (like an SD card or a USB drive) available for users to access via the computer’s file system. The root directory itself should remain clean, containing only the standard top-level directories and avoiding stray user files.

The directories directly under / are essential for basic system operation. They must contain everything needed to boot, restore, recover, or repair the system, even if the larger, secondary directories like /usr are unavailable.

Here are the FHS top level Directories:

Core Executables: /bin and /sbin

The /bin directory (short for binaries) contains essential command-line programs that are available to all users. These are the fundamental utilities required for system functionality, such as ls (list files), cp (copy), mv (move), and bash (the shell). The key characteristic of /bin is that it holds the executables needed for both single-user mode and general operation, ensuring that the system is usable even in a minimal recovery state.

Adjacent to /bin is the /sbin directory (short for system binaries). This directory holds executables that are reserved for system administration tasks. Commands like reboot, fdisk (disk partitioning), and ifconfig (network interface configuration) reside here. While any user can technically try to run these commands, they are typically only executable by the root user (the superuser) because they perform privileged operations that can fundamentally alter the system’s state. The separation between /bin and /sbin provides a clear distinction between general-purpose commands and powerful administrative tools.

System Configuration: /etc

If the filesystem is the skeleton of the system, the /etc directory (et cetera) is its nervous system. It contains all the host-specific, system-wide configuration files. Unlike the static binaries in /bin, the files in /etc are text-based and are meant to be edited by the administrator to configure the system’s behavior. You will not find any binary executables in /etc.

Examples of what you might find here include network configuration (/etc/network/interfaces), user account information (/etc/passwd, /etc/shadow), and startup scripts that launch services when the system boots (/etc/init.d/ or /etc/systemd/system/). For an embedded developer working on a Raspberry Pi 5, /etc is a frequent destination. Configuring Wi-Fi, setting up a static IP address, or customizing the hostname all involve modifying files within this directory. The contents of /etc are specific to a single machine, so you would never share this directory between different systems.

Tip: Before editing any configuration file in /etc, it is a best practice to create a backup copy. A simple command like sudo cp /etc/network/interfaces /etc/network/interfaces.bak can save you from a misconfiguration that renders the system inaccessible.

The Device Directory: /dev

The /dev directory is one of the most fascinating parts of a Linux system. It is not a traditional directory that stores files on a disk. Instead, it is a virtual filesystem, managed by the udev (userspace /dev) daemon, that provides an interface to the hardware devices connected to the system. In the UNIX philosophy, everything is a file, and /dev is the embodiment of this principle.

Each node in /dev represents a device. For example, your Raspberry Pi’s SD card might appear as /dev/mmcblk0, and its partitions as /dev/mmcblk0p1 and /dev/mmcblk0p2. Serial ports, crucial for embedded debugging, appear as /dev/ttyS0 or /dev/ttyAMA0. Interacting with these “files” is equivalent to interacting with the hardware itself. Reading from /dev/random provides a stream of random numbers generated by the kernel, while writing to /dev/null discards the data forever. For embedded work, you will frequently interact with /dev to access GPIO pins, I2C buses (/dev/i2c-*), and SPI devices (/dev/spidev*).

%%{ init: { 'theme': 'base', 'themeVariables': { 'primaryColor': '#f8fafc', 'edgeLabelBackground':'#f8fafc' } } }%%
sequenceDiagram
    participant HW as Hardware (e.g., USB Drive)
    participant Kernel as Linux Kernel
    participant UDEV as udev Daemon
    participant FS as /dev Filesystem

    HW->>+Kernel: Device Connected
    Kernel->>+UDEV: Netlink message<br>('uevent': ADD)
    UDEV->>UDEV: Parse uevent, check rules in /etc/udev/rules.d
    UDEV->>+FS: Create device node<br>(e.g., /dev/sda1)
    FS-->>-UDEV: Node created
    UDEV-->>-Kernel: Acknowledge
    Kernel-->>-HW: Device is ready

    Note over FS: The /dev/sda1 file<br>now represents the<br>USB drive partition.

Process and Kernel Information: /proc and /sys

Like /dev, the /proc directory (short for processes) is another virtual filesystem. It was originally created to provide real-time information about the running processes on the system. Each numbered directory within /proc corresponds to a Process ID (PID), containing files that expose details about that process’s memory usage, command-line arguments, and state.

However, /proc has evolved to become a window into the kernel itself. Files like /proc/cpuinfo provide details about the processor, and /proc/meminfo shows the system’s memory status. You can even change some kernel parameters on-the-fly by writing to certain files in /proc/sys.

The /sys directory is a newer virtual filesystem that was introduced to provide a more structured view of the system’s hardware and device drivers. While /proc is somewhat disorganized, /sys presents a clean hierarchy of devices, buses, and drivers as the kernel sees them. For example, you can trace the entire device tree, from the USB controller down to a specific connected device, by navigating the directories in /sys/bus and /sys/class.

Variable and Temporary Data: /var and /tmp

The /var directory (short for variable) is the designated place for files whose content is expected to grow and change during the normal operation of the system. This is where you will find system log files (/var/log), mail spools (/var/mail), and data for services like web servers (/var/www). Because the data here is dynamic, /var is often placed on its own partition on servers to prevent runaway log files from filling up the entire root filesystem and crashing the system. In an embedded context, managing the size of /var is critical, especially on devices with limited flash storage.

The /tmp directory is for temporary files. As the name implies, files and directories in /tmp are not guaranteed to persist across reboots. Many systems are configured to clear /tmp at startup. It is a scratchpad for applications that need to create short-lived files. A common mistake for novice developers is to store important data in /tmp, only to find it gone after a power cycle.

The User Filesystem: /usr

The /usr directory (often thought of as User System Resources) is one of the largest and most important directories on the system. Historically, /usr was where users’ home directories were kept. Today, however, /home serves that purpose. Instead, /usr is a secondary hierarchy for shareable, read-only user data. It contains the bulk of the system’s applications and files that are not needed for the initial boot process.

The structure of /usr mirrors the root directory. It has its own /usr/bin for non-essential user command binaries, /usr/sbin for non-essential system administration binaries, /usr/lib for program libraries, and /usr/include for C/C++ header files needed for compiling software.

A key subdirectory is /usr/local, which has its own bin, lib, and etc subdirectories. The /usr/local hierarchy is the designated place for software that is compiled and installed manually by the system administrator. This keeps it separate from the software that was installed by the system’s package manager (which places files in /usr/bin, etc.), preventing conflicts and making it easy to track locally installed packages.

%%{ init: { 'theme': 'base', 'themeVariables': { 'primaryColor': '#f8fafc', 'edgeLabelBackground':'#f8fafc' } } }%%
graph TD
    A[Start: Install New Software]:::primary --> B{Compiled from Source?};

    B -- Yes --> C{Is it a core system library<br>or part of a larger package?};
    B -- No --> D{Is it a self-contained<br>third-party application?};

    C -- Yes --> E[<b>Warning:</b> Rebuild system package<br>or use a custom distribution layer.<br><i>Avoid direct modification.</i>]:::warning;
    C -- No --> F[Install components into<br><b>/usr/local</b> hierarchy];

    F --> F1[Binary -> <b>/usr/local/bin</b>]:::process;
    F --> F2[Library -> <b>/usr/local/lib</b>]:::process;
    F --> F3[Config -> <b>/usr/local/etc</b>]:::process;
    F --> G[Success: Software is in system PATH]:::success;

    D -- Yes --> H[Install entire application<br>into its own directory under<br><b>/opt/<app_name></b>]:::process;
    D -- No --> I{Is it a single script or binary?};

    I -- Yes --> F
    I -- No --> J[Re-evaluate software type.<br>Does it fit FHS?]:::check

    H --> G;

    classDef primary fill:#1e3a8a,stroke:#1e3a8a,stroke-width:2px,color:#ffffff;
    classDef success fill:#10b981,stroke:#10b981,stroke-width:2px,color:#ffffff;
    classDef decision fill:#f59e0b,stroke:#f59e0b,stroke-width:1px,color:#ffffff;
    classDef process fill:#0d9488,stroke:#0d9488,stroke-width:1px,color:#ffffff;
    classDef check fill:#ef4444,stroke:#ef4444,stroke-width:1px,color:#ffffff;
    classDef warning fill:#eab308,stroke:#eab308,stroke-width:1px,color:#1f2937;

    class B,C,D,I decision;

User Home Directories: /home

The /home directory is where user-specific data is stored. For each user on the system (except root), a subdirectory is created within /home. For a user named pi, their home directory would be /home/pi. This directory contains the user’s personal files, application settings (often in hidden “dotfiles” like .bashrc), and documents. Separating user data in /home allows the rest of the filesystem to be upgraded or even reinstalled without affecting personal files. The root user’s home directory is an exception and is located at /root to ensure it is available even if /home is on a separate, unmounted partition.

Boot-Related Files: /boot

The /boot directory contains all the files needed for the system’s boot loader to start the operating system. This includes the Linux kernel itself (often named vmlinuz-*), an initial RAM disk image (initrd.img-*), and boot loader configuration files. On the Raspberry Pi, this is particularly important as it contains files like config.txt and cmdline.txt, which control low-level hardware settings and kernel boot parameters. The contents of /boot are static and critical; corrupting a file here can easily render the system unbootable.

Key Files in /boot:

Practical Examples

Theory is essential, but understanding comes from practice. In this section, we will use the Raspberry Pi 5 to explore the FHS in a hands-on manner. Ensure you have a Raspberry Pi 5 running a recent version of Raspberry Pi OS and are connected to it via SSH or have a keyboard and monitor attached.

Exploring the Filesystem with ls and tree

The most basic tool for looking at the filesystem is ls. Let’s start by listing the contents of the root directory.

# List the contents of the root directory in long format
ls -l /

You should see a list of the top-level directories we have just discussed.

total 80
lrwxrwxrwx   1 root root          7 Apr  9 15:06 bin -> usr/bin
drwxr-xr-x   4 root root       4096 Jul 21 01:54 boot
dr-xr-xr-x   2 root root       4096 Apr 15 21:50 cdrom
drwxr-xr-x  19 root root       4100 Jul 23 02:37 dev
drwxr-xr-x 139 root root      12288 Jul 21 01:54 etc
drwxr-xr-x   3 root root       4096 Jul 19 15:37 home
lrwxrwxrwx   1 root root          7 Apr  9 15:06 lib -> usr/lib
drwx------   2 root root      16384 Jul 19 15:27 lost+found
drwxr-xr-x   2 root root       4096 Apr 15 20:19 media
drwxr-xr-x   2 root root       4096 Apr 15 20:19 mnt
drwxr-xr-x   2 root root       4096 Apr 15 20:19 opt
dr-xr-xr-x 263 root root          0 Jul 23 02:37 proc
drwx------   5 root root       4096 Jul 19 15:37 root
drwxr-xr-x  36 root root        880 Jul 23 02:37 run
lrwxrwxrwx   1 root root          8 Apr  9 15:06 sbin -> usr/sbin
drwxr-xr-x  13 root root       4096 Apr 15 20:48 snap
drwxr-xr-x   2 root root       4096 Apr 15 20:19 srv
-rw-------   1 root root 4088397824 Jul 19 15:30 swap.img
dr-xr-xr-x  13 root root          0 Jul 23 02:37 sys
drwxrwxrwt  19 root root        460 Jul 23 02:38 tmp
drwxr-xr-x  11 root root       4096 Apr 15 20:19 usr
drwxr-xr-x  14 root root       4096 Jul 19 15:37 var

Notice that on many modern systems, /bin, /sbin, and /lib are symbolic links to their counterparts in /usr. This is a change to unify the location of binaries and libraries, simplifying the filesystem structure.

To get a better visual representation, we can use the tree command. You may need to install it first (sudo apt update && sudo apt install tree).

# Display the root directory as a tree, but only one level deep
tree -d -L 1 /

Investigating System Configuration in /etc

Let’s look at a practical configuration file. The file /etc/hostname stores the name of your device on the network.

# View the contents of the hostname file
cat /etc/hostname

This will likely print raspberrypi. If you wanted to change your device’s name, you would edit this file (with sudo privileges) and reboot.

Now, let’s look at the systemd service files. These define how services are started, stopped, and managed.

# List the service files that came with the system
ls -l /lib/systemd/system/*.service

# List user-created or modified service files
ls -l /etc/systemd/system/*.service

This separation is a perfect example of the FHS in action. Base package files go in /lib (or /usr/lib), while local administrator modifications go in /etc. This prevents system updates from overwriting your custom configurations.

Peeking into Device Files in /dev

The /dev directory is best explored by looking for specific devices. Let’s check for the primary serial port on the Raspberry Pi, which is often used for a debug console.

# List all serial devices
ls -l /dev/serial*

You will likely see /dev/serial0 and /dev/serial1, which are symbolic links to the actual hardware device files (ttyS0 or ttyAMA0).

Warning: When connecting a serial adapter, ensure the voltage levels match. The Raspberry Pi 5 uses 3.3V logic. Connecting a 5V adapter can permanently damage the GPIO pins.

Installing a Custom Application in /opt and /usr/local

Imagine you have downloaded a custom application that is not available in the Raspberry Pi OS repositories. The FHS provides two primary locations for this: /opt and /usr/local.

• /opt (for optional) is intended for large, self-contained applications from a third-party vendor. The application would be installed in a directory like /opt/myapp, which would contain all its binaries, libraries, and configuration files.
• /usr/local is for software you compile from source. The compiled binary would go in /usr/local/bin, its libraries in /usr/local/lib, and so on.

Let’s simulate installing a simple application into /usr/local.

Step 1: Create a simple C program.

Create a file named hello.c.

// hello.c
// A simple "hello world" program to demonstrate installation.

#include <stdio.h>

int main() {
    printf("Hello from /usr/local/bin!\n");
    return 0;
}

Step 2: Compile the program.

We will use gcc, the GNU C Compiler, which is a standard part of most Linux development environments.

# Compile hello.c and create an executable named 'hello'
gcc hello.c -o hello

Step 3: Install the program into /usr/local/bin.

This requires superuser privileges. The install command is preferred over cp because it can set permissions correctly.

# Install the 'hello' executable into the correct FHS location
sudo install -m 755 hello /usr/local/bin/hello

The -m 755 sets the permissions to be readable and executable by everyone, but only writable by the owner (root).

Step 4: Run the program.

Because /usr/local/bin is in the system’s default PATH, you can now run this program from anywhere.

# Execute the newly installed program
hello

Expected Output:

Hello from /usr/local/bin!

This exercise demonstrates the correct, FHS-compliant way to add software to a system, keeping it cleanly separated from the OS-managed files.

Common Mistakes & Troubleshooting

Adhering to the FHS prevents many common issues, but mistakes still happen, especially for those new to Linux. Here are some frequent pitfalls and how to resolve them.

Exercises

1. Filesystem Scavenger Hunt:
   • Objective: Locate several key system files to reinforce your knowledge of the FHS.
   • Steps:
     1. Find the exact path to the Linux kernel image currently running on your Raspberry Pi. (Hint: Look in /boot and use the uname -r command).
     2. Locate the configuration file that defines the users and groups on your system.
     3. Find the log file that records authentication attempts (e.g., successful and failed ssh logins).
     4. Determine which device file in /dev represents your primary display. (Hint: Look for fb0, for framebuffer).
   • Verification: Use the file command on each path you find to confirm the file type. For example, file /boot/vmlinuz-$(uname -r) should identify it as a Linux kernel image.
2. Scripting a Filesystem Audit:
   • Objective: Write a shell script that analyzes the executables in /bin, /sbin, /usr/bin, and /usr/sbin and reports the count for each.
   • Steps:
     1. Create a new shell script file named audit_bins.sh.
     2. In the script, use a combination of ls and wc -l (word count, lines) to count the number of files in each of the four binary directories.
     3. Print a formatted summary, like “/bin contains 120 executables.”
   • Verification: Run the script (bash audit_bins.sh). Manually run ls /bin | wc -l and compare the output to your script’s report to ensure it is accurate.
3. Creating a Compliant Service:
   • Objective: Create a simple background service and install it according to FHS and systemd best practices.
   • Steps:
     1. Write a simple Python script named heartbeat.py that prints a message and the current time to the system log every 10 seconds. Use the logging and time libraries.
     2. Place this script in /usr/local/bin/.
     3. Create a systemd service file named heartbeat.service in /etc/systemd/system/. This file should specify the ExecStart path as /usr/local/bin/heartbeat.py.
     4. Use sudo systemctl enable heartbeat.service and sudo systemctl start heartbeat.service to start your service.
   • Verification: Use systemctl status heartbeat.service to check that the service is running. Use journalctl -u heartbeat.service -f to view the log output from your Python script in real-time.

Summary

• The Filesystem Hierarchy Standard (FHS) provides a predictable and portable directory structure for Linux systems, which is critical for embedded development.
• The filesystem starts at the root (/) directory, which contains everything needed for a minimal system boot.
• /bin and /sbin contain essential binaries for all users and system administrators, respectively.
• /etc is the central location for all system-wide, host-specific configuration files.
• /dev, /proc, and /sys are virtual filesystems that provide interfaces to hardware devices and kernel information.
• /var stores variable data like logs, while /tmp is for non-persistent temporary files.
• /usr holds the majority of user applications and libraries, with /usr/local reserved for manually installed software.
• /home contains personal directories for each user, and /boot holds the kernel and other files needed for the boot process.
• Adhering to the FHS prevents conflicts, simplifies administration, and ensures system stability.

Further Reading

1. Filesystem Hierarchy Standard (FHS) Main Page – The official specification from the Linux Foundation.
   • https://refspecs.linuxfoundation.org/fhs.shtml
2. hier(7) Linux Manual Page – The canonical man page describing the filesystem hierarchy.
   • https://man7.org/linux/man-pages/man7/hier.7.html
3. Raspberry Pi Documentation – config.txt – Official documentation for the critical boot configuration file on the Raspberry Pi.
   • https://www.raspberrypi.com/documentation/computers/config_txt.html
4. LWN.net – “The /usr Merge” – An in-depth article explaining the rationale behind merging /bin into /usr/bin.
   • https://lwn.net/Articles/486492/
5. systemd.unit(5) Manual Page – Essential reading for understanding how to write service files that integrate with the system.
   • https://www.freedesktop.org/software/systemd/man/systemd.unit.html
6. “How to Make a Custom Linux Distro” by Robert Nelson – A practical guide that often touches on filesystem layout for custom embedded systems.
   • https://www.digikey.com/en/maker/projects/how-to-make-a-custom-linux-distro/d52735465a494d47a46083d00227914f
7. The Linux Programming Interface by Michael Kerrisk – While a comprehensive book on Linux APIs, its initial chapters provide excellent context on system layout.