Chapter 246: Advanced Partition Management in ESP32

Chapter Objectives

By the end of this chapter, you will be able to:

• Understand the role and structure of the partition table in the ESP-IDF.
• Read and interpret the default partition table layouts.
• Design and create custom partition tables for specific application needs.
• Configure an ESP-IDF project to use a custom partition table.
• Programmatically access, read from, and write to custom data partitions.
• Identify and troubleshoot common partition-related errors.
• Apply these concepts to organize data and firmware on different ESP32 variants.

Introduction

In previous chapters, we have built and flashed numerous applications without paying much attention to how our code and data are physically organized on the ESP32‘s flash memory. The ESP-IDF build system intelligently handled this for us using a default configuration. However, for real-world products, the default layout is often insufficient. You might need a larger section for application data, space for factory configuration, or multiple Over-The-Air (OTA) update slots.

This is where partition management becomes critical. By defining a custom partition table, you take direct control over the flash memory layout, tailoring it precisely to your project’s requirements. This chapter delves into the theory behind partition tables and provides practical, step-by-step guidance on creating and managing them, empowering you to build more complex and robust embedded systems.

Theory

The flash memory of an ESP32 is like a large, empty book. Before you can write in it, you need a table of contents that defines where each chapter begins and ends. In the ESP-IDF, this “table of contents” is called the partition table. It is a small but crucial piece of data, stored at a fixed offset (default 0x8000) in the flash, that dictates the layout of the entire memory space.

The bootloader reads this table upon startup to locate the application it needs to execute. The ESP-IDF framework also uses it to manage other regions, such as Non-Volatile Storage (NVS), file systems (like SPIFFS or FAT), and OTA data.

%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': '"Open Sans", sans-serif'}}}%%
sequenceDiagram
    actor User as Power On / Reset
    participant BL as Bootloader (in ROM/Flash)
    participant PT as Partition Table (at 0x8000)
    participant OTA as otadata Partition
    participant App0 as ota_0 Partition
    participant App1 as ota_1 Partition

    User->>+BL: Applies power or resets device
    BL->>+PT: Reads partition table from flash
    PT-->>-BL: Returns partition layout
    
    BL->>BL: Finds 'otadata' partition
    BL->>+OTA: Reads otadata content
    OTA-->>-BL: Info on which app to boot
    
    BL->>+App0: Verifies integrity of bootable app (e.g., ota_0)
    App0-->>-BL: CRC/Hash OK
    
    BL->>App0: Jumps to application entry point
    deactivate BL
    
    App0-->>User: Application code starts running...

1. Partition Table Structure

A partition table is simply a list of entries. Each entry defines a single partition and has the following attributes:

• Name: A unique, human-readable ASCII string (up to 16 characters) to identify the partition (e.g., app0, nvs, spiffs).
• Type: A number indicating the partition’s general purpose. The two main types are:
  • app (0x00): A partition that contains executable application code.
  • data (0x01): A partition that contains non-executable data, such as a filesystem, NVS library data, or custom user data.
• SubType: A number that specifies the particular kind of app or data. For example, an app partition can have a subtype of factory (0x00), ota_0 (0x10), ota_1 (0x11), etc. A data partition can have subtypes like ota (0x00), nvs (0x02), spiffs (0x82), etc.
• Offset: The starting address of the partition in the flash memory, relative to the beginning of the flash. This must be aligned to a 4 KiB (0x1000) boundary. If left blank, the build system will automatically place it after the previous partition.
• Size: The total size of the partition. You can use standard suffixes like K for kilobytes and M for megabytes (e.g., 1M, 1024K).
• Flags: An optional field to specify properties like encryption. We will cover this in Chapter 247: Partition Encryption Techniques.

These entries are defined in a simple Comma-Separated Values (.csv) file.

2. Default Partition Tables

ESP-IDF provides several default partition tables. A common one for a 4MB flash without OTA enabled is single_factory_app.csv:

# Name,   Type, SubType, Offset,  Size,   Flags
nvs,      data, nvs,     ,        24K,
phy_init, data, phy,     ,        4K,
factory,  app,  factory, ,        1M,

Let’s break this down:

• nvs: A 24KB data partition for the NVS library to store key-value pairs.
• phy_init: A 4KB data partition to store PHY (physical layer) calibration data.
• factory: A 1MB app partition to hold the main application firmware. Note the subtype factory, indicating it’s the default, non-OTA application.

Another common table is for OTA updates, partitions_two_ota.csv:

# Name,   Type, SubType, Offset,   Size,   Flags
nvs,      data, nvs,     ,         24K,
otadata,  data, ota,     ,         8K,
ota_0,    app,  ota_0,   ,         1M,
ota_1,    app,  ota_1,   ,         1M,

Here, factory is replaced by two OTA app slots (ota_0, ota_1) and an otadata partition. The otadata partition is used by the bootloader to determine which of the two OTA app partitions to boot from.

3. Why Create a Custom Partition Table?

You’ll need a custom partition table when:

• Your application binary is larger than the default partition size (e.g., > 1MB).
• You need to store large amounts of data, like sensor logs, audio files, images, or machine learning models.
• You want to integrate a filesystem like SPIFFS, FAT, or LittleFS.
• You need to change the NVS partition size.
• You are designing a complex OTA scheme with more than two app slots or dedicated factory reset partitions.

Practical Examples

Let’s create a custom partition table for a project that requires a dedicated 512KB partition for storing binary data, such as images or firmware patches.

%%{init: {'theme': 'base', 'themeVariables': { 'fontFamily': '"Open Sans", sans-serif'}}}%%
graph TD
    subgraph "Phase 1: Design & Configuration"
        A(Start: Define Requirements) --> B{"Need custom<br>flash layout?"};
        B -- Yes --> C["1- Create <b>partitions_custom.csv</b><br>Define names, types, and sizes"];
        C --> D["2- Configure Project<br>Run menuconfig"];
        D --> E["Navigate to Partition Table<br>Set to Custom and point<br>to your .csv file"];
    end

    subgraph "Phase 2: Build & Flash"
        E --> F["3- Build Project<br>ESP-IDF reads .csv and<br>generates partition-table.bin"];
        F --> G["4- Flash ESP32<br>Uploads bootloader, app,<br>and new partition table"];
    end

    subgraph "Phase 3: Application Code"
        G --> H[5- Access Partition in Code];
        H --> I["<b>esp_partition_find_first()</b><br>Locate partition by<br>Type and SubType"];
        I --> J{Partition Found?};
        J -- No --> K["Error: Partition not found!<br>Check menuconfig & .csv file"];
        J -- Yes --> L["<b>esp_partition_write()</b><br><b>esp_partition_read()</b><br>Interact with storage"];
        L --> M(End: Verification Successful);
    end

    %% Styling
    classDef start-node fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef process-node fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decision-node fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef check-node fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef endo-node fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;

    class A,H start-node;
    class C,D,E,F,G,I,L process-node;
    class B,J decision-node;
    class K check-node;
    class M endo-node;

Example: Adding a Custom ‘storage’ Partition

Our goal is to create a layout for a 4MB flash chip with two OTA slots and our new custom data partition.

1. Create the Custom Partition .csv File

1. In your VS Code project explorer, create a new file at the root of your project named partitions_custom.csv.
2. Add the following content to the file. We will place our new storage partition after the OTA slots.
   # ESP-IDF Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, , 28K,
otadata, data, ota, , 8K,
ota_0, app, ota_0, , 1856K,
ota_1, app, ota_1, , 1856K,
storage, data, 0x99, , 256K,
Analysis of the custom table:
   • nvs: We slightly increased the NVS size to 28K for more configuration storage.
   • otadata: Kept at 8K, which is standard.
   • ota_0 / ota_1: We’ve allocated 1856K (1.8MB) for each OTA slot to accommodate larger applications.
   • storage: This is our custom partition.* Name: storage.* Type: data, since it holds non-executable data.* SubType: 0x99. We’ve chosen a custom subtype value. For custom data partitions, any value from 0x80 to 0xFE is available for application-specific use.* Offset: We leave it blank to let the build system place it automatically.* Size: 256K.

2. Configure the Project to Use the Custom Table

Now, we must instruct the build system to use our new partitions_custom.csv file instead of a default one.

1. Open the Project Configuration menu by pressing Ctrl+P (or Cmd+P on Mac) and typing >ESP-IDF: SDK Configuration editor (menuconfig).
2. Navigate to Partition Table —>.
3. Change Partition Table from Single factory app, no OTA to Custom partition table CSV.
4. A new field will appear: (partitions_custom.csv) Custom partition CSV file. Ensure this matches the name of the file you created.
5. Save the configuration and exit.

3. Write Code to Access the Custom Partition

Now, let’s write an application that finds our storage partition, writes some data to it, and reads it back to verify.

Modify your main/main.c file with the following code:

#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_log.h"
#include "esp_partition.h"

static const char *TAG = "PARTITION_EXAMPLE";

void app_main(void)
{
    ESP_LOGI(TAG, "Starting Advanced Partition Example...");

    // 1. Find the custom partition
    // We are searching for a partition with type 'data' and subtype '0x99'.
    // The label "storage" is not used for searching, but for identifying partitions in the CSV.
    const esp_partition_t *storage_partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, 0x99, "storage");

    // It's crucial to check if the partition was found.
    if (!storage_partition) {
        ESP_LOGE(TAG, "Failed to find 'storage' partition. Type=data, SubType=0x99");
        // Halt execution if the partition is essential for the application
        while(1) { vTaskDelay(pdMS_TO_TICKS(1000)); }
    } else {
        ESP_LOGI(TAG, "Found 'storage' partition: size=0x%x, offset=0x%x", storage_partition->size, storage_partition->address);
    }

    // 2. Prepare data to write
    const char *message = "Hello from the custom storage partition!";
    char read_buffer[64] = {0}; // Buffer to read data back into

    // NOTE: Writing to flash requires erasing a sector first. The esp_partition_write
    // function handles this transparently, but it's important to know that flash
    // memory has an erase-write cycle limitation. Avoid frequent writes to the same location.

    // 3. Write data to the partition
    ESP_LOGI(TAG, "Writing data to partition at offset 0...");
    esp_err_t err = esp_partition_write(storage_partition, 0, message, strlen(message) + 1); // +1 to include null terminator
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to write data to partition: %s", esp_err_to_name(err));
    } else {
        ESP_LOGI(TAG, "Successfully wrote data.");
    }

    // 4. Read data back from the partition
    ESP_LOGI(TAG, "Reading data back from partition at offset 0...");
    err = esp_partition_read(storage_partition, 0, read_buffer, sizeof(read_buffer));
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to read data from partition: %s", esp_err_to_name(err));
    } else {
        ESP_LOGI(TAG, "Successfully read data: '%s'", read_buffer);
    }

    // 5. Verify data
    if (strcmp(message, read_buffer) == 0) {
        ESP_LOGI(TAG, "Data verification successful!");
    } else {
        ESP_LOGE(TAG, "Data verification FAILED!");
    }

    ESP_LOGI(TAG, "Example finished.");
}

Warning: Raw partition writes are a low-level operation. The esp_partition_write function does not perform wear-leveling. For applications requiring frequent data updates, it is highly recommended to use a higher-level library like NVS or a filesystem like SPIFFS or LittleFS, which manage wear-leveling for you.

4. Build, Flash, and Monitor

1. Build: Click the Build button (cylinder icon) in the VS Code status bar. The build system will now use your partitions_custom.csv to generate the final partition-table.bin.
2. Flash: Connect your ESP32 board, select the correct COM port, and click the Flash button (lightning bolt icon).
3. Monitor: Click the Monitor button (plug icon) to view the serial output.

You should see output similar to this:

...
I (314) PARTITION_EXAMPLE: Starting Advanced Partition Example...
I (324) PARTITION_EXAMPLE: Found 'storage' partition: size=0x40000, offset=0x3b0000
I (324) PARTITION_EXAMPLE: Writing data to partition at offset 0...
I (344) PARTITION_EXAMPLE: Successfully wrote data.
I (344) PARTITION_EXAMPLE: Reading data back from partition at offset 0...
I (354) PARTITION_EXAMPLE: Successfully read data: 'Hello from the custom storage partition!'
I (364) PARTITION_EXAMPLE: Data verification successful!
I (374) PARTITION_EXAMPLE: Example finished.

Variant Notes

Partition management concepts are universal across the ESP32 family (ESP32, S2, S3, C3, C6, H2). The primary difference is the total amount of integrated flash memory, which can range from 2MB to 16MB or more if external flash is used.

• ESP32-C3/H2/C6 (RISC-V): These variants often come in smaller flash configurations (e.g., 4MB). You must be mindful of the total size when defining your partitions. Their bootloaders and radio firmware may have slightly different sizes, which can affect the starting offset of the first partition, but the parttool.py and build system handle this automatically.
• ESP32-S2/S3: These variants can support larger external flash and PSRAM. Custom partition tables are essential when working with large assets for displays (using the LCD peripheral) or AI models (for the S3’s vector instructions). The process of creating and using the table remains identical.

The key takeaway is to always design your partition table with the target device’s physical flash size as your upper limit. The build system will error out if your defined partitions exceed this limit.

Common Mistakes & Troubleshooting Tips

Exercises

1. Expand NVS: Modify the partitions_custom.csv file to give the nvs partition a size of 40K. Re-flash the device and use the nvs_get_stats() function to programmatically verify that the total available space in NVS has increased.
2. SPIFFS Filesystem Partition: Create a new partition table named partitions_spiffs.csv. This table should be for an 8MB flash chip and include: two 2MB OTA slots, a 32KB NVS partition, the standard 8KB otadata partition, and a large partition for a SPIFFS filesystem that uses the remaining space. (Hint: Use SubType spiffs and leave the Size blank for the SPIFFS partition to automatically fill the rest of the flash).
3. Multi-Data Partition Project: Create a partition layout with two distinct custom data partitions: logs (64K, subtype 0x90) and certs (16K, subtype 0x91). Write an application that writes the string “System rebooted” to the logs partition and a mock certificate string “—–BEGIN CERT—–…” to the certs partition, then reads both back to the serial monitor.

Summary

• The partition table defines the memory map of the ESP32’s flash, telling the system where applications and data are located.
• It is defined in a simple .csv file with fields for Name, Type, SubType, Offset, and Size.
• You can switch from a default to a custom partition table via the project’s menuconfig.
• Custom partitions are necessary for storing large assets, adding filesystems, or creating complex OTA schemes.
• The esp_partition API provides functions like esp_partition_find_first, esp_partition_read, and esp_partition_write to interact with partitions programmatically.
• Always be mindful of your target device’s total flash size and avoid hardcoding offsets unless absolutely necessary.
• Using a higher-level abstraction like NVS or a filesystem is recommended over raw partition writes for data that changes frequently.

Further Reading

• ESP-IDF Partition Tables Documentation: https://docs.espressif.com/projects/esp-idf/en/v5.2.1/api-reference/system/partition.html
• Partition Tool (parttool.py): For command-line inspection and modification of partitions on a flashed device. https://docs.espressif.com/projects/esp-idf/en/v5.2.1/api-reference/system/parttool.html
• SPIFFS Filesystem Support: https://docs.espressif.com/projects/esp-idf/en/v5.2.1/api-reference/storage/spiffs.html