Chapter 18: Debugging FreeRTOS Applications

Chapter Objectives

• Understand the unique challenges of debugging concurrent, real-time applications.
• Learn about different debugging techniques: logging, assertions, hardware debugging, and post-mortem analysis.
• Configure and use the VS Code debugger with ESP-IDF and a JTAG probe (like ESP-PROG).
• Set breakpoints, step through code, inspect variables, and view call stacks in a multi-tasking environment.
• Enable and utilize FreeRTOS debugging features: stack overflow checking, runtime statistics.
• Use GDB commands or VS Code extensions to inspect FreeRTOS task states.
• Understand the basics of core dumps for analyzing crashes after they occur.
• Identify common pitfalls when debugging FreeRTOS applications on the ESP32.

Introduction

Debugging is an inevitable part of software development. While debugging simple, single-threaded programs can be straightforward, debugging applications built on a Real-Time Operating System (RTOS) like FreeRTOS introduces significant new challenges. Issues related to task interaction, resource sharing, timing dependencies, and concurrency can lead to bugs that are difficult to reproduce and diagnose.

Problems like tasks failing to run, deadlocks where tasks wait indefinitely for each other, race conditions causing data corruption, or unexplained system crashes become more common. Traditional debugging techniques, like simple print statements, might be insufficient or even alter the system’s timing behavior, masking the very bugs you are trying to find.

Fortunately, ESP-IDF and FreeRTOS provide a range of tools and techniques specifically designed to tackle these complexities. From sophisticated hardware debuggers that let you halt the CPU and inspect its state, to RTOS-aware tools that show task statuses, and post-mortem analysis using core dumps, this chapter equips you with the knowledge and practical skills needed to effectively debug your FreeRTOS-based ESP32 applications.

Theory

Common Debugging Challenges in RTOS Applications

Debugging concurrent systems requires a different mindset than debugging sequential programs. You need to consider interactions between multiple threads of execution (tasks) and the RTOS scheduler itself. Common challenges include:

Debugging Tools and Techniques

A combination of tools and techniques is usually required to effectively debug RTOS applications.

1. Logging (ESP_LOGx)

• Description: Inserting print statements (ESP_LOGI, ESP_LOGD, etc.) to trace execution flow and variable values.
• Pros: Simple to implement, non-interactive (doesn’t halt execution).
• Cons: Can generate large amounts of data, significantly impacts timing (especially if logging to slower interfaces like UART), may hide timing-dependent bugs, requires code modification and recompilation.
• Usage: Useful for high-level tracing and understanding general program flow, but use sparingly in performance-critical sections or ISRs.

2. Assertions (assert())

• Description: Checks for conditions that should always be true at a certain point in the code. If the condition is false, the program halts (usually by entering an infinite loop or triggering a panic), indicating a critical error. The standard C assert() macro is available. FreeRTOS also provides configASSERT().
• Pros: Catches unexpected states early, clearly indicates programming errors or violated assumptions. Can often be disabled in release builds (NDEBUG macro) to avoid performance overhead.
• Cons: Only checks specific points; doesn’t help diagnose how the invalid state was reached. Halts the system.
• Usage: Excellent for validating function preconditions, postconditions, invariants, and checking return values of critical functions (e.g., ensuring malloc didn’t return NULL if that case isn’t handled).

3. Hardware Debugger (JTAG/SWD)

• Description: Uses a hardware probe (like ESP-PROG, J-Link, FTDI-based adapters) connected to the ESP32’s JTAG or SWD pins. This allows an external tool (OpenOCD) running on the host PC to control the CPU directly. The GNU Debugger (GDB) communicates with OpenOCD to provide debugging capabilities.
• Pros:
  • Full Control: Halt execution, set breakpoints (hardware and software), step through code (line-by-line, instruction-by-instruction).
  • Inspection: Examine and modify memory, CPU registers, and variables without modifying application code.
  • Non-Intrusive (Mostly): Doesn’t rely on application code for its core functionality (though breakpoints inherently alter timing when hit).
  • Works on Crashes: Can often connect to a halted or crashed CPU to inspect its state (if the crash didn’t corrupt the debug interface).
• Cons: Requires dedicated hardware probe, initial setup can be complex, halts the system when breakpoints are hit (can mask timing bugs), may require specific GPIO pins for JTAG/SWD connection.
• Usage: The most powerful tool for interactive debugging, essential for diagnosing crashes and complex logic errors.

4. VS Code Debugger Integration

• Description: The Espressif IDF extension for VS Code provides a user-friendly graphical interface on top of OpenOCD and GDB.
• Pros: Integrates debugging directly into the IDE, provides visual tools for breakpoints, variable inspection, call stack viewing, memory viewing, and even RTOS task awareness. Simplifies the GDB command-line interface.
• Cons: Relies on the underlying JTAG/OpenOCD/GDB setup.
• Usage: The standard way to perform interactive hardware debugging for ESP-IDF projects.

5. FreeRTOS-Specific Debugging Aids

FreeRTOS includes several configuration options (set via menuconfig) to aid debugging:

• Stack Overflow Checking (CONFIG_FREERTOS_CHECK_STACKOVERFLOW):
  • Enables mechanisms (configurable methods) for detecting when a task exceeds its allocated stack space.
  • When an overflow is detected, a hook function (vApplicationStackOverflowHook) is called, allowing you to log the error or take specific action.
  • Essential for diagnosing crashes caused by stack corruption. Set via menuconfig -> Component config -> FreeRTOS -> Check for stack overflow. Choose Method 1 or 2.

graph TD
    A[Task Execution] --> B{Stack Usage Exceeds Limit?};
    B -- No --> A;
    B -- Yes (Overflow Occurs!) --> C["Stack Overflow Detected by FreeRTOS<br>(e.g., on context switch - Method 2)"];
    C --> D["Call vApplicationStackOverflowHook(taskHandle, taskName)"];
    
    subgraph "vApplicationStackOverflowHookImplementation"
        D --> E{"Log Error Information<br>(Task Name, Handle)"};
        E --> F{Take Action:};
        F --> G1["Option 1: Halt System<br>(e.g., configASSERT(pdFAIL), infinite loop)"];
        F --> G2[Option 2: Trigger System Reset];
        F --> G3["Option 3: Persistent Logging (if possible)"];
    end
    
    G1 --> H[System Halted / Debugger Attaches];
    G2 --> I[System Reboots];
    G3 --> J[Error Logged for Later Analysis];

    %% Styling
    classDef startNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef errorNode fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef hookNode fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;

    class A processNode;
    class B decisionNode;
    class C errorNode;
    class D hookNode;
    class E,F processNode;
    class G1,G2,G3 decisionNode;
    class H,I,J errorNode;

• Runtime Statistics (CONFIG_FREERTOS_GENERATE_RUN_TIME_STATS):
  • Collects information about how much CPU time each task consumes.
  • Requires a high-frequency timer for accurate time measurement (ESP-IDF provides this automatically when enabled).
  • Functions like vTaskGetRunTimeStats() can be used to retrieve and print a formatted table showing the absolute time and percentage of CPU time used by each task.
  • Useful for identifying CPU-hungry tasks, performance bottlenecks, and potential task starvation. Set via menuconfig -> Component config -> FreeRTOS -> Enable FreeRTOS runtime stats collection.
• Task List & State Inspection:
  • Hardware debuggers (GDB) can inspect the internal data structures of FreeRTOS to determine the state of each task (Running, Ready, Blocked, Suspended), what object a task is blocked on (e.g., which queue or semaphore), and the call stack of each task.
  • VS Code extensions often provide graphical views for this (e.g., the “FreeRTOS” view in the Debug panel). GDB commands like info threads list tasks, and more advanced GDB scripts or extensions can provide detailed state information.
• Trace Hooks and Formatting Functions (CONFIG_FREERTOS_USE_TRACE_FACILITY, CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS):
  • configUSE_TRACE_FACILITY: Enables additional trace features and data structures within FreeRTOS, used by some debugging tools and state viewers. Also required for vTaskList() and vTaskGetRunTimeStats().
  • configUSE_STATS_FORMATTING_FUNCTIONS: Includes the helper functions (vTaskList, vTaskGetRunTimeStats) in the build.
  • Set via menuconfig -> Component config -> FreeRTOS. Both are usually enabled automatically when runtime stats are enabled.
• SystemView Integration (CONFIG_FREERTOS_USE_SYSTEM_VIEW_TRACE):
  • SEGGER SystemView is an advanced real-time recording and visualization tool. ESP-IDF provides integration hooks.
  • Allows capturing detailed traces of RTOS events (task switches, API calls, interrupts) with very low overhead.
  • Requires separate SystemView host software. Excellent for analyzing complex timing interactions and performance issues. Set via menuconfig -> Component config -> FreeRTOS -> Enable SystemView tracing.
• Assertions (configASSERT):
  • As mentioned before, FreeRTOS uses configASSERT(x) internally to check for critical errors (e.g., calling an ISR function from a task, invalid parameters).
  • When an assertion fails, it calls vAssertCalled(), which typically enters a panic state, halting execution and printing information about the failed assertion (file and line number).

6. Core Dumps

• Description: When a critical error or crash occurs (like an unhandled exception or assertion failure), ESP-IDF can be configured to save the state of the CPU (registers) and optionally snapshots of task stacks and memory to a persistent storage area (like Flash or UART output).
• Pros: Allows post-mortem analysis – debugging after the crash has happened, without needing a debugger attached at the exact moment of failure. Essential for diagnosing infrequent or field-reported crashes.
• Cons: Requires configuration, consumes storage space, analysis requires specific tools (espcoredump.py). Doesn’t provide interactive debugging.
• Usage: Configure via menuconfig -> Component config -> Core dump. Use the espcoredump.py utility with the application’s ELF file and the retrieved core dump data to reconstruct the state at the time of the crash, including backtraces for each task.

graph TD
    A["Critical Error Occurs in ESP32 Application<br>(e.g., Unhandled Exception, Panic, Assertion)"] --> B{"Core Dump Feature Enabled?"};
    B -- Yes --> C["System Halts or Enters Panic Handler"];
    C --> D["Core Dump Module Captures CPU State<br>(Registers, Task Stacks, Memory Snapshots)"];
    D --> E{"Storage Method Configured?"};
    E -- UART --> F["Core Dump Data Output to UART<br>(e.g., Base64 encoded)"];
    E -- Flash --> G["Core Dump Data Written to Dedicated Flash Partition"];
    F --> H[User Captures UART Output];
    G --> I["User Retrieves Dump from Flash<br>(e.g., via idf.py read_core_dump)"];
    
    subgraph "Post-Mortem Analysis on Host PC"
        H --> J["Core Dump Data File"];
        I --> J;
        K["Application ELF File<br> project/build/app_name.elf"] --> L{"Use espcoredump.py Utility"};
        J --> L;
        L --> M["Reconstruct Call Stacks for Tasks"];
        M --> N["View Register Values"];
        N --> O["Inspect Memory Contents (if included)"];
        O --> P["Identify Crash Location and Cause"];
    end

    B -- No --> Z["System Crashes or Reboots without Core Dump"];

    %% Styling
    classDef startNode fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#991B1B;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef storageNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:1px,color:#5B21B6;
    classDef analysisNode fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46;
    classDef errorNode fill:#FECACA,stroke:#DC2626,stroke-width:1px,color:#991B1B;

    class A startNode;
    class B,E decisionNode;
    class C,D,H,I processNode;
    class F,G storageNode;
    class J,K processNode; 
    class L,M,N,O,P analysisNode;
    class Z errorNode;

Summary of Tools and Techniques

Practical Examples

Example 1: Using the VS Code Debugger (with JTAG/ESP-PROG)

This example assumes you have a compatible JTAG probe (like ESP-PROG) connected to your ESP32 and configured in VS Code. The exact setup steps depend on your probe and OS; refer to the official ESP-IDF documentation for details.

Hardware Setup:

1. Connect your JTAG probe (e.g., ESP-PROG) to the ESP32’s JTAG pins (typically GPIOs 12-15 on ESP32, but check your board/variant documentation).
2. Connect the probe to your host PC via USB.

VS Code Setup (One-time or per-project):

1. Ensure the Espressif IDF extension is installed in VS Code.
2. Configure the project for debugging:
   • Open the Command Palette (Ctrl+Shift+P).
   • Run ESP-IDF: Device Configuration.
   • Select your target ESP32 variant.
   • Select your JTAG probe (e.g., “ESP-PROG”).
   • The extension may automatically create or update the necessary launch.json and settings.json files in the .vscode directory of your project. If not, you might need to create a debug configuration manually (refer to ESP-IDF docs). A typical launch.json might look like this (paths may vary):

// .vscode/launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch",
            "type": "espidf",
            "request": "launch",
            "debugPort": 3333, // Default OpenOCD GDB port
            "program": "${workspaceFolder}/build/${config:idf.appName}.elf",
            "miDebuggerPath": "${config:idf.toolsPath}/tools/xtensa-esp32-elf/esp-2022r1-11.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gdb", // Adjust path as needed
            "internalConsoleOptions": "openOnSessionStart",
            "windows": {
                "miDebuggerPath": "${config:idf.toolsPath}\\tools\\xtensa-esp32-elf\\esp-2022r1-11.2.0\\xtensa-esp32-elf\\bin\\xtensa-esp32-elf-gdb.exe" // Adjust path
            },
            "setupCommands": [
                "target remote :3333",
                "set remote hardware-watchpoint-limit 2", // Example GDB command
                "mon reset halt", // Reset and halt the target
                "thb app_main", // Set temporary breakpoint at app_main
                "c" // Continue to app_main
            ],
            "logging": { // Optional: Enable OpenOCD/GDB logging
                "engineLogging": true,
                "programOutput": true,
                "adapterOutput": true
            }
        }
    ]
}

Debugging Steps:

1. Open your ESP-IDF project (e.g., a simple multi-task example like the one from Chapter 10).
2. Set a Breakpoint: Click in the gutter to the left of a line number in your code (e.g., inside a task’s while(1) loop). A red dot will appear.
3. Start Debugging:
   • Go to the “Run and Debug” panel in VS Code (Ctrl+Shift+D).
   • Select the “Launch” configuration from the dropdown menu.
   • Click the green “Start Debugging” arrow (or press F5).
4. Process:
   • VS Code will build the project (if needed).
   • It will launch OpenOCD to connect to the JTAG probe and the ESP32.
   • It will launch GDB and connect it to OpenOCD.
   • The setupCommands in launch.json will run (resetting the ESP32, halting it, setting a temporary breakpoint at app_main, and continuing execution).
   • Execution will likely pause at app_main first.
5. Control Execution:
   • Continue (F5): Run until the next breakpoint is hit.
   • Step Over (F10): Execute the current line and stop at the next line in the same function.
   • Step Into (F11): If the current line is a function call, step into that function.
   • Step Out (Shift+F11): Continue execution until the current function returns.
   • Restart (Ctrl+Shift+F5): Restart the debugging session.
   • Stop (Shift+F5): Terminate the debugging session.
6. Inspect State:
   • Variables Panel: Shows local and global variables in the current scope. You can often modify values here.
   • Watch Panel: Add specific variables or expressions to monitor.
   • Call Stack Panel: Shows the function call sequence leading to the current location. You can switch between different tasks listed here (threads) to see their individual call stacks.
   • Debug Console Panel: Allows you to type GDB commands directly (e.g., p myVariable, info locals, info threads).
   • FreeRTOS Panel (If extension provides it): May show a list of tasks, their states (Running, Ready, Blocked), stack usage, etc.

Try This: Set breakpoints in different tasks and observe how the debugger switches context when you continue execution and hit breakpoints in another task. Inspect variables local to each task.

Example 2: Detecting Stack Overflow

1. Enable Stack Checking:
   • Run idf.py menuconfig (or use VS Code’s ESP-IDF: SDK Configuration editor).
   • Go to Component config -> FreeRTOS.
   • Enable Check for stack overflow.
   • Choose Method 2 (generally preferred as it checks on context switch).
   • Save and exit.
2. Add Hook Function: Add the following function to your main.c:

// This hook function is called by FreeRTOS when a stack overflow is detected
void vApplicationStackOverflowHook(TaskHandle_t xTask, char *pcTaskName) {
    printf("!!! STACK OVERFLOW DETECTED !!!\n");
    printf("Task Handle: %p\n", xTask);
    printf("Task Name: %s\n", pcTaskName);
    // --- Critical Error ---
    // You should not return from this function.
    // Options:
    // 1. Log error persistently (if possible).
    // 2. Trigger a system reset.
    // 3. Halt execution (e.g., using an assertion or infinite loop).
    esp_rom_printf("Halting due to stack overflow in task %s\n", pcTaskName);
    configASSERT(pdFAIL); // Trigger assertion failure
    while(1); // Ensure halt
}

3. Create Overflowing Task: Create a task that intentionally uses too much stack.

// Task designed to overflow its stack
static void stack_overflow_task(void *pvParameters) {
    // Allocate a large buffer on the stack - adjust size based on task stack size
    // If task stack is 2048, this will likely overflow.
    char large_local_buffer[2000];
    volatile int counter = 0; // Use volatile to prevent optimization

    ESP_LOGI(TAG, "Stack overflow task started. Trying to overflow...");

    // Fill the buffer to ensure stack pages are touched
    memset(large_local_buffer, 0xAA, sizeof(large_local_buffer));

    // Deep function call (optional, adds more stack usage)
    volatile int result = deep_function_call(10, large_local_buffer);

    ESP_LOGI(TAG, "Stack overflow task survived? Result: %d", result); // Should not reach here

    while(1) {
        counter++;
        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

// Helper for deep calls (optional)
volatile int deep_function_call(int depth, char* buf) {
    char local_in_deep[50]; // Add some stack usage per level
    memset(local_in_deep, 0xBB, sizeof(local_in_deep));
    if (depth <= 0) {
        return (int)buf[0]; // Access buffer
    }
    // Recursive call
    return deep_function_call(depth - 1, buf) + depth + local_in_deep[0];
}

void app_main(void) {
    ESP_LOGI(TAG, "Starting Stack Overflow Example...");

    // Create the task with a known, relatively small stack size
    // NOTE: Adjust stack size (e.g., 2048) and buffer size in task
    //       to reliably trigger the overflow based on your ESP-IDF version/config.
    xTaskCreate(stack_overflow_task, "OverflowTask", 2048, NULL, 5, NULL);

    ESP_LOGI(TAG, "Overflow task created. Waiting for potential overflow...");

    // Keep main running
    while(1) {
        vTaskDelay(pdMS_TO_TICKS(5000));
    }
}

Build, Flash, and Monitor:

Build and flash with the new configuration. Monitor the output.

Expected Output:

The stack_overflow_task will start. When it tries to allocate large_local_buffer or during the deep function calls, it will likely exceed its stack limit. FreeRTOS will detect this (likely on the next context switch attempt) and call vApplicationStackOverflowHook.

I (XXX) INTR_EXAMPLE: Starting Stack Overflow Example...
I (XXX) INTR_EXAMPLE: Overflow task created. Waiting for potential overflow...
I (XXX) TAG: Stack overflow task started. Trying to overflow...
!!! STACK OVERFLOW DETECTED !!!
Task Handle: 0x........
Task Name: OverflowTask
Halting due to stack overflow in task OverflowTask
Guru Meditation Error: Core  0 panic'ed (abort).

// --- Backtrace follows ---
// ... (Look for vApplicationStackOverflowHook and the task function)

Example 3: Collecting Runtime Statistics

1. Enable Runtime Stats:
   • Run idf.py menuconfig.
   • Go to Component config -> FreeRTOS.
   • Enable Enable FreeRTOS runtime stats collection. This usually automatically enables CONFIG_FREERTOS_USE_TRACE_FACILITY and CONFIG_FREERTOS_USE_STATS_FORMATTING_FUNCTIONS. Ensure they are enabled.
   • (Crucial) Ensure Enable generation of run time stats tables is also enabled if you want to use vTaskGetRunTimeStats.
   • Save and exit.
2. Configure High-Resolution Timer (ESP-IDF usually does this automatically):
   • ESP-IDF typically configures a hardware timer (like ESP Timer) to provide the high-resolution clock needed for portCONFIGURE_TIMER_FOR_RUN_TIME_STATS and portGET_RUN_TIME_COUNTER_VALUE. No manual configuration is usually needed when using ESP-IDF’s FreeRTOS component.
3. Add Stats Printing Task: Create a low-priority task to periodically print the stats.

#define RUNTIME_STATS_BUFFER_SIZE 1024 // Adjust if you have many tasks

// Task to periodically print runtime stats
static void runtime_stats_task(void *pvParameters) {
    char *stats_buffer = NULL; // Buffer for the formatted stats string

    while(1) {
        // Allocate buffer dynamically - consider static if preferred
        stats_buffer = (char*)malloc(RUNTIME_STATS_BUFFER_SIZE);
        if (stats_buffer == NULL) {
            ESP_LOGE(TAG, "Failed to allocate buffer for runtime stats");
            vTaskDelay(pdMS_TO_TICKS(5000)); // Retry later
            continue;
        }

        ESP_LOGI(TAG, "\n\n--- FreeRTOS Runtime Stats ---");
        vTaskGetRunTimeStats(stats_buffer); // Generate the stats table
        printf("%s\n", stats_buffer); // Print the buffer content
        ESP_LOGI(TAG, "------------------------------\n");

        free(stats_buffer); // Free the buffer
        stats_buffer = NULL;

        // Wait for the next interval
        vTaskDelay(pdMS_TO_TICKS(10000)); // Print every 10 seconds
    }
}

// Example busy task to consume CPU time
static void busy_task(void *pvParameters) {
    volatile uint64_t counter = 0;
    ESP_LOGI(TAG, "Busy task started.");
    while(1) {
        counter++;
        // No delay - this task will consume a lot of CPU
        // Add a small delay if it completely starves other tasks
        // vTaskDelay(pdMS_TO_TICKS(1));
    }
}

void app_main(void) {
    ESP_LOGI(TAG, "Starting Runtime Stats Example...");

    // Create some tasks to monitor
    xTaskCreate(busy_task, "BusyTask", 2048, NULL, 5, NULL);
    // Create the stats printing task - give it a reasonable priority
    xTaskCreate(runtime_stats_task, "StatsTask", 4096, NULL, 3, NULL);

    ESP_LOGI(TAG, "Tasks created. Runtime stats will be printed periodically.");

    // Main can idle or do other things
    // while(1) { vTaskDelay(portMAX_DELAY); }
}

Build, Flash, and Monitor:

Build and flash with the new configuration. Monitor the output.

Expected Output:

Every 10 seconds, the runtime_stats_task will print a table showing each task, its absolute runtime in timer ticks (a large number), and its percentage of the total CPU runtime since the stats were last reset (or since boot). You should see BusyTask consuming a high percentage of CPU time.

I (XXX) TAG: Starting Runtime Stats Example...
I (XXX) TAG: Busy task started.
I (XXX) TAG: Tasks created. Runtime stats will be printed periodically.

... (after 10 seconds) ...

I (XXX) TAG:

--- FreeRTOS Runtime Stats ---
Task            Abs Time        % Time
************************************************
IDLE0           8954321         89%
BusyTask        1012345         10%
StatsTask       5432            <1%
Tmr Svc         8765            <1%
ipc0            1234            <1%
main            2345            <1%
esp_timer       9876            <1%

I (XXX) TAG: ------------------------------

... (stats update every 10 seconds) ...

(Note: Task names and percentages are examples and will vary significantly)

Variant Notes

• JTAG Pinout: The specific GPIO pins used for JTAG communication can differ between ESP32 variants (ESP32, S2, S3, C3, C6, H2). Always consult the datasheet for your specific chip or module and the documentation for your development board to identify the correct JTAG pins.
• Tool Compatibility: OpenOCD, GDB, and the VS Code Espressif IDF extension generally support debugging across all common ESP32 variants. Ensure you have up-to-date versions of the tools and the extension for the best compatibility.
• Core Architecture: While debugging concepts are similar, remember that Xtensa (ESP32, S2, S3) and RISC-V (C3, C6, H2) have different instruction sets and register layouts. GDB handles this automatically based on the target configuration, but register names and assembly code will differ if you delve that deep.
• FreeRTOS Features: Stack checking, runtime stats, tracing, and core dump features are implemented consistently by ESP-IDF across the supported variants.

Common Mistakes & Troubleshooting Tips

Exercises

1. Basic Breakpointing: Take the simple multi-task blinking LED example (from Chapter 10 or create one). Set up the VS Code debugger, place breakpoints in each task’s loop, and step through the execution. Observe how the debugger switches between task contexts when you hit ‘Continue’. Inspect local variables in each task.
2. Stack Overflow Hook Details: Modify the vApplicationStackOverflowHook function from Example 2 to print more details, such as the high water mark for the overflowing task’s stack using uxTaskGetStackHighWaterMark(xTask) (call this before the overflow actually corrupts everything, perhaps periodically in the task itself for monitoring, or rely on the hook). Trigger the overflow and observe the output.
3. Runtime Stats Analysis: Create three tasks with different workloads: one that delays frequently (vTaskDelay), one that performs CPU-intensive calculations in a loop, and one that does a mix of work and delays. Use vTaskGetRunTimeStats to observe how the CPU time percentages reflect their behavior.
4. Deadlock Debugging: Create two tasks (Task A, Task B) and two mutexes (Mutex X, Mutex Y). Make Task A take Mutex X then try to take Mutex Y. Make Task B take Mutex Y then try to take Mutex X. This will cause a deadlock. Run this under the debugger. When the system hangs, pause execution. Use the debugger’s task/thread view to inspect the state of Task A and Task B. Identify that both are blocked and determine (by examining call stacks or FreeRTOS state) which mutex each task is waiting for.
5. Core Dump Analysis: Create a simple program that crashes due to a NULL pointer dereference in a specific task. Configure core dumps to output via UART. Trigger the crash, capture the Base64 encoded core dump output from the serial monitor. Use espcoredump.py info_corefile --core <coredump_file> <elf_file> and espcoredump.py dbg_corefile --core <coredump_file> <elf_file> (replace <...> placeholders) to view the backtrace and identify the crashing function and line number.

Summary

• Debugging FreeRTOS applications requires understanding concurrency issues like race conditions, deadlocks, and priority inversion.
• Use a combination of techniques: logging (ESP_LOGx), assertions (configASSERT), hardware debugging (JTAG/VS Code), and post-mortem analysis (Core Dumps).
• The VS Code debugger integrated with ESP-IDF provides powerful interactive debugging capabilities (breakpoints, stepping, variable inspection, task views).
• Enable FreeRTOS features like Stack Overflow Checking (CONFIG_FREERTOS_CHECK_STACKOVERFLOW) and Runtime Statistics (CONFIG_FREERTOS_GENERATE_RUN_TIME_STATS) via menuconfig to gain crucial insights.
• Be aware that debugging can alter system timing, potentially hiding “Heisenbugs”.
• Core dumps allow analyzing crashes after they occur, essential for field diagnostics.
• Always check menuconfig settings and JTAG connections when troubleshooting debugger issues.
• Inspecting the state of all relevant tasks is crucial for diagnosing RTOS-specific problems.

Further Reading

• ESP-IDF Programming Guide – JTAG Debugging: https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-guides/jtag-debugging/index.html (Includes setup for different probes and VS Code).
• ESP-IDF Programming Guide – Core Dump: https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-guides/core_dump.html
• ESP-IDF Programming Guide – FreeRTOS: (Check sections related to configuration options like stack checking, runtime stats).
• Mastering RTOS Debugging Techniques (FreeRTOS Blog/Resources): Search the FreeRTOS website and related blogs for articles on common RTOS debugging patterns and anti-patterns.
• SEGGER SystemView Documentation: https://www.segger.com/products/development-tools/systemview/ (For advanced tracing).
• GDB Documentation: https://www.gnu.org/software/gdb/documentation/ (For advanced command-line debugging).