Chapter 12: Task Communication: Queues Basics

Chapter Objectives

Upon completing this chapter, you will be able to:

• Explain why direct sharing of global variables between tasks is problematic and the need for Inter-Task Communication (ITC) mechanisms.
• Define what a FreeRTOS queue is and describe its key characteristics: FIFO, fixed length, fixed item size, and data copying.
• Create and manage queues using xQueueCreate and vQueueDelete.
• Send data items (including primitive types and structures) to a queue using xQueueSend and its variants.
• Receive data items from a queue using xQueueReceive.
• Understand how tasks block when sending to a full queue or receiving from an empty queue, using timeouts (xTicksToWait).
• Check the status of a queue (number of items waiting, available spaces).
• Implement the basic Producer-Consumer pattern using tasks and queues.
• Appreciate queues as a tool for decoupling tasks and improving application structure.

Introduction

In the previous chapters, we learned how to create multiple tasks that run concurrently under the control of the FreeRTOS scheduler. While this allows us to structure our application into logical, independent units, these units often need to cooperate and exchange information. For instance, a task reading sensor data might need to pass that data to another task responsible for processing or transmitting it over Wi-Fi.

A naive approach might be to use global variables shared between tasks. However, this is fraught with peril in a preemptive multitasking environment. Uncontrolled access to shared data can lead to race conditions, where the final state of the data depends on the unpredictable timing of task execution, leading to subtle and hard-to-debug errors.

FreeRTOS provides several robust and safe mechanisms for Inter-Task Communication (ITC) and synchronization. The most fundamental and commonly used mechanism for passing data between tasks is the Queue. This chapter introduces FreeRTOS queues, explaining how they work and how to use them to safely transfer information between your tasks in ESP-IDF.

Theory

The Problem with Shared Global Variables

Imagine two tasks, Task A and Task B, sharing a simple global counter variable g_counter. Task A reads the counter, increments it, and writes it back. Task B does the same.

1. Task A reads g_counter (value is 5).
2. Task A gets preempted by Task B before writing the incremented value back.
3. Task B reads g_counter (value is still 5).
4. Task B increments the value to 6.
5. Task B writes 6 back to g_counter.
6. Task B gets preempted, and Task A resumes.
7. Task A, unaware that Task B ran, increments its local copy (which was 5) to 6.
8. Task A writes 6 back to g_counter.

The counter was incremented twice, but the final value is 6, not 7. This is a simple example of a race condition. Protecting shared variables requires synchronization mechanisms (like Mutexes, covered later), but often a better approach for passing data is to avoid direct sharing altogether using queues.

Introduction to Queues

A FreeRTOS queue is a thread-safe, fixed-size buffer designed to pass data items between tasks or between interrupts and tasks. Think of it like a pipe or a conveyor belt: one task (the producer) puts items in one end, and another task (the consumer) takes them out the other end.

Key Characteristics of FreeRTOS Queues

1. FIFO (First-In, First-Out): By default, items are retrieved from the queue in the same order they were inserted. (Functions exist to add items to the front, but the basic model is FIFO).
2. Fixed Size: When a queue is created, you specify:
   • Length: The maximum number of items the queue can hold simultaneously.
   • Item Size: The size (in bytes) of each individual item stored in the queue.
3. Data Copying: This is a crucial concept. When an item is sent to a queue, the data itself is copied into the queue’s internal buffer. When an item is received, the data is copied from the queue’s buffer into a buffer provided by the receiving task. The queue does not store pointers to the original data (unless you explicitly send pointers, which is a more advanced technique). This copying ensures that the producer and consumer tasks operate on independent copies of the data, avoiding shared memory issues.
4. Thread Safety: Queue operations (Send, Receive) are implemented in a way that allows them to be safely called from multiple tasks simultaneously without causing corruption. FreeRTOS handles the internal locking required.
5. Blocking: Tasks attempting to read from an empty queue or write to a full queue can choose to enter the Blocked state for a specified duration (timeout), waiting for space to become available or an item to arrive. This allows tasks to wait efficiently without consuming CPU time (no busy-waiting).

Creating a Queue: xQueueCreate

You create a queue using xQueueCreate:

QueueHandle_t xQueueCreate( UBaseType_t uxQueueLength, UBaseType_t uxItemSize );

• uxQueueLength: The maximum number of items the queue can hold.
• uxItemSize: The size, in bytes, of each item that will be stored in the queue. Use the sizeof() operator to determine the size of the data type you intend to send (e.g., sizeof(int), sizeof(my_struct_t)). If uxItemSize is 0, the queue can only be used for synchronization (like a binary semaphore) and cannot store data.
• Return Value (QueueHandle_t): Returns a handle to the created queue if successful (memory could be allocated), otherwise returns NULL. This handle is used in all subsequent queue operations.

Memory Allocation: xQueueCreate dynamically allocates memory from the FreeRTOS heap to store the queue structure and its data buffer (uxQueueLength * uxItemSize bytes for the buffer, plus overhead for the structure). This memory is freed when the queue is deleted using vQueueDelete.

Sending Data to a Queue: xQueueSend family

The primary function to send an item to the back (end) of the queue is xQueueSend (aliased as xQueueSendToBack):

BaseType_t xQueueSend( QueueHandle_t xQueue,
const void * pvItemToQueue,
TickType_t xTicksToWait );

// Alias:
// BaseType_t xQueueSendToBack( QueueHandle_t xQueue, const void * pvItemToQueue, TickType_t xTicksToWait );

• xQueue: The handle of the queue to send to (obtained from xQueueCreate).
• pvItemToQueue: A pointer to the data item to be sent. The data pointed to by pvItemToQueue will be copied into the queue. The size of the data copied is determined by the uxItemSize specified when the queue was created.
• xTicksToWait: The maximum amount of time (in RTOS ticks) the task should remain in the Blocked state if the queue is already full, waiting for space to become available.
  • 0: Don’t block. If the queue is full, return immediately.
  • portMAX_DELAY: Block indefinitely until space becomes available. Use with caution! If the receiving task never makes space, this task will block forever.
  • Any other value: Block for that maximum number of ticks. Use pdMS_TO_TICKS() to convert milliseconds to ticks.
• Return Value (BaseType_t):
  • pdPASS: The item was successfully copied into the queue.
  • errQUEUE_FULL: The queue was full, and no space became available within the xTicksToWait period.

There is also xQueueSendToFront(), which works identically but adds the item to the front of the queue (making it the next item to be received).

Receiving Data from a Queue: xQueueReceive

To retrieve an item from the front of the queue:

BaseType_t xQueueReceive( QueueHandle_t xQueue,
                          void * pvBuffer,
                          TickType_t xTicksToWait );

• xQueue: The handle of the queue to receive from.
• pvBuffer: A pointer to a memory buffer where the received item will be copied. The buffer must be large enough to hold one item (uxItemSize bytes).
• xTicksToWait: The maximum amount of time (in RTOS ticks) the task should remain in the Blocked state if the queue is currently empty, waiting for an item to arrive. Usage is the same as for xQueueSend (0, portMAX_DELAY, specific timeout).
• Return Value (BaseType_t):
  • pdPASS: An item was successfully copied from the queue into pvBuffer.
  • errQUEUE_EMPTY: The queue was empty, and no item arrived within the xTicksToWait period.

Checking Queue State

Sometimes it’s useful to know how many items are in a queue without trying to receive one.

Deleting a Queue: vQueueDelete

When a queue is no longer needed, you should delete it to free the memory it occupies.

void vQueueDelete( QueueHandle_t xQueue );

• xQueue: The handle of the queue to delete.

Warning: Deleting a queue that tasks are still blocked on (waiting to send or receive) will cause those tasks to unblock immediately, and their send/receive operation will return failure (errQUEUE_EMPTY or errQUEUE_FULL). Ensure tasks are no longer using the queue before deleting it.

Use Cases

Queues are incredibly versatile:

• Producer-Consumer: The classic pattern where one or more tasks produce data (e.g., read sensors, receive network packets) and send it via a queue to one or more consumer tasks that process it (e.g., filter data, update display, send responses). This decouples the producer and consumer logic.
• Distributing Work: A central task can receive requests and distribute work items via a queue to a pool of worker tasks.
• Event Notification with Data: Sending a specific value or structure through a queue can signal an event and provide associated data simultaneously.

ISR (Interrupt Service Routine) Usage

Queues can also be used to send data from an ISR to a task, or receive data in an ISR (less common). However, ISRs cannot block. Therefore, special ISR-safe versions of the queue functions must be used within ISR code:

These functions have slightly different parameters (e.g., managing task wake-ups) and will be covered in more detail in chapters dealing specifically with interrupts. For now, remember do not call the standard xQueueSend or xQueueReceive functions directly from an ISR.

Practical Examples

Let’s see queues in action.

Example 1: Simple Producer-Consumer (Integers)

One task produces integer counts, another consumes and prints them.

Code:

#include <stdio.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/queue.h"
#include "esp_log.h"

static const char *TAG = "QUEUE_SIMPLE";

// Declare a handle for the queue
static QueueHandle_t integer_queue = NULL;

// Producer Task: Sends integers to the queue
void producer_task(void *pvParameter)
{
    int count = 0;
    ESP_LOGI(TAG, "Producer Task started.");

    while(1) {
        count++;
        ESP_LOGI(TAG, "Producer: Sending value %d", count);

        // Send the integer 'count' to the queue.
        // Block for a maximum of 100ms if the queue is full.
        BaseType_t result = xQueueSend(integer_queue, &count, pdMS_TO_TICKS(100));

        if (result == pdPASS) {
            ESP_LOGD(TAG, "Producer: Successfully sent %d", count);
        } else {
            ESP_LOGE(TAG, "Producer: Failed to send value %d (Queue Full?)", count);
            // Handle error - maybe delay longer or discard data
            vTaskDelay(pdMS_TO_TICKS(50)); // Small delay if queue was full
        }

        // Produce data every 1 second
        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

// Consumer Task: Receives integers from the queue
void consumer_task(void *pvParameter)
{
    int received_value;
    ESP_LOGI(TAG, "Consumer Task started.");

    while(1) {
        ESP_LOGI(TAG, "Consumer: Waiting for data...");

        // Receive an integer from the queue.
        // Block indefinitely (portMAX_DELAY) until an item arrives.
        BaseType_t result = xQueueReceive(integer_queue, &received_value, portMAX_DELAY);

        if (result == pdPASS) {
            ESP_LOGW(TAG, "Consumer: Received value = %d", received_value);
        } else {
            // This should not happen if blocking indefinitely unless queue is deleted
            ESP_LOGE(TAG, "Consumer: Failed to receive data (Queue Empty or Deleted?)");
        }
         // No delay here, task blocks on xQueueReceive
    }
}


void app_main(void)
{
    ESP_LOGI(TAG, "App Main: Starting Queue Example.");

    // Create the queue: Can hold 5 integers.
    integer_queue = xQueueCreate(5, sizeof(int));

    if (integer_queue == NULL) {
        ESP_LOGE(TAG, "Failed to create integer queue.");
        // Handle error - perhaps restart or enter error state
        return;
    }
    ESP_LOGI(TAG, "Integer queue created successfully.");

    // Create the producer and consumer tasks
    xTaskCreate(producer_task, "Producer", 2048, NULL, 5, NULL);
    xTaskCreate(consumer_task, "Consumer", 2048, NULL, 5, NULL);

    ESP_LOGI(TAG, "App Main: Tasks created.");
    // app_main can exit or do other things
}

Build, Flash, Monitor.

Observe:

• The Producer task sends values (1, 2, 3…).
• The Consumer task waits (Waiting for data...) until a value arrives, then receives and prints it.
• You’ll see the “Sending” and “Received” messages appear roughly one second apart.
• Try changing the Producer’s delay to be much faster than the Consumer’s processing (e.g., send every 100ms). Observe the queue fill up (Producer might log “Failed to send” if the queue stays full for more than its 100ms timeout). Then slow the Producer down and watch the Consumer catch up.

Example 2: Sending Structures via Queue

Demonstrates sending more complex data.

Code:

#include <stdio.h>
#include <string.h> // For strcpy
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/queue.h"
#include "esp_log.h"
#include "esp_random.h" // For random data

static const char *TAG = "QUEUE_STRUCT";

// Define a structure for sensor data
typedef struct {
    int sensor_id;
    float value;
    uint32_t timestamp;
    char status[10];
} sensor_data_t;

// Declare queue handle
static QueueHandle_t sensor_data_queue = NULL;

// Producer Task: Generates sensor data structs and sends them
void sensor_producer_task(void *pvParameter)
{
    sensor_data_t data_to_send;
    ESP_LOGI(TAG, "Sensor Producer Task started.");

    while(1) {
        // Populate the struct with some data
        data_to_send.sensor_id = 101 + (esp_random() % 5); // ID 101-105
        data_to_send.value = (float)(esp_random() % 1000) / 10.0f; // Value 0.0 - 99.9
        data_to_send.timestamp = xTaskGetTickCount(); // Use tick count as simple timestamp
        strcpy(data_to_send.status, (esp_random() % 10 < 8) ? "OK" : "WARN"); // Status OK or WARN

        ESP_LOGI(TAG, "Producer: Sending Sensor ID %d, Value %.1f, Status %s",
                 data_to_send.sensor_id, data_to_send.value, data_to_send.status);

        // Send the entire structure. FreeRTOS copies sizeof(sensor_data_t) bytes.
        BaseType_t result = xQueueSend(sensor_data_queue, &data_to_send, pdMS_TO_TICKS(50));

        if (result != pdPASS) {
            ESP_LOGE(TAG, "Producer: Failed to send sensor data (Queue Full?)");
        }

        // Produce data every 2 seconds
        vTaskDelay(pdMS_TO_TICKS(2000));
    }
}

// Consumer Task: Receives sensor data structs and processes them
void sensor_consumer_task(void *pvParameter)
{
    sensor_data_t received_data; // Buffer to hold the received struct
    ESP_LOGI(TAG, "Sensor Consumer Task started.");

    while(1) {
        ESP_LOGD(TAG, "Consumer: Waiting for sensor data...");

        // Receive a struct. Block indefinitely.
        if (xQueueReceive(sensor_data_queue, &received_data, portMAX_DELAY) == pdPASS)
        {
            ESP_LOGW(TAG, "Consumer: Received Sensor ID %d:", received_data.sensor_id);
            ESP_LOGW(TAG, "  Value: %.1f", received_data.value);
            ESP_LOGW(TAG, "  Timestamp: %lu ticks", received_data.timestamp);
            ESP_LOGW(TAG, "  Status: %s", received_data.status);
            // Process the data...
        } else {
             ESP_LOGE(TAG, "Consumer: Failed to receive sensor data!");
        }
    }
}


void app_main(void)
{
    ESP_LOGI(TAG, "App Main: Starting Struct Queue Example.");

    // Create the queue: Can hold 3 sensor_data_t structures.
    sensor_data_queue = xQueueCreate(3, sizeof(sensor_data_t));

    if (sensor_data_queue == NULL) {
        ESP_LOGE(TAG, "Failed to create sensor data queue.");
        return;
    }
    ESP_LOGI(TAG, "Sensor data queue created successfully.");

    // Create tasks
    xTaskCreate(sensor_producer_task, "SensorProd", 3072, NULL, 5, NULL);
    xTaskCreate(sensor_consumer_task, "SensorCons", 3072, NULL, 5, NULL);

    ESP_LOGI(TAG, "App Main: Tasks created.");
}

Build, Flash, Monitor.

Observe:

• The producer task creates sensor_data_t structs with varying data and sends them.
• The consumer task receives the complete structs and prints their contents.
• This demonstrates that complex data types can be easily passed by value using queues, ensuring data integrity between tasks.

Variant Notes

• API Consistency: The FreeRTOS Queue API (xQueueCreate, xQueueSend, xQueueReceive, vQueueDelete, etc.) is identical and behaves consistently across all ESP32 variants supported by ESP-IDF v5.x (ESP32, S2, S3, C3, C6, H2).
• Performance: The time taken to copy data into and out of the queue depends on the uxItemSize. Sending very large items frequently can impact performance. The underlying memory allocation and context switching speeds may vary slightly between different ESP32 core types (Xtensa vs. RISC-V) and clock speeds, but the queue logic itself remains the same.

Common Mistakes & Troubleshooting Tips

Exercises

1. Multiple Producers, Single Consumer:
   • Modify “Example 1: Simple Producer-Consumer (Integers)”.
   • Create two producer tasks (Producer A, Producer B).
   • Producer A sends positive integers (1, 2, 3…).
   • Producer B sends negative integers (-1, -2, -3…).
   • Both producers send to the same queue.
   • Keep the single consumer task.
   • Observe how the consumer receives an interleaved stream of positive and negative numbers from the shared queue.
2. Command Processor:
   • Define an enum for commands (e.g., CMD_LED_ON, CMD_LED_OFF, CMD_PRINT_MSG).
   • Define a struct for commands, containing the command enum and potentially a parameter (e.g., a string for CMD_PRINT_MSG).
   • Create a “Command Sender” task. This task periodically creates command structs (e.g., cycle through LED ON, print “Hello”, LED OFF, print “World”) and sends them to a command queue using xQueueSend.
   • Create a “Command Processor” task. This task blocks on xQueueReceive waiting for command structs.
   • When a command struct is received, the processor task uses a switch statement on the command enum to perform the requested action (toggle an LED using GPIO functions, print the message from the struct parameter). Use a queue length of at least 5 and an appropriate item size (sizeof(command_struct)).

Summary

• Directly sharing global variables between tasks is unsafe due to potential race conditions.
• Queues provide a thread-safe mechanism for Inter-Task Communication (ITC) by allowing tasks to send and receive data items via a buffer.
• Queues operate FIFO by default, have a fixed length and item size, and crucially, copy data between tasks and the queue buffer.
• Use xQueueCreate to create a queue, specifying length and item size (sizeof(type)), obtaining a QueueHandle_t.
• Use xQueueSend (or variants) to copy data into the queue. Tasks can block if the queue is full, controlled by xTicksToWait.
• Use xQueueReceive to copy data from the queue. Tasks can block if the queue is empty, controlled by xTicksToWait.
• Always check return codes (pdPASS, errQUEUE_FULL, errQUEUE_EMPTY).
• Use vQueueDelete to free queue memory when done.
• Queues decouple tasks, enabling common patterns like Producer-Consumer.
• Use ISR-safe queue functions (...FromISR) when interacting with queues from interrupts.

Further Reading

• FreeRTOS Queue Documentation: https://www.freertos.org/Embedded-RTOS-Queues.html
• FreeRTOS Queue API Reference: https://www.freertos.org/a00018.html
• ESP-IDF Programming Guide: FreeRTOS Queues: https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-reference/system/freertos_idf.html#queues (Select your target chip)
• “Using the FreeRTOS Real Time Kernel – A Practical Guide” – Chapters on Queues.