Chapter 108: MQTT Retained Messages in ESP-IDF

Chapter Objectives

By the end of this chapter, you will be able to:

• Understand the concept and purpose of MQTT retained messages.
• Explain how retained messages differ from regular (non-retained) MQTT messages.
• Describe how the MQTT broker handles the storage and delivery of retained messages.
• Configure and publish messages with the RETAIN flag set from an ESP32 client using ESP-IDF.
• Subscribe to topics and observe how retained messages are immediately delivered to new subscribers.
• Implement the mechanism to clear a retained message from an MQTT broker.
• Identify common and effective use cases for retained messages, such as device status, configuration, and last known values.
• Recognize that the functionality of retained messages is a standard MQTT feature and is consistent across all ESP32 variants.

Introduction

In our exploration of MQTT, we’ve seen how it facilitates a decoupled, asynchronous communication model through its publish-subscribe architecture. Clients publish messages to topics without needing to know who, if anyone, is currently subscribed. Similarly, subscribers receive messages without direct knowledge of the publisher. While this is powerful, a common challenge arises: when a new client subscribes to a topic, how does it get the most recent or “last known good” value for that topic if no new messages are being actively published at that moment?

Imagine a sensor that publishes its temperature reading every minute, or a device that publishes its operational status (“online”, “error”, “maintenance”) only when it changes. If a monitoring application starts up and subscribes to these topics, it would have to wait for the next publication to know the current temperature or status. This is where MQTT’s “retained messages” feature provides an elegant solution. This chapter will delve into what retained messages are, how they work, and how you can leverage them in your ESP32 projects to provide timely state information.

Theory

What are Retained Messages?

An MQTT retained message is a standard MQTT PUBLISH message that has a special flag, the RETAIN flag, set to true. When an MQTT broker receives a message with this RETAIN flag set, it performs two key actions:

1. Normal Delivery: It delivers the message to all current subscribers of that topic, just like any non-retained message.
2. Storage: It stores this message (including its payload, QoS level, and the fact that it’s retained) specifically for that topic. If there was a previously retained message for that exact topic, the new retained message replaces it. Only one retained message is stored per topic.

The magic happens when a new client subscribes to a topic. If the broker has a retained message stored for a topic that matches the new subscription’s topic filter, the broker immediately sends that last retained message to the newly subscribing client. This allows the new subscriber to get the most recent “persistent” or “last known” value for that topic without having to wait for a fresh publication from an active publisher.

graph TD
    subgraph "Later: New Subscriber Client (e.g., Dashboard App)"
        S1["<b>New Subscriber Client</b><br>(e.g., Dashboard App)<br>Subscribes to topic:<br><i>devices/thermostat_01/temperature/current</i>"]
    end

    subgraph "Broker Response to New Subscriber"
        B4["Broker checks for retained message<br>on <i>devices/thermostat_01/temperature/current</i>"]
        B5["Finds stored retained message!"]
        B6["<b>Immediately sends stored message:</b><br>Payload: {\value\: 22.5, \unit\: \C\}<br>QoS: 1, <b>RETAIN Flag: true</b><br>to New Subscriber Client"]
    end

    subgraph "Publisher Client (ESP32)"
        P1["<b>ESP32 Client</b><br>Publishes to topic:<br><i>devices/thermostat_01/temperature/current</i><br>Payload: {\value\: 22.5, \unit\: \C\}<br><b>RETAIN Flag: true</b>, QoS: 1"]
    end

    subgraph "MQTT Broker"
        B1["<b>MQTT Broker</b>"]
        B2["Delivers message to<br>CURRENT subscribers of<br><i>devices/thermostat_01/temperature/current</i>"]
        B3["<b>Stores Message for Topic:</b><br><i>devices/thermostat_01/temperature/current</i><br>Payload: {\value\: 22.5, \unit\: \C\}<br>QoS: 1, RETAIN: true<br>(Replaces any previous retained message for this topic)"]
    end


    P1 --> B1;
    B1 --> B2;
    B1 --> B3;

    S1 --> B4;
    B4 -- Yes --> B5;
    B5 --> B6;
    B4 -- No --> B7["No retained message found.<br>Subscriber waits for live messages."];

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,font-family:'Open Sans',color:#333;
    classDef primaryNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef successNode fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;

    class P1 primaryNode;
    class B1,B2,B3,B4,B6 processNode;
    class S1 primaryNode;
    class B5 successNode;
    class B7 decisionNode;

How Retained Messages Work in Detail

• Publishing with Retain: When a client publishes a message, it can set the RETAIN flag (usually a boolean parameter in the publish function). If true, the broker treats it as a retained message for that specific topic.
• Broker Storage: The broker effectively keeps a “sticky note” for each topic that has received a retained message. This note contains the last message (payload and QoS) published with the RETAIN flag set to true.

• Subscription and Delivery:
  • When a client sends a SUBSCRIBE packet, the broker processes the subscription request.
  • For each topic filter in the subscription that successfully matches one or more topics, the broker checks if any of those matching topics have a retained message.
  • If a retained message exists for a matching topic, the broker sends that message to the subscribing client. This message will also have its RETAIN flag set to true in the PUBLISH packet sent to the subscriber, allowing the subscriber to distinguish it from a “live” non-retained message.
  • This happens before any subsequent “live” messages published to that topic are forwarded.

graph TD
    A["Client sends SUBSCRIBE packet<br>with Topic Filter(s)"] --> B{"Broker receives<br>SUBSCRIBE packet"};
    B --> C["Broker processes each<br>Topic Filter in the subscription"];
    C --> D{"For each Topic Filter:"};
    D -- Loop for each filter --> E{"Identify matching Topic(s)<br>on the broker"};
    E -- For each matching Topic --> F{"Does this specific Topic<br>have a Retained Message stored?"};
    F -- Yes --> G["Broker sends the stored<br>Retained Message to the Client.<br>(PUBLISH packet with RETAIN flag = true)"];
    F -- No --> H["No Retained Message sent for this topic.<br>Client waits for new/live messages."];
    G --> I["Client receives immediate state/value"];
    H --> I;
    I --> J["Broker will then forward any<br>NEW (live) messages published<br>to matching topics."];

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,font-family:'Open Sans',color:#333;
    classDef primaryNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef checkNode fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef successNode fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;

    class A primaryNode;
    class B,C,E,J processNode;
    class D,F decisionNode;
    class G successNode;
    class H checkNode;
    class I processNode;

• Single Retained Message per Topic: It’s crucial to remember that the broker only stores one retained message per topic – the most recent one. If a new message is published to the same topic with the RETAIN flag set, it overwrites the previous retained message for that topic.
• QoS of Retained Messages: The retained message is stored with its original Quality of Service (QoS) level. When delivered to a new subscriber, the message is sent with a QoS that is the lower of the original retained message’s QoS and the maximum QoS level requested by the subscriber in their SUBSCRIBE packet for that topic.

Clearing a Retained Message

Sometimes, you need to remove a retained message from a topic on the broker, perhaps because the data is no longer valid or the device that published it is decommissioned. To clear a retained message:

1. A client must publish a message to the exact same topic for which the retained message exists.
2. This new message must have the RETAIN flag set to true.
3. The payload of this new message must be empty (zero-length).

When the broker receives such a message (RETAIN=true, empty payload), it discards the currently stored retained message for that topic. Subsequent new subscribers to that topic will no longer receive a retained message (unless a new one is published later).

graph TD

    subgraph "Result for Future Subscribers"
        S1["Later, a new Client subscribes to<br><i>devices/sensor_X/status</i>"]
        S2["Broker checks for retained message..."]
        S3["<b>No retained message found.</b><br>New Client does not receive an<br>immediate message for this topic."]
    end
    subgraph "Client Action to Clear Retained Message"
        C1["<b>Client Intention:</b><br>Clear retained message for topic:<br><i>devices/sensor_X/status</i>"]
        C2["Client constructs a PUBLISH packet:<br>- Topic: <i>devices/sensor_X/status</i> (<b>exact match</b>)<br>- Payload: <b>Empty</b> (zero-length)<br>- <b>RETAIN Flag: true</b><br>- QoS: (e.g., 0 or 1)"]
        C3["Client publishes this special message"]
    end

    subgraph "MQTT Broker Action"
        B1["Broker receives PUBLISH packet from Client"]
        B2{"Is RETAIN flag true?"}
        B3{"Is Payload empty?"}
        B4["Broker identifies the topic:<br><i>devices/sensor_X/status</i>"]
        B5["<b>Broker discards/deletes</b><br>the currently stored retained message<br>for <i>devices/sensor_X/status</i>."]
        B6["The topic <i>devices/sensor_X/status</i><br>now has no retained message."]
    end


    C1 --> C2 --> C3 --> B1;
    B1 --> B2;
    B2 -- True --> B3;
    B2 -- False --> BFail1["Not a clear operation<br>(or not a retained publish)"];
    B3 -- True --> B4;
    B3 -- False --> BFail2["Not a clear operation<br>(payload not empty)"];
    B4 --> B5 --> B6;

    S1 --> S2 --> S3;

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,font-family:'Open Sans',color:#333;
    classDef primaryNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef successNode fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;
    classDef checkNode fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;


    class C1,C2,C3 primaryNode;
    class B1,B4,B5,B6 processNode;
    class B2,B3 decisionNode;
    class BFail1,BFail2 checkNode;
    class S1,S2,S3 successNode;

Warning: Simply publishing a message with RETAIN=false to a topic does not clear an existing retained message on that topic. The explicit action of publishing an empty message with RETAIN=true is required.

Retained Messages vs. Last Will and Testament (LWT)

It’s easy to confuse retained messages with the Last Will and Testament (LWT) feature discussed in Chapter 107, but they serve different primary purposes, though they can be used together:

• Retained Message:
  • Published by the client intentionally during its normal operation.
  • Purpose: To provide the last known good value or status for a topic to new subscribers.
  • Stored by the broker as long as it’s not cleared or replaced.
• Last Will and Testament (LWT):
  • A message pre-configured by the client but published by the broker on the client’s behalf.
  • Purpose: To signal an ungraceful disconnection of the client.
  • Published only when the client disconnects abnormally.

Synergy: A common and powerful pattern is to make the LWT message itself a retained message. For example, a device’s LWT could be to publish {"status": "offline"} to devices/my_device/status with RETAIN=true. If the device disconnects ungracefully, this “offline” status is published and retained. When the device reconnects, it would then publish {"status": "online"} to the same topic, also with RETAIN=true, to update the retained status. This ensures that any client subscribing to devices/my_device/status always gets the latest online/offline state.

Common Use Cases for Retained Messages

Important Considerations

• Broker Storage: While powerful, retained messages consume resources (memory/disk) on the MQTT broker. Avoid overusing them, especially on topics with very frequent updates where only the live data stream is typically of interest. Retain messages that represent a persistent state or a significant last value.
• Payload Size: Keep retained message payloads reasonably concise.
• Topic Design: Design your topics carefully. A retained message is specific to one topic.
• Security (ACLs): Broker Access Control Lists (ACLs) can be configured to control which clients are allowed to publish messages with the RETAIN flag (and thus alter or clear retained messages) on specific topics. This is important to prevent unauthorized modification of persistent state.
• Message Expiry (MQTT 5.0): MQTT 5.0 introduces a “Message Expiry Interval” property, which can also be applied to retained messages. This allows a retained message to be automatically removed by the broker after a certain period, which can be useful for data that becomes stale. ESP-IDF’s esp-mqtt client primarily targets MQTT 3.1.1, but awareness of MQTT 5.0 features is good for future context.

Practical Examples

Let’s demonstrate using retained messages with an ESP32. Our ESP32 will:

1. Publish its current boot count (read from NVS) as a retained message.
2. Publish a static “device type” as a retained message.
3. Provide a mechanism (e.g., after a delay or a button press in a more complex app) to clear one of these retained messages.

Assumed Setup:

• An MQTT broker is running and accessible.
• MQTT client tool (e.g., MQTT Explorer) for observation.
• ESP-IDF v5.x project environment, including NVS for storing the boot count.

Code Snippets (ESP-IDF v5.x)

#include <stdio.h>
#include <string.h>
#include "esp_log.h"
#include "esp_wifi.h"
#include "nvs_flash.h"
#include "nvs.h"
#include "esp_event.h"
#include "esp_netif.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "mqtt_client.h"
#include "cJSON.h"
#include "esp_mac.h"

static const char *TAG = "RETAIN_EXAMPLE";
static const char *NVS_NAMESPACE = "storage";
static const char *NVS_BOOT_COUNT_KEY = "boot_count";

// MQTT Configuration
#define MQTT_BROKER_URI "mqtt://test.mosquitto.org" // Example broker
#define DEVICE_BASE_TOPIC "esp32_devices/"

// Buffers
char device_id_str[18];
char boot_count_topic[128];
char device_type_topic[128];
char boot_count_payload[32];
char device_type_payload[] = "{\"type\": \"ESP32_SensorNode\", \"version\": \"1.0\"}";

esp_mqtt_client_handle_t client_handle = NULL;

static void log_error_if_nonzero(const char *message, int error_code) {
    if (error_code != 0) {
        ESP_LOGE(TAG, "Last error %s: 0x%x", message, error_code);
    }
}

static esp_err_t get_and_increment_boot_count(uint32_t *boot_count) {
    nvs_handle_t my_handle;
    esp_err_t err;

    err = nvs_open(NVS_NAMESPACE, NVS_READWRITE, &my_handle);
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Error (%s) opening NVS handle!", esp_err_to_name(err));
        return err;
    }

    err = nvs_get_u32(my_handle, NVS_BOOT_COUNT_KEY, boot_count);
    if (err == ESP_ERR_NVS_NOT_FOUND) {
        ESP_LOGI(TAG, "Boot count not found in NVS, initializing to 0.");
        *boot_count = 0;
    } else if (err != ESP_OK) {
        ESP_LOGE(TAG, "Error (%s) reading boot count from NVS!", esp_err_to_name(err));
        nvs_close(my_handle);
        return err;
    }

    ESP_LOGI(TAG, "Current boot count: %lu", *boot_count);
    (*boot_count)++; // Increment boot count

    err = nvs_set_u32(my_handle, NVS_BOOT_COUNT_KEY, *boot_count);
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Error (%s) writing updated boot count to NVS!", esp_err_to_name(err));
    } else {
        err = nvs_commit(my_handle);
        if (err != ESP_OK) {
            ESP_LOGE(TAG, "Error (%s) committing NVS changes!", esp_err_to_name(err));
        } else {
            ESP_LOGI(TAG, "Incremented boot count to: %lu and saved to NVS.", *boot_count);
        }
    }
    nvs_close(my_handle);
    return err;
}

static void mqtt_event_handler(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data) {
    ESP_LOGD(TAG, "Event dispatched from event loop base=%s, event_id=%ld", base, event_id);
    esp_mqtt_event_handle_t event = event_data;
    client_handle = event->client; // Store client handle
    int msg_id;
    uint32_t current_boot_count = 0;

    switch ((esp_mqtt_event_id_t)event_id) {
    case MQTT_EVENT_CONNECTED:
        ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");

        // Get current boot count
        get_and_increment_boot_count(&current_boot_count); // This also increments for next boot
                                                          // For this publication, we use the value *before* increment for current boot.
                                                          // Or, get it first, then publish, then increment. Let's adjust for clarity.
        
        // Let's re-fetch the value that was just incremented to publish it.
        nvs_handle_t temp_handle;
        if (nvs_open(NVS_NAMESPACE, NVS_READONLY, &temp_handle) == ESP_OK) {
            nvs_get_u32(temp_handle, NVS_BOOT_COUNT_KEY, &current_boot_count);
            nvs_close(temp_handle);
        }


        // Publish boot count as a retained message
        snprintf(boot_count_payload, sizeof(boot_count_payload), "{\"count\": %lu}", current_boot_count);
        msg_id = esp_mqtt_client_publish(client_handle, boot_count_topic, boot_count_payload, 0, 1, 1); // QoS 1, Retain 1
        ESP_LOGI(TAG, "Published boot count (retained): msg_id=%d, topic=%s, data=%s", msg_id, boot_count_topic, boot_count_payload);

        // Publish device type as a retained message
        msg_id = esp_mqtt_client_publish(client_handle, device_type_topic, device_type_payload, 0, 1, 1); // QoS 1, Retain 1
        ESP_LOGI(TAG, "Published device type (retained): msg_id=%d, topic=%s, data=%s", msg_id, device_type_topic, device_type_payload);
        
        // Example: Subscribe to a dummy topic (not strictly needed for this example)
        // msg_id = esp_mqtt_client_subscribe(client_handle, "esp32_devices/some_esp_cmd_topic", 0);
        // ESP_LOGI(TAG, "Sent subscribe successful, msg_id=%d", msg_id);
        break;
    case MQTT_EVENT_DISCONNECTED:
        ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
        break;
    case MQTT_EVENT_SUBSCRIBED:
        ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
        break;
    case MQTT_EVENT_UNSUBSCRIBED:
        ESP_LOGI(TAG, "MQTT_EVENT_UNSUBSCRIBED, msg_id=%d", event->msg_id);
        break;
    case MQTT_EVENT_PUBLISHED:
        ESP_LOGI(TAG, "MQTT_EVENT_PUBLISHED, msg_id=%d", event->msg_id);
        break;
    case MQTT_EVENT_DATA:
        ESP_LOGI(TAG, "MQTT_EVENT_DATA");
        // printf("TOPIC=%.*s\r\n", event->topic_len, event->topic);
        // printf("DATA=%.*s\r\n", event->data_len, event->data);
        break;
    case MQTT_EVENT_ERROR:
        ESP_LOGI(TAG, "MQTT_EVENT_ERROR");
        if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
            log_error_if_nonzero("reported from esp-tls", event->error_handle->esp_tls_last_esp_err);
            // ... other error logging
        }
        break;
    default:
        ESP_LOGI(TAG, "Other event id:%d", event->id);
        break;
    }
}

static void clear_retained_message_task(void *pvParameters) {
    vTaskDelay(pdMS_TO_TICKS(30000)); // Wait 30 seconds after connection

    if (client_handle) {
        ESP_LOGI(TAG, "Attempting to clear retained message for boot count topic: %s", boot_count_topic);
        int msg_id = esp_mqtt_client_publish(client_handle, boot_count_topic, "", 0, 0, 1); // Empty payload, QoS 0, Retain 1
        if (msg_id != -1) {
            ESP_LOGI(TAG, "Sent publish to clear retained boot count, msg_id=%d", msg_id);
        } else {
            ESP_LOGE(TAG, "Failed to send publish to clear retained boot count.");
        }
    } else {
        ESP_LOGE(TAG, "MQTT client handle not available to clear retained message.");
    }
    vTaskDelete(NULL); // Delete self
}


static void mqtt_app_start(void) {
    uint8_t mac[6];
    esp_read_mac(mac, ESP_MAC_WIFI_STA);
    snprintf(device_id_str, sizeof(device_id_str), "%02X%02X%02X%02X%02X%02X",
             mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);

    snprintf(boot_count_topic, sizeof(boot_count_topic), "%s%s/boot_count", DEVICE_BASE_TOPIC, device_id_str);
    snprintf(device_type_topic, sizeof(device_type_topic), "%s%s/device_type", DEVICE_BASE_TOPIC, device_id_str);

    ESP_LOGI(TAG, "Device ID: %s", device_id_str);
    ESP_LOGI(TAG, "Boot Count Topic: %s", boot_count_topic);
    ESP_LOGI(TAG, "Device Type Topic: %s", device_type_topic);
    
    // Initialize NVS and increment boot count *before* MQTT starts,
    // so the handler can read the correct current value.
    uint32_t initial_boot_count;
    get_and_increment_boot_count(&initial_boot_count); // This ensures NVS is initialized and value is ready

    const esp_mqtt_client_config_t mqtt_cfg = {
        .broker.address.uri = MQTT_BROKER_URI,
        .credentials.client_id = device_id_str,
        // No LWT for this specific example, focusing on retained messages
    };

    ESP_LOGI(TAG, "Initializing MQTT client...");
    esp_mqtt_client_handle_t local_client = esp_mqtt_client_init(&mqtt_cfg);
    esp_mqtt_client_register_event(local_client, ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
    esp_mqtt_client_start(local_client);

    // Create a task to clear a retained message after some time
    xTaskCreate(clear_retained_message_task, "clear_retained_task", 2048, NULL, 5, NULL);
}

// Simplified WiFi Init (use your robust version from previous chapters)
static void wifi_init_sta(void) {
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta();
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    wifi_config_t wifi_config = {
        .sta = { .ssid = "YOUR_WIFI_SSID", .password = "YOUR_WIFI_PASSWORD" },
    };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start());
    ESP_LOGI(TAG, "wifi_init_sta finished. Connecting...");
    // In a real app, wait for IP_EVENT_STA_GOT_IP.
    vTaskDelay(pdMS_TO_TICKS(7000)); // Allow time for connection
}

void app_main(void) {
    ESP_LOGI(TAG, "[APP] Startup..");
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    wifi_init_sta();
    mqtt_app_start();
}

Tip:

• The esp_mqtt_client_publish() function takes retain as its last boolean argument.
• get_and_increment_boot_count handles NVS operations. Ensure NVS is initialized in app_main.
• The logic for publishing the boot count in MQTT_EVENT_CONNECTED was slightly adjusted to ensure the current boot number (after increment for this boot) is published.
• The clear_retained_message_task demonstrates publishing an empty string ("") with retain = 1 to clear the message. The msg_len = 0 in esp_mqtt_client_publish is important here, or you can pass strlen("") which is 0.

Build Instructions

1. Save the code: Place into main/main.c.
2. Configure WiFi: Update "YOUR_WIFI_SSID" and "YOUR_WIFI_PASSWORD".
3. Configure MQTT Broker: Update MQTT_BROKER_URI if needed.
4. CMakeLists.txt: Ensure nvs_flash, esp_wifi, cJSON (if used for complex payloads, though simpler here), and esp_event are in REQUIRES for your main component.
   # In main/CMakeLists.txt
# ...
REQUIRES nvs_flash esp_wifi esp_event # cJSON if you use it more extensively
# ...
5. Build: idf.py build

Run/Flash/Observe Steps

1. Flash: idf.py -p /dev/ttyUSB0 flash monitor (adjust port).
2. Prepare MQTT Explorer (or other client):
   • Connect to your MQTT broker.
   • Before the ESP32 connects for the first time, subscribe to:
     • esp32_devices/YOUR_ESP32_MAC/boot_count
     • esp32_devices/YOUR_ESP32_MAC/device_type
     • (You’ll see the MAC in ESP32 logs first time, then use it. Or use wildcard esp32_devices/+/+)
   • Initially, you should see no messages for these topics.
3. Observe ESP32 Connection:
   • The ESP32 boots, connects to WiFi, then MQTT.
   • In MQTT_EVENT_CONNECTED, it publishes the boot count and device type with retain=1.
   • In MQTT Explorer: You should immediately see these messages appear on their respective topics. Notice that MQTT Explorer usually indicates if a message is retained (often with a yellow “retained” marker or similar).
4. Test Retention:
   • In MQTT Explorer, unsubscribe from one of the topics (e.g., .../boot_count).
   • Then, re-subscribe to it. You should instantly receive the last published boot count message again because it was retained by the broker.
   • Restart the ESP32. Observe the boot count increment and the new value being published and retained, replacing the old one.
5. Observe Clearing Retained Message:
   • After about 30 seconds (as per clear_retained_message_task), the ESP32 will attempt to clear the retained message for the .../boot_count topic by publishing an empty payload with retain=1.
   • In MQTT Explorer:
     • You might see the .../boot_count topic either disappear (if your client hides topics with no retained message and no recent live messages) or its retained message marker vanish. Some clients might show the empty message briefly.
     • If you now unsubscribe and re-subscribe to .../boot_count, you should not receive any message immediately. The retained message is gone.
     • The .../device_type topic should still have its retained message.

Variant Notes

The concept and usage of MQTT retained messages are defined by the MQTT protocol standard. The ESP-IDF esp-mqtt client library provides the means to set the RETAIN flag during publication, and this functionality is consistent across all ESP32 variants:

• ESP32
• ESP32-S2
• ESP32-S3
• ESP32-C3
• ESP32-C6
• ESP32-H2

Key Points:

1. API Consistency: The esp_mqtt_client_publish() function and its retain parameter are the same regardless of the ESP32 chip.
2. Broker Interaction: The handling of retained messages (storage, delivery to new subscribers, clearing) is done by the MQTT broker, not the ESP32 client directly (beyond setting the flag).
3. Use Cases: While the data you choose to retain might be influenced by the specific capabilities or application of a particular ESP32 variant (e.g., an ESP32-S3 might retain the status of an AI model, while an ESP32-C3 might retain a simpler sensor state), the mechanism of retaining the message is identical.

Therefore, the code examples and principles discussed in this chapter for publishing and clearing retained messages are directly applicable to any ESP32 project using ESP-IDF v5.x without variant-specific modifications to the MQTT logic itself.

Common Mistakes & Troubleshooting Tips

Exercises

1. Dynamic Device Configuration via Retained Messages:
   • Scenario: An ESP32 device needs to fetch its operational configuration (e.g., a “logging_level”: “INFO” or “DEBUG”) when it boots. This configuration is set by a central server/script.
   • Task:
     1. A Python script (or MQTT Explorer) publishes a JSON configuration string (e.g., {"logging_level": "DEBUG", "sample_interval_ms": 5000}) as a retained message to a topic like config/devices/YOUR_ESP32_ID/settings.
     2. The ESP32, upon connecting to MQTT, subscribes to this topic.
     3. In the MQTT_EVENT_DATA handler, when it receives this retained configuration message, it parses the JSON and applies the settings (e.g., adjusts its esp_log_level_set() and a variable for sample interval).
     4. Test by changing the retained message from the Python script/MQTT Explorer and restarting the ESP32 to see if it picks up the new configuration.
2. Retained Last Motion Detected Timestamp:
   • Scenario: An ESP32 with a PIR motion sensor.
   • Task:
     1. When motion is detected, the ESP32 publishes a message to sensors/pir_livingroom/last_motion with a payload containing the current timestamp (e.g., {"timestamp": 1678886400}). This message must be retained.
     2. A monitoring client subscribing to this topic will always get the timestamp of the last detected motion.
     3. Consider how this differs from publishing every motion event without retain. What’s the benefit of retaining only the last one?
3. Visual Thermostat Control with Retained Setpoint:
   • Scenario: Simulate a thermostat. An ESP32 “controls” a virtual temperature.
   • Task:
     1. The ESP32 publishes its current temperature setpoint (e.g., 22.0) as a retained message to home/thermostat/setpoint_celsius/status.
     2. A client (MQTT Explorer) can subscribe to this to always see the current setpoint.
     3. The client can publish a new desired setpoint (e.g., 23.5) to a command topic like home/thermostat/setpoint_celsius/command.
     4. The ESP32 subscribes to the command topic. When it receives a new setpoint, it updates its internal value and then re-publishes the new setpoint to home/thermostat/setpoint_celsius/status as a retained message, overwriting the old one.
     5. Observe in MQTT Explorer how the retained status updates immediately after a command.

Summary

• MQTT Retained Messages persist the last message published with the RETAIN flag set to true on a specific topic within the broker.
• They allow new subscribers to immediately receive the last known value for a topic without waiting for a live update.
• To publish a retained message, set the retain parameter to true in the MQTT publish function.
• Only one message is retained per topic; a new retained message overwrites the previous one.
• To clear a retained message, publish an empty (zero-length) payload to the same topic with the RETAIN flag set to true.
• Common use cases include device status, last known sensor readings, configuration parameters, and initial states for dashboards.
• Retained messages are distinct from LWT but can be used together (e.g., LWT message itself being retained).
• The functionality is standard MQTT and works consistently across all ESP32 variants with ESP-IDF.
• Use retained messages judiciously to avoid unnecessary broker load, especially for high-frequency data.

Further Reading

• HiveMQ MQTT Essentials – Part 8: Retained Messages:
  • https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages/
• EMQx Blog: MQTT Retained Messages: Explained and Explored:
  • https://www.emqx.com/en/blog/mqtt-retained-messages-explained-and-explored
• ESP-IDF Programming Guide – MQTT Client: (Refer to your specific ESP-IDF version for esp_mqtt_client_publish details)
  • https://docs.espressif.com/projects/esp-idf/en/v5.2.1/esp32/api-reference/protocols/mqtt.html
• OASIS MQTT Version 3.1.1 Specification – RETAIN flag:
  • http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718038 (Section 3.3.1.3 RETAIN)