Chapter 103: MQTT QoS Levels and Message Persistence

Chapter Objectives

After completing this chapter, you will be able to:

• Understand the three MQTT Quality of Service (QoS) levels and their guarantees.
• Explain the message exchange flows for QoS 0, QoS 1, and QoS 2.
• Implement message publishing and subscribing with different QoS levels using ESP-IDF.
• Understand the concept of MQTT clean sessions and message persistence.
• Configure persistent sessions for reliable message delivery to occasionally connected clients.
• Identify common issues related to QoS and persistence and how to troubleshoot them.

Introduction

In the previous chapters, we introduced the MQTT protocol and how to implement a basic MQTT client on the ESP32. While sending and receiving messages is fundamental, ensuring that these messages are delivered reliably according to the application’s needs is equally critical, especially in IoT scenarios where network connectivity can be intermittent or unreliable.

MQTT provides a mechanism to address this through Quality of Service (QoS) levels. These levels allow the client and broker to agree on a guarantee for message delivery. Furthermore, MQTT supports message persistence through session management, enabling messages to be delivered to clients even if they are temporarily disconnected.

This chapter will explore the intricacies of MQTT QoS levels and message persistence. We will examine the theory behind each QoS level, how they are implemented in practice using the esp-mqtt client library in ESP-IDF, and how to manage client sessions for robust IoT communication. Understanding these concepts is vital for building resilient and dependable IoT applications that can handle the uncertainties of real-world deployments.

Theory

MQTT Quality of Service (QoS) Levels

MQTT defines three levels of Quality of Service. These levels determine the guarantee of message delivery between the client and the broker. It’s important to note that QoS applies to the path between a publishing client and the broker, and separately to the path between the broker and a subscribing client. The end-to-end QoS is effectively the minimum of the QoS level used for publishing and the QoS level used for subscribing.

1. QoS 0: At most once delivery (Fire and Forget)

• Guarantee: The message is delivered at most once, or not at all. There is no acknowledgment of receipt from the broker (for publishes) or the client (for subscribed messages).
• Mechanism:
  • Publishing (Client to Broker): The client sends a PUBLISH packet to the broker and does not expect any acknowledgment.
  • Delivering (Broker to Client): The broker sends a PUBLISH packet to the subscribing client and does not expect any acknowledgment.
• Message Flow:
  • Client PUBLISH -> Broker
  • Broker PUBLISH -> Subscriber
• Pros:
  • Lowest overhead (fastest).
  • Least network traffic.
  • Simple to implement.
• Cons:
  • Messages can be lost if the network connection is broken or if the recipient is unavailable. No guarantee of delivery.
• Use Cases: Suitable for non-critical sensor data where occasional loss is acceptable, or for applications where data is sent very frequently, and the latest value is more important than historical ones (e.g., telemetry).

sequenceDiagram
    participant Client
    participant Broker
    participant Subscriber

    Client->>Broker: PUBLISH (Topic, Payload, QoS 0)
    Note over Client,Broker: No acknowledgment expected by Client

    Broker->>Subscriber: PUBLISH (Topic, Payload, QoS 0)
    Note over Broker,Subscriber: No acknowledgment expected by Broker <br> (Subscriber receives if connected and subscribed)

2. QoS 1: At least once delivery (Acknowledged Delivery)

• Guarantee: The message is guaranteed to be delivered at least once. It might be delivered multiple times if acknowledgments are lost and the message is resent.
• Mechanism:
  • Publishing (Client to Broker):
    1. The client sends a PUBLISH packet with a unique Packet Identifier.
    2. The client stores the message locally until it receives a PUBACK (Publish Acknowledgment) packet from the broker with the same Packet Identifier.
    3. If the client does not receive a PUBACK within a certain time, or if the connection drops, it re-sends the PUBLISH packet with the DUP (duplicate) flag set.
  • Delivering (Broker to Client):
    1. The broker sends a PUBLISH packet with a unique Packet Identifier to the subscribing client.
    2. The broker expects a PUBACK packet from the client.
    3. If the broker does not receive a PUBACK, it re-sends the PUBLISH packet (with DUP flag set) until acknowledged.
• Message Flow (Client to Broker):
  1. Client —PUBLISH (Packet ID: X, DUP=0)–> Broker
  2. Broker —PUBACK (Packet ID: X)–> Client
• Message Flow (Broker to Subscriber):
  1. Broker —PUBLISH (Packet ID: Y, DUP=0)–> Subscriber
  2. Subscriber —PUBACK (Packet ID: Y)–> Broker
• Pros:
  • Guarantees message delivery.
  • Relatively simple acknowledgment mechanism.
• Cons:
  • Higher overhead than QoS 0 due to acknowledgments and potential retransmissions.
  • Possibility of duplicate messages if PUBACK is lost and the sender retransmits. The receiving application must be able to handle duplicates (e.g., by using message IDs within the payload).
• Use Cases: Suitable for most IoT applications where message loss is not acceptable, but occasional duplicate messages can be handled by the application logic (e.g., commands, important alerts).

sequenceDiagram
    participant Client
    participant Broker
    participant Subscriber

    rect rgb(250, 250, 210)
        note right of Client: Path 1: Client to Broker
        Client->>+Broker: PUBLISH (Packet ID: X, QoS 1, DUP=0)
        Broker-->>-Client: PUBACK (Packet ID: X)
        Note over Client: Message stored until PUBACK received. <br> Retransmits with DUP=1 if no PUBACK.
    end

    rect rgb(230, 240, 250)
        note right of Broker: Path 2: Broker to Subscriber
        Broker->>+Subscriber: PUBLISH (Packet ID: Y, QoS 1, DUP=0)
        Subscriber-->>-Broker: PUBACK (Packet ID: Y)
        Note over Broker: Message (or state) stored until PUBACK received. <br> Retransmits with DUP=1 if no PUBACK.
    end

3. QoS 2: Exactly once delivery (Assured Delivery)

• Guarantee: The message is guaranteed to be delivered exactly once. This is the highest and most reliable QoS level.
• Mechanism: This involves a four-part handshake to ensure that the message is delivered once and only once.
  • Publishing (Client to Broker):
    1. Client to Broker: Client sends PUBLISH (Packet ID: X, DUP=0). Client stores the message.
    2. Broker to Client: Broker receives PUBLISH, stores the Packet ID (and message if it’s a new one), and replies with PUBREC (Publish Received).
    3. Client to Broker: Client receives PUBREC, discards the stored message, stores the Packet ID (to track the PUBREC), and replies with PUBREL (Publish Release).
    4. Broker to Client: Broker receives PUBREL, discards the stored Packet ID, and replies with PUBCOMP (Publish Complete).
    5. Client: Client receives PUBCOMP and discards the stored Packet ID. The message is now successfully published.
  • Delivering (Broker to Client): A similar four-part handshake occurs between the broker and the subscribing client.
    1. Broker to Subscriber: Broker sends PUBLISH (Packet ID: Y, DUP=0). Broker stores the message state.
    2. Subscriber to Broker: Subscriber receives PUBLISH, stores Packet ID, and replies with PUBREC.
    3. Broker to Subscriber: Broker receives PUBREC, discards message, stores Packet ID, and replies with PUBREL.
    4. Subscriber to Broker: Subscriber receives PUBREL, processes message, discards Packet ID, and replies with PUBCOMP.
    5. Broker: Broker receives PUBCOMP and discards stored Packet ID.
• Message Flow (Client to Broker):
  1. Client —PUBLISH (Packet ID: X)–> Broker
  2. Broker —PUBREC (Packet ID: X)–> Client
  3. Client —PUBREL (Packet ID: X)–> Broker
  4. Broker —PUBCOMP (Packet ID: X)–> Client
• Pros:
  • Highest level of reliability; no message loss and no duplicates.
• Cons:
  • Highest overhead due to the four-part handshake.
  • Increased network traffic and latency.
  • More complex to implement (though largely handled by the MQTT library).
• Use Cases: Critical applications where message loss or duplication is unacceptable, such as financial transactions, critical control commands (e.g., stopping machinery), or billing systems.

sequenceDiagram
    participant Client
    participant Broker

    Note over Client, Broker: This illustrates Client to Broker. <br> A similar 4-part handshake occurs Broker to Subscriber.

    Client->>+Broker: 1. PUBLISH (Packet ID: X, QoS 2, DUP=0)
    Note over Client: Stores message.
    
    Broker-->>-Client: 2. PUBREC (Packet ID: X)
    Note over Broker: Stores Packet ID (and message if new).
    
    Client->>+Broker: 3. PUBREL (Packet ID: X)
    Note over Client: Discards message, stores Packet ID for PUBREC.
    
    Broker-->>-Client: 4. PUBCOMP (Packet ID: X)
    Note over Broker: Discards Packet ID.
    Note over Client: Discards Packet ID. Message delivered exactly once.

Important Note on QoS Downgrade: When a message is published with a certain QoS, and a client subscribes to that topic with a different QoS, the broker will typically deliver the message at the lower of the two QoS levels. For example, if a message is published with QoS 2, but a client subscribes with QoS 1, the broker will deliver that message to that specific client using the QoS 1 flow.

Message Persistence and Clean Sessions

Message persistence in MQTT refers to the broker’s ability to store messages for subscribing clients that are not currently connected. This is tightly coupled with the concept of “sessions.”

Sessions

When an MQTT client connects to a broker, it can request the start of a session. This session can outlive the network connection.

• Clean Session (MQTT v3.1.1) / Clean Start (MQTT v5):
  • CleanSession = 1 (true): This is the default for many clients. When the client connects, it starts a new session. Any previous session information (subscriptions, queued messages) for that ClientID is discarded by the broker. When the client disconnects, the session is terminated, and any messages published for its subscriptions while it was offline are not stored.
  • CleanSession = 0 (false): The client requests a persistent session.
    • If the broker has a previous session for this ClientID, it resumes that session. This includes any existing subscriptions and any QoS 1 or QoS 2 messages that were published for those subscriptions while the client was disconnected.
    • If no session exists, a new persistent session is created.
    • When the client disconnects, the broker keeps the session alive, including its subscriptions, and queues incoming QoS 1 and QoS 2 messages for those subscriptions.
    • MQTT v5 Note: MQTT v5 refines this with CleanStart and SessionExpiryInterval.
      • CleanStart = 1: Similar to CleanSession = 1.
      • CleanStart = 0: The client requests to resume an existing session.
      • SessionExpiryInterval: A value in seconds indicating how long the broker should keep the session information after the client disconnects. If 0, the session ends on disconnect. If > 0, the session persists for that duration. If set to maximum (0xFFFFFFFF), the session does not expire.

graph TD


    subgraph "Scenario: Clean Session (CleanSession = true / CleanStart = 1)"
        direction TB
        CS_Connect["<b>1. Client Connects</b> <br> ClientID: 'clientA' <br> CleanSession: true"]
        CS_Broker_Connect["Broker: Creates <b>NEW</b> session for 'clientA'. <br> Discards any old session for 'clientA'."]
        CS_Subscribe["<b>2. Client Subscribes</b> <br> Topic: '/updates' (QoS 1)"]
        CS_Broker_Subscribe["Broker: Adds subscription to current session."]
        CS_Disconnect["<b>3. Client Disconnects</b>"]
        CS_Broker_Disconnect["Broker: <b>Terminates session</b> for 'clientA'. <br> Subscriptions are lost. <br> No messages queued."]
        CS_Offline_Publish["<b>4. Message Published to '/updates'</b> <br> (while 'clientA' is offline)"]
        CS_Broker_Offline_Publish["Broker: Message <b>NOT</b> queued for 'clientA'."]
        CS_Reconnect["<b>5. Client Reconnects</b> <br> ClientID: 'clientA' <br> CleanSession: true"]
        CS_Broker_Reconnect["Broker: Creates <b>NEW</b> session. <br> Client must re-subscribe. <br> No queued messages delivered."]

        CS_Connect --> CS_Broker_Connect
        CS_Broker_Connect --> CS_Subscribe
        CS_Subscribe --> CS_Broker_Subscribe
        CS_Broker_Subscribe --> CS_Disconnect
        CS_Disconnect --> CS_Broker_Disconnect
        CS_Broker_Disconnect --> CS_Offline_Publish
        CS_Offline_Publish --> CS_Broker_Offline_Publish
        CS_Broker_Offline_Publish --> CS_Reconnect
        CS_Reconnect --> CS_Broker_Reconnect
    end

    subgraph "Scenario: Persistent Session (CleanSession = false / CleanStart = 0, SessionExpiryInterval > 0)"
        direction TB
        PS_Connect["<b>1. Client Connects</b> <br> ClientID: 'clientB' <br> CleanSession: false"]
        PS_Broker_Connect["Broker: Resumes existing session for 'clientB' <br> OR creates NEW persistent session."]
        PS_Subscribe["<b>2. Client Subscribes</b> <br> Topic: '/data' (QoS 1)"]
        PS_Broker_Subscribe["Broker: Adds subscription to persistent session."]
        PS_Disconnect["<b>3. Client Disconnects</b>"]
        PS_Broker_Disconnect["Broker: <b>Keeps session</b> for 'clientB' active. <br> Subscriptions maintained. <br> Queues QoS 1 & 2 messages."]
        PS_Offline_Publish["<b>4. Message Published to '/data' (QoS 1)</b> <br> (while 'clientB' is offline)"]
        PS_Broker_Offline_Publish["Broker: Message <b>IS QUEUED</b> for 'clientB'."]
        PS_Reconnect["<b>5. Client Reconnects</b> <br> ClientID: 'clientB' <br> CleanSession: false"]
        PS_Broker_Reconnect["Broker: Resumes session. <br> Delivers queued messages. <br> Subscriptions are still active."]

        PS_Connect --> PS_Broker_Connect
        PS_Broker_Connect --> PS_Subscribe
        PS_Subscribe --> PS_Broker_Subscribe
        PS_Broker_Subscribe --> PS_Disconnect
        PS_Disconnect --> PS_Broker_Disconnect
        PS_Broker_Disconnect --> PS_Offline_Publish
        PS_Offline_Publish --> PS_Broker_Offline_Publish
        PS_Broker_Offline_Publish --> PS_Reconnect
        PS_Reconnect --> PS_Broker_Reconnect
    end
    
    classDef clientAction fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef brokerAction fill:#EDE9FE,stroke:#5B21B6,stroke-width:1px,color:#5B21B6;
    classDef messageFlow fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46;

    class CS_Connect,CS_Subscribe,CS_Disconnect,CS_Offline_Publish,CS_Reconnect clientAction;
    class CS_Broker_Connect,CS_Broker_Subscribe,CS_Broker_Disconnect,CS_Broker_Offline_Publish,CS_Broker_Reconnect brokerAction;
    
    class PS_Connect,PS_Subscribe,PS_Disconnect,PS_Offline_Publish,PS_Reconnect clientAction;
    class PS_Broker_Connect,PS_Broker_Subscribe,PS_Broker_Disconnect,PS_Broker_Offline_Publish,PS_Broker_Reconnect brokerAction;

Message Queuing by the Broker

When a client has a persistent session (CleanSession = 0) and is disconnected:

1. The broker maintains the client’s subscriptions.
2. If messages are published to topics the disconnected client is subscribed to with QoS 1 or QoS 2, the broker stores these messages.
3. When the client reconnects with CleanSession = 0 and the same ClientID, the broker sends these queued messages to the client.

Messages published with QoS 0 are generally not queued for offline clients, even with a persistent session, because QoS 0 does not guarantee delivery.

Client-Side Persistence

Client-side persistence refers to the MQTT client library’s ability to store outgoing QoS 1 and QoS 2 messages if the network connection to the broker is lost before they are acknowledged. The esp-mqtt client in ESP-IDF has an internal buffer (out_buffer_size in esp_mqtt_client_config_t) and will attempt to retransmit these messages upon reconnection, adhering to the respective QoS protocols.

Tip: For true end-to-end reliability, especially with QoS 1 and 2, ensure both the publishing client and the subscribing client correctly handle their parts of the protocol, and the broker is configured to support persistent sessions and message queuing if needed.

Practical Examples

Let’s see how to use different QoS levels and manage sessions with the esp-mqtt client in ESP-IDF. We assume you have a basic MQTT client setup as covered in Chapter 102.

1. Setting up the MQTT Client

The basic setup remains similar. The key aspects for QoS and sessions are within the esp_mqtt_client_config_t structure and the publish/subscribe API calls.

#include "esp_log.h"
#include "mqtt_client.h"
#include "esp_wifi.h" // For Wi-Fi connection events, if needed

static const char *TAG = "MQTT_QOS_EXAMPLE";

esp_mqtt_client_handle_t client = NULL;

static void log_error_if_nonzero(const char *message, int error_code)
{
    if (error_code != 0) {
        ESP_LOGE(TAG, "Last error %s: 0x%x", message, error_code);
    }
}

static esp_err_t mqtt_event_handler_cb(esp_mqtt_event_handle_t event)
{
    esp_mqtt_client_handle_t client = event->client;
    int msg_id;
    // your_context_t *context = event->context;
    switch (event->event_id) {
        case MQTT_EVENT_CONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
            // Example: Subscribe to a topic with QoS 1 upon connection
            msg_id = esp_mqtt_client_subscribe(client, "/topic/qos1_test", 1);
            ESP_LOGI(TAG, "sent subscribe successful, msg_id=%d", msg_id);

            msg_id = esp_mqtt_client_subscribe(client, "/topic/qos2_test", 2);
            ESP_LOGI(TAG, "sent subscribe successful for qos2, msg_id=%d", msg_id);
            break;
        case MQTT_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
            break;
        case MQTT_EVENT_SUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
            ESP_LOGI(TAG, "Subscribed to topic: %.*s", event->topic_len, event->topic);
            // Example: Publish a QoS 0 message after subscribing
            msg_id = esp_mqtt_client_publish(client, "/topic/qos0_test", "data_qos0", 0, 0, 0);
            ESP_LOGI(TAG, "sent publish successful, msg_id=%d", msg_id);
            break;
        case MQTT_EVENT_UNSUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_UNSUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_PUBLISHED:
            ESP_LOGI(TAG, "MQTT_EVENT_PUBLISHED, msg_id=%d", event->msg_id);
            // This event confirms PUBACK (for QoS 1) or PUBCOMP (for QoS 2) received from broker
            break;
        case MQTT_EVENT_DATA:
            ESP_LOGI(TAG, "MQTT_EVENT_DATA");
            ESP_LOGI(TAG, "TOPIC=%.*s", event->topic_len, event->topic);
            ESP_LOGI(TAG, "DATA=%.*s", event->data_len, event->data);
            ESP_LOGI(TAG, "QOS=%d, RETAIN=%d, MSG_ID=%d", event->qos, event->retain, event->msg_id);
            // If QoS 1 or 2, the library handles sending PUBACK or PUBREC/PUBREL internally
            break;
        case MQTT_EVENT_ERROR:
            ESP_LOGI(TAG, "MQTT_EVENT_ERROR");
            if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
                log_error_if_nonzero("reported from esp-tls", event->error_handle->esp_tls_last_esp_err);
                log_error_if_nonzero("reported from tls stack", event->error_handle->esp_tls_stack_err);
                log_error_if_nonzero("captured as transport's socket errno",  event->error_handle->esp_transport_sock_errno);
                ESP_LOGI(TAG, "Last errno string (%s)", strerror(event->error_handle->esp_transport_sock_errno));
            }
            break;
        default:
            ESP_LOGI(TAG, "Other event id:%d", event->event_id);
            break;
    }
    return ESP_OK;
}

static void mqtt_app_start(void)
{
    esp_mqtt_client_config_t mqtt_cfg = {
        .broker.address.uri = CONFIG_BROKER_URL, // Set this in menuconfig
        // .credentials.client_id = "esp32_client_qos_example", // Set if needed
        // For persistent session (CleanSession = 0 in MQTTv3.1.1)
        // .session.disable_clean_session = true, // Uncomment for persistent session
        // .session.last_will.topic = "/lwt/topic",
        // .session.last_will.msg = "offline",
        // .session.last_will.qos = 1,
        // .session.last_will.retain = 0,
    };

    client = esp_mqtt_client_init(&mqtt_cfg);
    esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler_cb, client);
    esp_mqtt_client_start(client);
}

// In your main_app function:
// Ensure Wi-Fi is connected first
// mqtt_app_start();

Configuration:

• CONFIG_BROKER_URL: Set your MQTT broker URL via idf.py menuconfig -> Component config -> ESP MQTT Client.
• disable_clean_session: In esp_mqtt_client_config_t, setting session.disable_clean_session = true; means CleanSession = 0 (persistent session). By default, it’s false (CleanSession = 1).
  • Warning: The field name disable_clean_session can be slightly confusing. true means “disable the clean session feature”, which translates to MQTT CleanSession = 0 (persistent). false means “do not disable clean session”, which translates to MQTT CleanSession = 1 (not persistent).

2. Publishing Messages with Different QoS Levels

The esp_mqtt_client_publish() function allows specifying the QoS level:

int esp_mqtt_client_publish(esp_mqtt_client_handle_t client, const char *topic, const char *data, int len, int qos, int retain)

• qos: 0, 1, or 2.

// Inside a function, assuming 'client' is initialized and connected.

// Publish with QoS 0
int msg_id_qos0 = esp_mqtt_client_publish(client, "/sensor/temp", "25.5_qos0", 0, 0, 0);
if (msg_id_qos0 != -1) {
    ESP_LOGI(TAG, "Sent QoS 0 publish successful, msg_id=%d", msg_id_qos0);
} else {
    ESP_LOGE(TAG, "Failed to send QoS 0 publish");
}

// Publish with QoS 1
// For QoS 1 and 2, msg_id will be > 0 if successfully enqueued.
// The MQTT_EVENT_PUBLISHED event will later confirm broker acknowledgment.
int msg_id_qos1 = esp_mqtt_client_publish(client, "/sensor/humidity", "60_qos1", 0, 1, 0);
if (msg_id_qos1 != -1) {
    ESP_LOGI(TAG, "Sent QoS 1 publish successful, msg_id=%d", msg_id_qos1);
     // The library will handle retransmissions if PUBACK is not received.
} else {
    ESP_LOGE(TAG, "Failed to send QoS 1 publish, possibly outbox full");
}

// Publish with QoS 2
int msg_id_qos2 = esp_mqtt_client_publish(client, "/actuator/control", "ON_qos2", 0, 2, 0);
if (msg_id_qos2 != -1) {
    ESP_LOGI(TAG, "Sent QoS 2 publish successful, msg_id=%d", msg_id_qos2);
    // The library handles the PUBREC/PUBREL/PUBCOMP handshake.
    // MQTT_EVENT_PUBLISHED confirms PUBCOMP.
} else {
    ESP_LOGE(TAG, "Failed to send QoS 2 publish, possibly outbox full");
}

Observation:

• QoS 0: esp_mqtt_client_publish returns a msg_id (can be 0 or positive if successful, -1 on error like outbox full). No MQTT_EVENT_PUBLISHED is generated for QoS 0 publishes as there’s no broker acknowledgment.
• QoS 1 & 2: esp_mqtt_client_publish returns a positive msg_id if the message is successfully enqueued for sending. The actual confirmation of delivery to the broker comes via the MQTT_EVENT_PUBLISHED event, which carries the same msg_id. This event signifies that a PUBACK (for QoS 1) or PUBCOMP (for QoS 2) has been received from the broker.

3. Subscribing with Different QoS Levels

The esp_mqtt_client_subscribe() function also allows specifying the maximum QoS level the client wishes to receive messages on for that topic:

int esp_mqtt_client_subscribe(esp_mqtt_client_handle_t client, const char *topic, int qos)

• qos: 0, 1, or 2. This is the maximum QoS the client requests. The broker will send messages at this QoS or lower, depending on the QoS of the published message.

// Inside MQTT_EVENT_CONNECTED handler or similar logic:

int sub_msg_id_qos0 = esp_mqtt_client_subscribe(client, "/notifications/info", 0);
if (sub_msg_id_qos0 != -1) {
    ESP_LOGI(TAG, "Sent subscribe to /notifications/info (QoS 0) successful, msg_id=%d", sub_msg_id_qos0);
} else {
    ESP_LOGE(TAG, "Failed to send subscribe to /notifications/info (QoS 0)");
}

int sub_msg_id_qos1 = esp_mqtt_client_subscribe(client, "/commands/device1", 1);
if (sub_msg_id_qos1 != -1) {
    ESP_LOGI(TAG, "Sent subscribe to /commands/device1 (QoS 1) successful, msg_id=%d", sub_msg_id_qos1);
} else {
    ESP_LOGE(TAG, "Failed to send subscribe to /commands/device1 (QoS 1)");
}

int sub_msg_id_qos2 = esp_mqtt_client_subscribe(client, "/critical/alerts", 2);
if (sub_msg_id_qos2 != -1) {
    ESP_LOGI(TAG, "Sent subscribe to /critical/alerts (QoS 2) successful, msg_id=%d", sub_msg_id_qos2);
} else {
    ESP_LOGE(TAG, "Failed to send subscribe to /critical/alerts (QoS 2)");
}

The MQTT_EVENT_SUBSCRIBED event confirms that the SUBACK was received from the broker, indicating the subscription was successful and also reports the granted QoS by the broker for that subscription. The MQTT_EVENT_DATA event for an incoming message will indicate the QoS level at which that particular message was delivered.

4. Persistent Sessions (CleanSession = 0)

To use a persistent session:

1. Set a unique client_id in esp_mqtt_client_config_t. This is crucial for the broker to identify the session.// In esp_mqtt_client_config_t .credentials.client_id = "my-persistent-esp32",
2. Set disable_clean_session to true in esp_mqtt_client_config_t.// In esp_mqtt_client_config_t .session.disable_clean_session = true,

Workflow for testing persistent sessions:

1. Client A (ESP32): Connects with ClientID="my-persistent-esp32" and disable_clean_session = true. Subscribes to /topic/persistent_test with QoS 1.
2. Disconnect Client A: Power off or disconnect the ESP32.
3. Client B (e.g., MQTT Explorer, mosquitto_pub): Publish a message to /topic/persistent_test with QoS 1.mosquitto_pub -h your_broker_ip -t /topic/persistent_test -m "Hello offline ESP32" -q 1
4. Reconnect Client A: Power on the ESP32. It should reconnect with the same ClientID and disable_clean_session = true.
5. Observe: Client A should receive the message “Hello offline ESP32” that was published while it was disconnected. This message was queued by the broker due to the persistent session.

Tip: Ensure your MQTT broker supports and is configured to allow persistent sessions and message queuing. Most brokers like Mosquitto, HiveMQ, etc., support this by default. Check broker documentation for limits on queued messages or session expiry.

Build and Run Instructions

1. Configure:
   • Set up your Wi-Fi credentials and MQTT broker URL (CONFIG_BROKER_URL) using idf.py menuconfig.
   • (Optional) Adjust client_id and disable_clean_session as needed for your tests.
2. Build:idf.py build
3. Flash:idf.py -p /dev/YOUR_SERIAL_PORT flash
(Replace /dev/YOUR_SERIAL_PORT with your ESP32’s serial port, e.g., /dev/ttyUSB0 on Linux, COM3 on Windows)
4. Monitor:idf.py -p /dev/YOUR_SERIAL_PORT monitor
5. Observe:
   • Watch the serial monitor for ESP32 logs (MQTT_EVENT_...).
   • Use an external MQTT client (like MQTT Explorer or mosquitto_sub/mosquitto_pub) connected to the same broker to publish messages to topics the ESP32 is subscribed to, and subscribe to topics the ESP32 is publishing to.
   • Observe your MQTT broker’s logs if possible. This can provide valuable insight into the message flows and acknowledgments (PUBACK, PUBREC, etc.). For Mosquitto, running it with mosquitto -v provides verbose logging.

Variant Notes

The MQTT protocol and the esp-mqtt client library behavior regarding QoS and sessions are generally consistent across all ESP32 variants (ESP32, ESP32-S2, ESP32-S3, ESP32-C3, ESP32-C6, ESP32-H2). However, resource constraints can play a role:

• RAM:
  • QoS 1 and QoS 2 require the client to store messages until they are acknowledged (PUBACK or PUBCOMP). If the ESP32 is publishing many high-QoS messages while disconnected or experiencing network latency, its internal buffers (buffer_size and out_buffer_size in esp_mqtt_client_config_t) can fill up. Variants with more RAM (e.g., ESP32-S3 with PSRAM) might handle larger buffers or more concurrent complex operations more gracefully.
  • The default buffer_size (for incoming messages) and out_buffer_size (for outgoing messages waiting for ACK) are typically 1024 bytes each. If you are sending large messages or expect many messages to be queued, you might need to increase these, keeping an eye on available heap.
• Processing Power:
  • QoS 2 involves more complex handshake logic. While esp-mqtt handles this, the increased processing could be a minor factor on less powerful variants if combined with other intensive tasks, though generally, MQTT operations are not CPU-bound for typical IoT use cases.
  • If using TLS for secure MQTT (covered in a later chapter), the cryptographic operations can be CPU-intensive. Variants with hardware cryptographic acceleration (most ESP32 variants have this to some degree) will perform better.
• Flash Memory: The esp-mqtt library and its dependencies will consume some flash memory. This is usually not a limiting factor unless the application is extremely large and the variant has very limited flash (e.g., some early ESP32-C3 modules).
• ESP32-H2: While the ESP32-H2 supports Wi-Fi through coexistence with an ESP32 companion chip or by using an external Wi-Fi module connected via SPI/SDIO, its primary radio is 802.15.4 (Thread, Zigbee). If MQTT is used over Wi-Fi with ESP32-H2, the considerations are similar to other Wi-Fi enabled ESP32s. If MQTT-SN (a variant for sensor networks) were used over 802.15.4, different considerations would apply, but standard MQTT typically implies TCP/IP over Wi-Fi or Ethernet.

In summary, the choice of ESP32 variant is less about fundamental MQTT QoS/session feature compatibility and more about the overall resource demands of your application (including other tasks, TLS, memory for application logic, etc.). Always monitor heap usage and task performance.

Common Mistakes & Troubleshooting Tips

Exercises

1. QoS Level Observation:
   • Modify the example code to publish three distinct messages: one with QoS 0, one with QoS 1, and one with QoS 2, each to a different topic (e.g., /test/qos0, /test/qos1, /test/qos2).
   • Subscribe to these topics using an external MQTT client (like MQTT Explorer or mosquitto_sub).
   • Observe the ESP32’s serial monitor logs, specifically noting the msg_id returned by esp_mqtt_client_publish() and the occurrence (or absence for QoS 0) of MQTT_EVENT_PUBLISHED.
   • If your broker supports verbose logging (e.g., mosquitto -v), observe the MQTT control packets (PUBACK, PUBREC, PUBREL, PUBCOMP) exchanged.
2. Persistent Session Test:
   • Configure your ESP32 MQTT client with a unique ClientID and disable_clean_session = true.
   • In the MQTT_EVENT_CONNECTED handler, make the ESP32 subscribe to a specific topic (e.g., /mydevice/commands) with QoS 1.
   • Run the ESP32 client and let it connect and subscribe.
   • Disconnect/power off the ESP32.
   • Using another MQTT client, publish a message with QoS 1 to /mydevice/commands.
   • Reconnect/power on the ESP32.
   • Verify that the ESP32 receives the message that was published while it was offline. Log the received message.
3. Simulate QoS 1 Retry (Advanced):
   • This is more challenging to perfectly simulate without network manipulation tools, but you can approximate it.
   • Set up the ESP32 to publish a QoS 1 message.
   • Temporarily disconnect the Wi-Fi or stop the MQTT broker immediately after the ESP32 logs that it has sent the publish request but before it logs MQTT_EVENT_PUBLISHED. (This might require quick action or a programmatic delay/disconnect).
   • Observe the ESP32’s behavior when connectivity is restored. The esp-mqtt client should attempt to resend the unacknowledged QoS 1 message. You might see the same message published again (possibly with the DUP flag, though the library handles this internally).
   • Alternatively: If your broker allows, you could temporarily block traffic from the ESP32’s IP after it sends the PUBLISH but before the broker sends PUBACK, then unblock.

Summary

• QoS 0 (At most once): Fastest, lowest overhead, but messages can be lost. No acknowledgment.
• QoS 1 (At least once): Guarantees delivery, but duplicates are possible. Uses PUBLISH <-> PUBACK handshake.
• QoS 2 (Exactly once): Highest reliability, no loss or duplicates. Uses a four-part handshake (PUBLISH <-> PUBREC <-> PUBREL <-> PUBCOMP).
• esp-mqtt Client: Handles the complexities of QoS handshakes and retransmissions internally. MQTT_EVENT_PUBLISHED confirms broker acknowledgment for QoS 1 and 2.
• Clean Session:
  • CleanSession = 1 (default, disable_clean_session = false): Session is discarded on disconnect. No offline message queuing for the client.
  • CleanSession = 0 (disable_clean_session = true): Session persists. Broker queues QoS 1 & 2 messages for subscribed topics if the client is offline. Requires a stable ClientID.
• Message Persistence: Relies on persistent sessions and the broker’s capability to store messages for offline clients.
• Resource Impact: Higher QoS levels and persistent sessions (especially with many queued messages) can increase network traffic, latency, and memory usage on both client and broker.

Further Reading

• ESP-IDF MQTT Client Documentation:
  • ESP-MQTT Client Documentation (Always refer to the version matching your ESP-IDF setup)
• MQTT Version 3.1.1 Specification:
  • OASIS MQTT Version 3.1.1 Spec (Sections on QoS and Session state are particularly relevant)
• MQTT Version 5.0 Specification:
  • OASIS MQTT Version 5.0 Spec (For understanding CleanStart and SessionExpiryInterval if using MQTTv5 features)
• HiveMQ MQTT Essentials Series:
  • Understanding MQTT QoS Levels
  • MQTT Persistent Session and Queued Messages