Chapter 101: MQTT Protocol Fundamentals

Chapter Objectives

After completing this chapter, you will be able to:

• Understand the core concepts and architecture of the MQTT protocol.
• Explain the publish/subscribe messaging model and its benefits.
• Identify the key components of an MQTT system: Broker and Client.
• Understand MQTT topics, their hierarchical structure, and the use of wildcards.
• Describe the different Quality of Service (QoS) levels and their implications.
• Recognize the advantages of using MQTT for Internet of Things (IoT) applications.
• Understand basic MQTT features like Retain Flag, Keep Alive, and Last Will and Testament (LWT).

Introduction

Welcome to the world of IoT protocols! As we build more sophisticated ESP32 applications that interact with the cloud and other devices, the need for efficient, reliable, and scalable communication mechanisms becomes paramount. One of the most widely adopted messaging protocols in the IoT landscape is MQTT (Message Queuing Telemetry Transport).

MQTT is a lightweight, publish-subscribe network protocol that transports messages between devices. It is designed for constrained devices and low-bandwidth, high-latency, or unreliable networks, making it an ideal choice for many IoT and mobile applications. Originally developed by IBM in the late 1990s to monitor oil pipelines over satellite connections, its efficiency and simplicity have led to its widespread use in everything from home automation to industrial control systems.

In this chapter, we will delve into the fundamental principles of MQTT. Understanding these basics is crucial before we move on to implementing MQTT clients on your ESP32 devices in subsequent chapters.

Theory

What is MQTT?

MQTT stands for Message Queuing Telemetry Transport. It is an OASIS standard messaging protocol built on top of TCP/IP (or other network protocols that provide ordered, lossless, bi-directional connections). Its primary design goals were to be lightweight, efficient, and reliable, especially in environments where network bandwidth is at a premium and devices have limited processing power and memory – common characteristics of IoT devices like the ESP32.

Core Concepts

MQTT’s architecture is based on a few simple but powerful concepts:

Publish/Subscribe Model

Unlike traditional client-server models where clients communicate directly with a server, MQTT uses a publish/subscribe (pub/sub) pattern. This model decouples message senders (publishers) from message receivers (subscribers).

• Publishers: Clients that send messages. They don’t need to know who the subscribers are. They simply send messages to a specific “topic” on an MQTT broker.

• Subscribers: Clients that receive messages. They don’t need to know who the publishers are. They express interest in one or more topics by subscribing to them on the MQTT broker.

• Decoupling: This decoupling in space (publisher and subscriber don’t need to know each other’s IP address/port), time (they don’t need to be running at the same time, depending on configuration), and synchronization (operations on both sides don’t halt each other) is a key strength of MQTT.

%%{init: {"theme": "base", "themeVariables": { "fontFamily": "Open Sans"}}}%%
graph TD
    subgraph "MQTT System"
        direction LR
        Broker["<b>MQTT Broker</b><br>(Central Hub)"]
        style Broker fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#1E40AF

        subgraph "Message Flow"
            direction TB
            Publisher1[/"<b>Publisher A</b><br>(e.g., ESP32 Sensor)"/] -- "PUBLISH (Topic: 'sensor/temp')" --> Broker
            Publisher2[/"<b>Publisher B</b><br>(e.g., ESP32 Button)"/] -- "PUBLISH (Topic: 'control/light')" --> Broker

            Broker -- "Message on 'sensor/temp'" --> Subscriber1[/"<b>Subscriber X</b><br>(e.g., Mobile App)"/]
            Broker -- "Message on 'control/light'" --> Subscriber2[/"<b>Subscriber Y</b><br>(e.g., ESP32 Actuator)"/]
            Broker -- "Message on 'sensor/temp'" --> Subscriber3[/"<b>Subscriber Z</b><br>(e.g., Data Logger)"/]
        end
    end

    Publisher1 --> P1Topic(("Topic: 'sensor/temp'"))
    P1Topic -.-> Broker
    Publisher2 --> P2Topic(("Topic: 'control/light'"))
    P2Topic -.-> Broker

    Broker -.-> S1Topic(("Subscribed to: 'sensor/temp'"))
    S1Topic --> Subscriber1
    Broker -.-> S2Topic(("Subscribed to: 'control/light'"))
    S2Topic --> Subscriber2
    Broker -.-> S3Topic(("Subscribed to: 'sensor/temp'"))
    S3Topic --> Subscriber3

    classDef publisher fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6
    classDef subscriber fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46
    classDef topic fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E

    class Publisher1,Publisher2 publisher
    class Subscriber1,Subscriber2,Subscriber3 subscriber
    class P1Topic,P2Topic,S1Topic,S2Topic,S3Topic topic

    linkStyle 0 stroke:#5B21B6,stroke-width:1.5px
    linkStyle 1 stroke:#5B21B6,stroke-width:1.5px
    linkStyle 2 stroke:#059669,stroke-width:1.5px
    linkStyle 3 stroke:#059669,stroke-width:1.5px
    linkStyle 4 stroke:#059669,stroke-width:1.5px
    linkStyle 5 stroke-dasharray: 5 5, stroke:#D97706,stroke-width:1px
    linkStyle 6 stroke-dasharray: 5 5, stroke:#D97706,stroke-width:1px
    linkStyle 7 stroke-dasharray: 5 5, stroke:#D97706,stroke-width:1px
    linkStyle 8 stroke-dasharray: 5 5, stroke:#D97706,stroke-width:1px
    linkStyle 9 stroke-dasharray: 5 5, stroke:#D97706,stroke-width:1px
    linkStyle 10 stroke-dasharray: 5 5, stroke:#D97706,stroke-width:1px

Broker (Server)

The MQTT broker is the central hub in an MQTT system. It’s a server that receives all messages from publishing clients and then routes these messages to the appropriate subscribing clients.

Responsibilities:

• Receiving messages from publishers.
• Filtering messages based on topics.
• Forwarding messages to subscribers interested in those topics.
• Handling client connections, disconnections, and security (authentication/authorization).
• Optionally, persisting messages (e.g., retained messages, messages for offline clients with persistent sessions).

Examples of Brokers: Popular open-source brokers include Mosquitto, EMQ X, and VerneMQ. Cloud providers also offer managed MQTT broker services, such as AWS IoT Core, Azure IoT Hub, and Google Cloud IoT Core.

Client

An MQTT client is any device or application that connects to an MQTT broker over a network.

• A client can be a publisher, sending data (e.g., an ESP32 with a sensor publishing temperature readings).
• A client can be a subscriber, receiving data (e.g., an ESP32 controlling an actuator based on received commands, or a mobile app displaying sensor data).
• A client can also be both a publisher and a subscriber simultaneously.
• Each client is uniquely identified by a ClientID when it connects to the broker.

MQTT Communication Flow

The interaction between clients and a broker typically follows these steps:

1. CONNECT: The client establishes a TCP/IP connection with the broker and sends a CONNECT packet. This packet includes the ClientID, Clean Session flag, Keep Alive interval, and optionally, username/password and Last Will and Testament (LWT) information.
2. CONNACK: The broker responds with a CONNACK (Connect Acknowledgment) packet, indicating whether the connection was successful.
3. SUBSCRIBE (for subscribers): If the client wants to receive messages, it sends a SUBSCRIBE packet to the broker. This packet lists the topics the client is interested in and the desired Quality of Service (QoS) level for each.
4. SUBACK: The broker responds with a SUBACK (Subscribe Acknowledgment) packet, confirming the subscriptions and the QoS level granted.
5. PUBLISH (for publishers/broker):
   • When a publishing client has data to send, it sends a PUBLISH packet to the broker. This packet contains the topic name, the message payload, the QoS level, and a retain flag.
   • The broker then forwards this PUBLISH packet to all clients that have a matching subscription for that topic and meet the QoS requirements.
6. Acknowledgment (for QoS > 0): Depending on the QoS level, there might be further acknowledgment packets exchanged (e.g., PUBACK, PUBREC, PUBREL, PUBCOMP).
7. UNSUBSCRIBE: A client can send an UNSUBSCRIBE packet to remove its interest in specific topics.
8. UNSUBACK: The broker confirms with an UNSUBACK.
9. DISCONNECT: When a client wants to terminate its connection gracefully, it sends a DISCONNECT packet to the broker. The TCP/IP connection is then closed.

%%{init: {"theme": "base", "themeVariables": { "fontFamily": "Open Sans"}}}%%
sequenceDiagram
    actor ClientA as Client A (Publisher)
    participant Broker
    actor ClientB as Client B (Subscriber)

    ClientA->>Broker: 1. CONNECT (ClientID: PubA, LWT, etc.)
    activate Broker
    Broker-->>ClientA: 2. CONNACK (Connection Accepted)
    deactivate Broker
    Note over ClientA, Broker: Client A is now connected

    ClientB->>Broker: 3. CONNECT (ClientID: SubB, CleanSession: false)
    activate Broker
    Broker-->>ClientB: 4. CONNACK (Connection Accepted)
    deactivate Broker
    Note over ClientB, Broker: Client B is now connected

    ClientB->>Broker: 5. SUBSCRIBE (Topic: "sensor/data/#", QoS: 1)
    activate Broker
    Broker-->>ClientB: 6. SUBACK (Subscription Accepted, QoS: 1)
    deactivate Broker
    Note over ClientB, Broker: Client B is subscribed to "sensor/data/#"

    ClientA->>Broker: 7. PUBLISH (Topic: "sensor/data/temp", Payload: "25C", QoS: 1, Retain: false)
    activate Broker
    Broker-->>ClientA: 8. PUBACK (QoS 1 Acknowledgment)
    Note over Broker: Broker receives message from Client A

    Broker->>ClientB: 9. PUBLISH (Topic: "sensor/data/temp", Payload: "25C", QoS: 1)
    activate ClientB
    Note over Broker, ClientB: Broker forwards message to subscribed Client B
    ClientB-->>Broker: 10. PUBACK (QoS 1 Acknowledgment)
    deactivate ClientB
    deactivate Broker

    ClientA->>Broker: 11. DISCONNECT
    activate Broker
    Note over ClientA, Broker: Client A disconnects gracefully
    deactivate Broker

    ClientB->>Broker: 12. UNSUBSCRIBE (Topic: "sensor/data/#")
    activate Broker
    Broker-->>ClientB: 13. UNSUBACK
    deactivate Broker
    Note over ClientB, Broker: Client B unsubscribes

    ClientB->>Broker: 14. DISCONNECT
    activate Broker
    Note over ClientB, Broker: Client B disconnects gracefully
    deactivate Broker

Topics

Topics are the “addresses” to which messages are published. They are UTF-8 strings that are structured hierarchically using forward slashes (/) as level separators. Think of them like file paths in a filesystem.

• Structure: e.g., home/livingroom/temperature, factory/machineA/status/alerts
• Case-sensitivity: Topics are case-sensitive. home/LivingRoom/temperature is different from home/livingroom/temperature.
• Purpose: Brokers use topics to filter and route messages from publishers to subscribers.

Wildcards (for Subscriptions):

Subscribers can use wildcards in topic filters to subscribe to multiple topics at once. Publishers cannot use wildcards in the topics they publish to.

1. Single-Level Wildcard (+):
   • The plus sign (+) matches exactly one topic level.
   • Example: home/+/temperature would match:
     • home/livingroom/temperature
     • home/kitchen/temperature
     • But NOT home/temperature or home/livingroom/main/temperature.
2. Multi-Level Wildcard (#):
   • The hash sign (#) matches multiple topic levels at the end of a topic filter. It must be the last character in the topic filter.
   • Example: home/livingroom/# would match:
     • home/livingroom/temperature
     • home/livingroom/light/brightness
     • home/livingroom (if messages are published directly to it)
   • Example: # by itself subscribes to all topics (use with caution on busy brokers, as it can overwhelm a client).

System Topics ($):

Topics beginning with a $ character (e.g., $SYS/…) are typically reserved for broker-specific statistics, internal information, or administrative messages. Clients usually cannot publish messages to topics starting with $. For example, $SYS/broker/clients/connected might provide the number of currently connected clients.

Quality of Service (QoS)

MQTT defines three Quality of Service levels, which determine the guarantee of message delivery between a sender (publisher or broker) and a receiver (broker or subscriber).

1. QoS 0 (At most once):
   • Description: The message is delivered at most once, or not at all. It’s a “fire and forget” mechanism.
   • Handshake: PUBLISH ->
   • Guarantee: No guarantee of delivery. The message is sent, but there’s no acknowledgment from the receiver.
   • Pros: Lowest network overhead, fastest transmission.
   • Cons: Messages can be lost if the network is unreliable or the receiver is unavailable.
   • Use Case: Suitable for non-critical sensor data where occasional loss is acceptable, or when data is sent very frequently and the latest value is what matters most.
2. QoS 1 (At least once):
   • Description: The message is guaranteed to be delivered at least once.
   • Handshake: PUBLISH -> <- PUBACK
   • Guarantee: The sender (publisher or broker) continues to resend the PUBLISH message (with a DUP flag set) at intervals until it receives a PUBACK (Publish Acknowledgment) from the receiver.
   • Pros: Ensures message delivery.
   • Cons: Higher overhead than QoS 0. It’s possible for the receiver to get the same message multiple times if an acknowledgment is lost and the sender retransmits. The receiving application must be prepared to handle potential duplicates (e.g., by checking message IDs if idempotency is required).
   • Use Case: Suitable for commands or status updates where delivery is important, and the application can tolerate or manage duplicates.
3. QoS 2 (Exactly once):
   • Description: The message is guaranteed to be delivered exactly once. This is the most reliable but also the most complex QoS level.
   • Handshake:PUBLISH -> <- PUBREC -> PUBREL -> <- PUBCOMP
     1. Sender sends PUBLISH.
     2. Receiver stores the message and sends PUBREC (Publish Received).
     3. Sender receives PUBREC, discards the original PUBLISH, stores the PUBREC, and sends PUBREL (Publish Release).
     4. Receiver receives PUBREL, processes the message, discards its stored state, and sends PUBCOMP (Publish Complete).
     5. Sender receives PUBCOMP and discards its stored state.
   • Guarantee: Ensures the message is delivered once and only once.
   • Pros: Highest level of reliability.
   • Cons: Highest network overhead due to the four-part handshake. Can introduce more latency.
   • Use Case: Suitable for critical messages where message loss or duplication is unacceptable (e.g., billing information, critical control commands).

sequenceDiagram
    participant P as Publisher
    participant B as Broker
    participant S as Subscriber

    Note over P,S: QoS 0 - At Most Once (Fire and Forget)
    P->>B: PUBLISH (QoS 0)
    B->>S: PUBLISH (QoS 0)
    Note over P,S: No acknowledgment - message may be lost

    Note over P,S: 
    Note over P,S: QoS 1 - At Least Once
    P->>B: PUBLISH (QoS 1, Message ID)
    B->>P: PUBACK (Message ID)
    B->>S: PUBLISH (QoS 1, Message ID)
    S->>B: PUBACK (Message ID)
    Note over P,S: Acknowledged delivery - possible duplicates

    Note over P,S: 
    Note over P,S: QoS 2 - Exactly Once (4-step handshake)
    P->>B: PUBLISH (QoS 2, Message ID)
    B->>P: PUBREC (Message ID)
    P->>B: PUBREL (Message ID)
    B->>P: PUBCOMP (Message ID)
    B->>S: PUBLISH (QoS 2, Message ID)
    S->>B: PUBREC (Message ID)
    B->>S: PUBREL (Message ID)
    S->>B: PUBCOMP (Message ID)
    Note over P,S: Guaranteed exactly once delivery

Tip: The QoS level for a message flow is determined by the lower of the QoS levels specified by the publisher and the subscriber. For example, if a message is published with QoS 2, but a client subscribes with QoS 1, the broker will deliver that message to that specific client with QoS 1 semantics.

Message Structure (PUBLISH packet)

A PUBLISH packet, which carries the application data, contains several important fields:

• Topic Name: The UTF-8 string identifying the topic to which the message is published.
• Payload: The actual message content. MQTT treats the payload as a binary blob, meaning it can be any data format (e.g., plain text, JSON, XML, Protocol Buffers, or custom binary data). The maximum payload size is 256MB, but in practice, IoT messages are usually much smaller (a few bytes to a few kilobytes).
• QoS Level: (0, 1, or 2) Specifies the Quality of Service for this message.
• Retain Flag (R):
  • A boolean flag. If set to true, the broker must store this message (and its QoS) as the “last known good” value for this specific topic.
  • When a new client subscribes to a topic, if there’s a retained message for that topic, the broker will immediately send that message to the new subscriber.
  • Only one message can be retained per topic. A new retained message on a topic replaces any existing one.
  • To clear a retained message, a publisher can send a message with an empty payload and the retain flag set to true on that topic.
  • Use Case: Useful for providing the latest status or initial state to newly connected clients (e.g., the current setting of a thermostat, the last reported state of a light).
• Duplicate Flag (DUP):
  • A boolean flag. Set by the sender (publisher or broker) when it is re-transmitting a PUBLISH packet for a QoS 1 or QoS 2 message because it has not received the expected acknowledgment.
  • The receiver can use this flag to detect potential duplicates, although the primary mechanism for handling duplicates in QoS 1 and exactly-once delivery in QoS 2 relies on packet identifiers.

Other Important MQTT Features

• Last Will and Testament (LWT):
  • A powerful feature for presence detection or handling unexpected client disconnections.
  • When a client connects to a broker, it can specify an LWT message, an LWT topic, LWT QoS, and LWT retain flag.
  • If the client disconnects ungracefully (e.g., due to a network failure, power outage, or software crash, without sending a DISCONNECT packet), the broker will automatically publish the LWT message to the LWT topic on behalf of the disconnected client.
  • If the client disconnects gracefully (sends DISCONNECT), the LWT message is discarded by the broker.
  • Use Case: Alerting other systems that a device has gone offline (e.g., publishing {"status": "offline"} to devices/device123/status).

%%{init: {"theme": "base", "themeVariables": { "fontFamily": "Open Sans"}}}%%
graph TD
    A[/"MQTT Client (e.g., ESP32)"/] -- "1- CONNECT Request" --> B{MQTT Broker};
    subgraph "CONNECT Packet Details"
        direction LR
        CID["ClientID: 'device123'"]
        LWT_Topic["LWT Topic: 'devices/device123/status'"]
        LWT_Msg["LWT Message: '{\<i>state\</i>: \<i>offline\</i>}'"]
        LWT_QoS["LWT QoS: 1"]
        LWT_Retain["LWT Retain: true"]
    end
    A -- "Includes LWT Info" --> LWT_Topic;

    B -- "2- Connection Established<br>(Broker stores LWT if provided)" --> A;

    subgraph "Scenario 1: Graceful Disconnect"
        direction TB
        A_Graceful[/"Client (device123)"/] -- "3a- DISCONNECT Packet" --> B_Graceful{Broker};
        B_Graceful -- "4a- LWT Discarded by Broker" --> X1[("LWT Not Published")];
    end

    subgraph "Scenario 2: Ungraceful Disconnect"
        direction TB
        A_Ungraceful[/"Client (device123)"/] -. "Network Failure / Crash" .-> B_Ungraceful{Broker}
        style A_Ungraceful fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B
        B_Ungraceful -- "3b- Detects Unresponsive Client<br>(e.g., Keep Alive Timeout)" --> C_PublishLWT{"Broker Publishes LWT"};
        C_PublishLWT -- "4b- PUBLISH LWT Message<br>To: 'devices/device123/status'<br>Payload: '{\<i>state\</i>: \<i>offline\</i>}'<br>QoS: 1, Retain: true" --> D[/"Other Subscribed Clients<br>(e.g., Monitoring System)"/];
    end

    classDef client fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef broker fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef lwtinfo fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef success fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;
    classDef failure fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef subscriber fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46

    class A client;
    class A_Graceful client;
    class B broker;
    class B_Graceful broker;
    class B_Ungraceful broker;
    class C_PublishLWT broker;
    class LWT_Topic lwtinfo;
    class LWT_Msg lwtinfo;
    class LWT_QoS lwtinfo;
    class LWT_Retain lwtinfo;
    class X1 success;
    class D subscriber;
    class A_Ungraceful failure;

• Keep Alive:
  • An interval (in seconds) defined by the client during the CONNECT phase. It’s the maximum time that can elapse without the client or broker sending a control packet.
  • If the client doesn’t send any messages within this interval, it must send a PINGREQ packet to the broker. The broker must respond with a PINGRESP packet.
  • Purpose:
    • Allows the client and broker to verify that the network connection is still active.
    • Helps the broker detect if a client has disconnected ungracefully (e.g., if the broker doesn’t receive any communication, including PINGREQ, from the client within approximately 1.5 times the Keep Alive interval, it can close the connection and publish the LWT if configured).

%%{init: {"theme": "base", "themeVariables": { "fontFamily": "Open Sans"}}}%%
sequenceDiagram
    actor Client
    participant Broker

    Client->>Broker: CONNECT (KeepAlive: 60s)
    activate Broker
    Broker-->>Client: CONNACK
    deactivate Broker
    Note over Client,Broker: Connection established with KeepAlive interval of 60s.

    loop Within KeepAlive Interval (e.g., < 60s)
        alt Client sends other MQTT packets (PUBLISH, SUBSCRIBE, etc.)
            Client->>Broker: MQTT Control Packet (e.g., PUBLISH)
            activate Broker
            Note over Client,Broker: Activity resets KeepAlive timer on both sides.
            Broker-->>Client: (Response if any, e.g., PUBACK)
            deactivate Broker
        else No other packets sent by Client
            Note over Client: KeepAlive interval approaching, no data to send.
            Client->>Broker: PINGREQ
            activate Broker
            Note over Client,Broker: Client sends PINGREQ to keep connection alive.
            Broker-->>Client: PINGRESP
            deactivate Broker
            Note over Client,Broker: Broker responds. Connection confirmed active.
        end
    end

    alt Client becomes unresponsive (e.g., > 1.5 * KeepAlive interval)
        Note over Broker: Broker does not receive any packets<br>(Control or PINGREQ) from Client<br>within approx. 1.5 x KeepAlive interval.
        Broker--xClient: Connection Timeout
        Note over Broker: Broker considers Client disconnected.<br>Closes connection. Publishes LWT if configured.
    end

• Clean Session / Persistent Session:This is controlled by the Clean Session flag in the CONNECT packet.
  • Clean Session (true):
    • When the client connects, the broker discards any previous session information for that ClientID.
    • When the client disconnects, all its subscriptions are removed, and any QoS 1 or QoS 2 messages that were queued for it (if it was temporarily offline) are discarded by the broker.
    • This is like starting fresh with each connection.
  • Persistent Session (Clean Session = false):
    • When the client connects with Clean Session = false and a specific ClientID:
      • If a session already exists for this ClientID, the broker resumes that session.
      • The broker must store the client’s subscriptions even after it disconnects.
      • If messages with QoS 1 or QoS 2 are published to topics the client is subscribed to while it’s offline, the broker must queue these messages (up to broker-defined limits).
      • When the client reconnects (with Clean Session = false and the same ClientID), the broker delivers the queued messages and its previous subscriptions are still active.
    • Use Case: Essential for clients on unreliable networks that need to receive all important messages even if they are temporarily disconnected.

• Client ID (ClientID):
  • A UTF-8 string that uniquely identifies the client to the broker. Each client connecting to a broker must have a unique ClientID.
  • The broker uses the ClientID to maintain session state, especially for persistent sessions.
  • If a client connects with a ClientID that is already in use by another active client, the broker will typically disconnect the older client.

Benefits of MQTT for IoT

• Lightweight and Efficient: Small message headers (as little as 2 bytes for the fixed header) minimize network bandwidth usage.
• Low Power Consumption: Reduced network traffic and simpler processing requirements help conserve battery life on constrained devices.
• Handles Unreliable Networks: QoS levels and persistent sessions allow for reliable message delivery even over flaky connections.
• Decoupled Architecture: Publishers and subscribers are independent, leading to more flexible and scalable systems.
• Scalability: Brokers can be designed to handle tens of thousands to millions of concurrently connected clients.
• Bidirectional Communication: Clients can both publish and subscribe, enabling two-way communication.
• Language and Platform Agnostic: Many client libraries are available for various programming languages and platforms, including C/C++ for ESP32.
• Strong Community and Industry Support: Widely adopted and well-documented.

Practical Examples (Conceptual)

Since this chapter focuses on MQTT fundamentals, we won’t be writing ESP32 code just yet. That will come in Chapter 102. For now, let’s illustrate the concepts using scenarios you might encounter and how they map to MQTT. You can experiment with these using command-line tools like mosquitto_pub and mosquitto_sub (if you have a Mosquitto broker installed) or a graphical tool like MQTT Explorer.

Assume you have an MQTT broker running at mqtt.example.com.

Example 1: Publishing Temperature Data

• An ESP32 device in your living room measures the temperature.
• Publisher: ESP32
• Action: Publishes the temperature.
• Topic: myhome/livingroom/sensors/temperature
• Payload (e.g., JSON): {"value": 24.5, "unit": "celsius"}
• QoS: 1 (ensure the reading reaches the broker)
• Retain: true (so new subscribers immediately get the latest temperature)
• Command-line (mosquitto_pub):

mosquitto_pub -h mqtt.example.com -t "myhome/livingroom/sensors/temperature" -m "{\"value\": 24.5, \"unit\": \"celsius\"}" -q 1 -r

Example 2: Subscribing to Control a Light

• An ESP32 device controls a smart light in the bedroom. A mobile app is used to turn it on/off.
• Subscriber: ESP32 controlling the light
• Action: Listens for commands.
• Subscribed Topic: myhome/bedroom/lights/light01/command
• QoS: 1 (ensure commands are received)
• Command-line (mosquitto_sub to listen):

mosquitto_sub -h mqtt.example.com -t "myhome/bedroom/lights/light01/command" -q 1 -v

• ,Publisher: Mobile App
• Action: Sends a command to turn the light ON.
• Topic: myhome/bedroom/lights/light01/command
• Payload: ON
• QoS: 1
• Command-line (mosquitto_pub to send command):

mosquitto_pub -h mqtt.example.com -t "myhome/bedroom/lights/light01/command" -m "ON" -q 1

Example 3: Using Wildcards for an Aggregated Dashboard

• A central dashboard application wants to display all temperature readings from all rooms and all sensor data from the kitchen.
• Subscriber: Dashboard Application
• Subscription 1 (all temperatures): myhome/+/sensors/temperature
• Subscription 2 (all kitchen data): myhome/kitchen/#
• Command-line (mosquitto_sub):

# Terminal 1: Listen for all temperatures
mosquitto_sub -h mqtt.example.com -t "myhome/+/sensors/temperature" -v

# Terminal 2: Listen for all kitchen data
mosquitto_sub -h mqtt.example.com -t "myhome/kitchen/#" -v

Build Instructions:

Not applicable for this conceptual chapter. Instructions for setting up an ESP32 MQTT client will be in Chapter 102.

Run/Flash/Observe Steps:

To experiment with the commands above:

1. You need an MQTT broker. You can install Mosquitto locally or use a public test broker like test.mosquitto.org or broker.hivemq.com.
   • Warning: Public brokers are insecure and data is visible to everyone. Do not send sensitive information.
2. Install MQTT command-line tools. If you installed Mosquitto, these are typically mosquitto_pub and mosquitto_sub.
3. Open multiple terminal windows.
4. In one terminal, run a mosquitto_sub command to subscribe to a topic (or topics with wildcards). The -v flag usually prints the topic along with the message.
5. In another terminal, run a mosquitto_pub command to publish a message to that topic.
6. Observe the message appearing in the subscriber’s terminal. Experiment with different topics, payloads, QoS levels, and the retain flag.

Variant Notes

The fundamental principles of the MQTT protocol are standardized and thus remain consistent across all ESP32 variants that support IP networking (Wi-Fi or Ethernet). This includes:

• ESP32
• ESP32-S2
• ESP32-S3
• ESP32-C3 (Wi-Fi)
• ESP32-C6 (Wi-Fi, 802.15.4 for Thread/Zigbee)
• ESP32-H2 (802.15.4 for Thread/Zigbee)

Key considerations for different variants:

1. Network Connectivity:
   • Most ESP32 variants primarily use Wi-Fi to connect to an MQTT broker.
   • Some ESP32 series (like the original ESP32) also support Ethernet via an external PHY, which can provide a more stable connection for MQTT.
   • For variants like ESP32-C6 and ESP32-H2 which support Thread or Zigbee (IEEE 802.15.4), connecting to an IP-based MQTT broker would typically require an 802.15.4 border router or a gateway device that bridges the 802.15.4 network to the IP network where the broker resides. The MQTT communication itself, once the IP bridge is crossed, remains standard.
2. Resource Constraints:
   • MQTT is designed to be lightweight. However, more resource-constrained variants (e.g., those with less RAM/Flash like some ESP32-C3 configurations) might be limited in the complexity of the MQTT application they can run. This could affect the number of active subscriptions, the size of message payloads handled, or the overhead of TLS connections.
   • The ESP-IDF esp-mqtt client library (which we will cover in Chapter 102) is optimized for ESP32 devices and generally performs well across variants.
3. TLS Hardware Acceleration:
   • For secure MQTT communication (MQTTS, MQTT over TLS), cryptographic operations can be computationally intensive.
   • Newer ESP32 variants (ESP32-S2, ESP32-S3, ESP32-C2, ESP32-C3, ESP32-C6, ESP32-H2) often include hardware acceleration for cryptographic functions (AES, SHA, RSA, ECC).
   • This hardware acceleration can significantly improve the performance of TLS handshakes and encryption/decryption, reduce CPU load, and lower power consumption when using MQTTS. This will be particularly relevant in Chapter 105 (MQTT TLS Security). The original ESP32 has some hardware acceleration, but newer chips often have more comprehensive support.

Essentially, the way you think about MQTT topics, publishing, subscribing, and QoS will be identical regardless of the specific ESP32 chip you are using. The differences will lie in how the device establishes network connectivity and the performance characteristics, especially for secure connections.

Common Mistakes & Troubleshooting Tips

Exercises

1. Broker Familiarization:
   • Install a local MQTT broker (e.g., Mosquitto) on your computer OR identify a public MQTT test broker (e.g., test.mosquitto.org, broker.hivemq.com).
   • Using a graphical MQTT client tool (like MQTT Explorer, MQTTLens, or MQTTBox), connect to your chosen broker. Explore its interface. Can you see any system topics (e.g., under $SYS)?
2. Topic Hierarchy Design:
   • Imagine you are designing an MQTT system for a small smart greenhouse. It has sensors for temperature, humidity, soil moisture, and light level. It also has actuators for a water pump, a fan, and grow lights.
   • Design a topic hierarchy for this system. List at least five example topics for publishing sensor data and five example topics for sending commands to actuators. Consider using a structure like greenhouse/<location>/<type>/<id_or_measurement>.
3. Command-Line MQTT Practice:
   • If you have mosquitto_pub and mosquitto_sub (or similar CLI tools) installed and a broker accessible:
     1. Open two terminal windows.
     2. In Terminal 1, subscribe to the topic smartgarden/plot1/soilmoisture.
     3. In Terminal 2, publish a message {"value": 65, "unit": "percent"} to smartgarden/plot1/soilmoisture with QoS 1. Observe it in Terminal 1.
     4. In Terminal 1, subscribe to smartgarden/alerts/#.
     5. In Terminal 2, publish an alert message Low water level! to smartgarden/alerts/pumps/pump01 with QoS 0. Observe it.
4. Wildcard and Retain Flag Exploration:
   • Using an MQTT client tool (CLI or GUI):
     1. Subscribe to office/+/status.
     2. Publish a message {"state": "online"} to office/desk_lamp/status with the Retain flag set to true.
     3. Publish a message {"state": "active"} to office/computer/status with the Retain flag set to true.
     4. Disconnect your MQTT client tool and then reconnect it. Subscribe again to office/+/status. Do you immediately receive the retained messages?
     5. Now, publish an empty message (zero-length payload) to office/desk_lamp/status with the Retain flag set to true. What happens to the retained message for this topic? (Verify by disconnecting/reconnecting and subscribing again, or by using a tool that shows retained messages).

Summary

• MQTT (Message Queuing Telemetry Transport) is a lightweight, efficient publish-subscribe messaging protocol ideal for IoT and constrained environments.
• It operates on a Broker-Client model: Clients connect to a central Broker.
• Publishers send messages to Topics; Subscribers express interest in Topics to receive messages. This decouples senders and receivers.
• Topics are hierarchical, case-sensitive UTF-8 strings (e.g., area/device/sensor_type).
• Subscribers can use wildcards (+ for single-level, # for multi-level) to subscribe to multiple topics.
• Quality of Service (QoS) levels (0, 1, 2) define message delivery guarantees:
  • QoS 0: At most once (fire and forget).
  • QoS 1: At least once (duplicates possible).
  • QoS 2: Exactly once (most reliable, highest overhead).
• Key MQTT features include:
  • Retain Flag: Allows the broker to store the last message on a topic for new subscribers.
  • Last Will and Testament (LWT): A message published by the broker if a client disconnects ungracefully.
  • Keep Alive: Mechanism to detect dead connections.
  • Clean Session / Persistent Session: Determines if the broker should store session state (subscriptions, queued messages) for a client across disconnections.
• MQTT’s design for low bandwidth, high latency, and unreliable networks makes it a cornerstone protocol for many ESP32-based IoT solutions.

Further Reading

• MQTT Version 5.0 Specification (OASIS Standard): https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html
• MQTT Version 3.1.1 Specification (OASIS Standard): https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html (Still widely used)
• HiveMQ MQTT Essentials: https://www.hivemq.com/mqtt-essentials/ (An excellent, easy-to-understand guide to MQTT concepts).
• Mosquitto MQTT Broker Documentation: https://mosquitto.org/documentation/
• ESP-IDF MQTT Client Documentation (esp-mqtt): (This will be covered in the next chapter, but for future reference) https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-reference/protocols/mqtt.html (Ensure to select your specific ESP-IDF version and target chip if the link defaults differently).