Chapter 104: MQTT Authentication Methods

Chapter Objectives

After completing this chapter, you will be able to:

• Understand the importance of authentication in MQTT communication.
• Explain common MQTT authentication mechanisms:
  • Username/Password authentication.
  • Client Certificate-based authentication (introduction).
  • Token-based authentication patterns.
• Implement username/password authentication for an ESP32 MQTT client using ESP-IDF.
• Configure the esp-mqtt client for basic authentication scenarios.
• Recognize the role of the MQTT broker in enforcing authentication.
• Identify common issues and troubleshooting steps related to MQTT authentication.

Introduction

In the preceding chapters, we’ve explored how to establish MQTT connections and ensure message reliability using QoS levels and persistent sessions. However, in most real-world IoT deployments, simply allowing any device to connect to your MQTT broker poses a significant security risk. Unauthorized devices could publish malicious data, subscribe to sensitive information, or disrupt the overall system.

Authentication is the process of verifying the identity of a client attempting to connect to the MQTT broker. It ensures that only legitimate and authorized devices or applications can participate in the MQTT communication. This chapter delves into various authentication methods commonly used with MQTT, focusing on their principles and practical implementation on ESP32 devices using the ESP-IDF. Securing your MQTT infrastructure starts with robust authentication, forming a critical layer in your IoT security posture.

Theory

MQTT itself, in its core specification (v3.1.1 and v5.0), provides direct support for username/password authentication. Other methods, like client certificate authentication, are typically layered on top using transport-level security (TLS), while token-based authentication is often implemented as an application-level pattern utilizing existing MQTT fields.

1. Username/Password Authentication

This is the most straightforward authentication method explicitly defined in the MQTT protocol.

• Mechanism:
  1. When a client initiates a connection, it sends a CONNECT packet to the broker.
  2. This CONNECT packet can optionally include a Username and a Password field.
  3. The MQTT broker receives these credentials and validates them against its configured user database or authentication backend.
  4. If the credentials are valid, the broker sends a CONNACK packet with a “Connection Accepted” status.
  5. If the credentials are invalid (or the client is not authorized for other reasons), the broker sends a CONNACK packet with an appropriate error code (e.g., “Not authorized” or “Bad username or password”) and closes the connection.
• CONNECT Packet Flags:
  • The CONNECT packet has User Name Flag and Password Flag bits in its header.
  • If the User Name Flag is set to 1, the Username field must be present in the payload.
  • If the Password Flag is set to 1, the Password field must be present in the payload.
  • It’s invalid to have the Password Flag set to 1 if the User Name Flag is 0.

sequenceDiagram
    participant Client
    participant Broker
    activate Client
    activate Broker
    Client->>+Broker: CONNECT Packet <br>(Fixed Header, Variable Header <br>[Flags: User=1, Pass=1], <br>Payload [ClientID, Username, Password])
    Note over Client: "Sends credentials."
    
    Broker->>Broker: Validate Credentials <br>(Username, Password) <br>against user database/auth backend
    
    alt "Credentials Valid"
        Broker-->>-Client: CONNACK Packet<br> (Return Code: 0 - Connection Accepted)
        Note over Client: "Connection successful."
    else "Credentials Invalid"
        Broker-->>-Client: CONNACK Packet <br>(Return Code: 4 - Bad username or password, <br>or 5 - Not authorized)
        Note over Client: "Connection failed. <br>Broker closes connection."
    end

• Security Considerations:
  • Transport Encryption: Username and password are sent in plaintext within the MQTT CONNECT packet. Therefore, it is highly recommended to use this method only over a TLS-encrypted connection (MQTT over TLS, often on port 8883). Chapter 105 will cover TLS in detail. Without TLS, credentials can be easily intercepted.
  • Strong Credentials: Use strong, unique passwords for each client.
  • Broker Configuration: The broker must be configured to require and validate these credentials (e.g., using password files, database lookups, or integration with authentication services).
• Pros:
  • Simple to understand and implement.
  • Widely supported by MQTT brokers and clients.
• Cons:
  • Insecure if not used over TLS.
  • Managing passwords for a large number of devices can be challenging.
  • Credentials might need to be stored on the device, requiring secure storage mechanisms.

2. Client Certificate-Based Authentication

This method leverages X.509 client certificates as part of the TLS handshake to authenticate the client. While the full TLS setup is covered in Chapter 105, it’s important to understand its role in authentication here.

• Mechanism:
  1. The MQTT connection is established over TLS.
  2. During the TLS handshake (before any MQTT packets are exchanged), the client presents its X.509 certificate to the broker.
  3. The broker verifies the client’s certificate:
     • Checks if it’s signed by a trusted Certificate Authority (CA).
     • Checks its validity period.
     • May check a Certificate Revocation List (CRL) or use OCSP.
     • May extract identity information (e.g., Common Name – CN) from the certificate for authorization purposes.
  4. If the certificate is valid and trusted, the TLS handshake completes, and the client is considered authenticated at the transport layer.
  5. The MQTT CONNECT packet is then sent over this secure channel. The Username and Password fields in the CONNECT packet might be omitted if certificate authentication is sufficient, or they could be used for an additional layer of application-level authorization.
• Pros:
  • Strong security: Relies on asymmetric cryptography.
  • No passwords transmitted over the network (authentication happens during TLS handshake).
  • Scalable for managing device identities, as certificates can be issued and managed centrally.
• Cons:
  • More complex to set up and manage (requires generating, distributing, and renewing certificates and CAs).
  • Higher computational overhead on the client and broker due to cryptographic operations.
  • Requires secure storage of the client’s private key on the device.

sequenceDiagram
    participant Client
    participant Broker
    activate Client
    activate Broker

    Note over Client,Broker: TLS Handshake <br>(Simplified for Authentication Focus)
    Client->>+Broker: 1. ClientHello (Proposes TLS version, ciphers)
    Broker-->>-Client: 2. ServerHello <br>(Selects TLS version, cipher), <br>Certificate (Server's Cert), <br>CertificateRequest (Requests Client's Cert)
    Client->>+Broker: 3. Certificate (Client's Cert),<br> ClientKeyExchange,<br> CertificateVerify (Proves possession of private key),<br> ChangeCipherSpec, Finished
    Broker->>Broker: 4. Validate Client Certificate <br>(Trusted CA, Validity, Revocation Status, Identity)
    
    alt Client Certificate Valid & Trusted
        Broker-->>-Client: 5. ChangeCipherSpec, Finished
        Note over Client,Broker: TLS Handshake Complete.<br> Secure channel established.
        Client->>+Broker: 6. MQTT CONNECT Packet<br> (Username/Password may be omitted or used for app-level auth)
        Broker-->>-Client: 7. MQTT CONNACK (Connection Accepted)
    else Client Certificate Invalid or Not Trusted
        Broker-->>-Client: TLS Alert (e.g., bad_certificate, certificate_unknown)
        Note over Client,Broker: TLS Handshake Fails. Connection aborted.
    end

Tip: Client certificate authentication is a robust method often preferred for production IoT deployments due to its strong security posture.

3. Token-Based Authentication

Token-based authentication is an application-level pattern, not a distinct MQTT protocol feature. It often uses short-lived, signed tokens (like JSON Web Tokens – JWTs) to authenticate clients.

sequenceDiagram
    participant Client
    participant Auth_Server as Authentication Server<br/>(e.g., HTTPS API)
    participant MQTT_Broker as MQTT Broker<br/>(with Token Auth Plugin/Logic)
    activate Client
    activate Auth_Server
    activate MQTT_Broker
    Client->>+Auth_Server: 1. Request Token<br/>(e.g., with device ID, initial secret)
    Auth_Server-->>-Client: 2. Issue Token<br/>(e.g., JWT - Signed, Short-lived)
    
    Client->>+MQTT_Broker: 3. MQTT CONNECT Packet<br/>(Token in Username or Password field)
    Note over Client: Example: Username="device123"<br/>Password="JWT_TOKEN"
    
    MQTT_Broker->>MQTT_Broker: 4. Validate Token<br/>(Signature, Expiry, Claims)
    Note over MQTT_Broker: May involve checking with<br/>Auth Server's public key if JWT
        
    alt Token Valid
        MQTT_Broker-->>-Client: 5. MQTT CONNACK<br/>(Connection Accepted)
    else Token Invalid
        MQTT_Broker-->>-Client: 5. MQTT CONNACK<br/>(Not Authorized / Bad Credentials)
        Note over Client: Connection failed
    end
    
    alt Token Nears Expiry (Client Logic)
        Client->>+Auth_Server: Request New Token (Token Refresh)
        Auth_Server-->>-Client: Issue New Token
        Note over Client: Client may need to disconnect<br/>and reconnect with new token,<br/>or broker might support<br/>in-session token update (less common)
    end

• Mechanism:
  1. The client first obtains a token from a separate authentication server/service (e.g., via an HTTPS request). This often involves presenting some initial credentials or device identity.
  2. The client then connects to the MQTT broker. The token is typically passed to the broker in one of these ways:
     • Username Field: The token is placed in the Username field of the MQTT CONNECT packet. The Password field might be left empty, used for another purpose, or contain a dummy value.
     • Password Field: The token is placed in the Password field. The Username field might contain the ClientID or a static identifier.
  3. The MQTT broker (or an integrated authentication plugin/service) validates the token:
     • Checks its signature.
     • Checks its expiry time.
     • Checks its claims (e.g., permissions, identity).
  4. If the token is valid, the connection is accepted.
• Pros:
  • Fine-grained Permissions: Tokens can carry specific permissions or claims.
  • Short-Lived Credentials: Tokens can be set to expire, reducing the risk if a token is compromised. Clients can implement logic to refresh tokens.
  • Stateless Broker Authentication: Brokers can often validate tokens without needing to store per-client credentials directly (if using asymmetric tokens like JWTs signed by a trusted authority).
  • Can be integrated with existing OAuth 2.0 or other identity management systems.
• Cons:
  • Requires an external authentication server/service to issue and manage tokens.
  • Client needs logic to obtain and potentially refresh tokens.
  • Increased complexity in the overall system architecture.
  • Like username/password, relies on TLS for secure transmission of the token within the MQTT packet.

Broker’s Role

It’s crucial to remember that the MQTT client only presents credentials. The MQTT broker is responsible for enforcing authentication by:

• Validating usernames and passwords against a user database.
• Verifying client certificates against trusted CAs.
• Validating tokens (signature, expiry, claims).
• Maintaining Access Control Lists (ACLs) to determine what topics an authenticated client can publish to or subscribe from (this is authorization, which typically follows authentication).

Practical Examples

Let’s focus on implementing username/password authentication using the esp-mqtt client in ESP-IDF. Configuration for client certificate authentication will be briefly shown, with full details in the next chapter.

1. Setting up MQTT Client with Username/Password

The esp_mqtt_client_config_t structure allows you to specify username and password.

#include "esp_log.h"
#include "mqtt_client.h"
#include "esp_wifi.h" // For Wi-Fi connection events
#include <string.h> // For strerror

static const char *TAG = "MQTT_AUTH_EXAMPLE";

esp_mqtt_client_handle_t client = NULL;

// Assume wifi_init_sta() and event handler for Wi-Fi connection are set up

static void log_error_if_nonzero(const char *message, int error_code)
{
    if (error_code != 0) {
        ESP_LOGE(TAG, "Last error %s: 0x%x", message, error_code);
    }
}

static esp_err_t mqtt_event_handler_cb(esp_mqtt_event_handle_t event)
{
    // esp_mqtt_client_handle_t client = event->client; // client is global in this example
    // int msg_id;
    switch (event->event_id) {
        case MQTT_EVENT_CONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
            // Now you are authenticated (if broker required it and accepted)
            // You can publish/subscribe here
            esp_mqtt_client_publish(client, "/topic/esp_status", "online_authed", 0, 1, 0);
            esp_mqtt_client_subscribe(client, "/topic/commands", 1);
            break;
        case MQTT_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
            break;
        case MQTT_EVENT_SUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
            ESP_LOGI(TAG, "Subscribed to topic: %.*s", event->topic_len, event->topic);
            break;
        case MQTT_EVENT_UNSUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_UNSUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_PUBLISHED:
            ESP_LOGI(TAG, "MQTT_EVENT_PUBLISHED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_DATA:
            ESP_LOGI(TAG, "MQTT_EVENT_DATA");
            ESP_LOGI(TAG, "TOPIC=%.*s", event->topic_len, event->topic);
            ESP_LOGI(TAG, "DATA=%.*s", event->data_len, event->data);
            break;
        case MQTT_EVENT_ERROR:
            ESP_LOGE(TAG, "MQTT_EVENT_ERROR");
            if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
                log_error_if_nonzero("reported from esp-tls", event->error_handle->esp_tls_last_esp_err);
                log_error_if_nonzero("reported from tls stack", event->error_handle->esp_tls_stack_err);
                log_error_if_nonzero("captured as transport's socket errno",  event->error_handle->esp_transport_sock_errno);
                ESP_LOGI(TAG, "Last errno string (%s)", strerror(event->error_handle->esp_transport_sock_errno));
            } else if (event->error_handle->error_type == MQTT_ERROR_TYPE_CONNECTION_REFUSED) {
                ESP_LOGE(TAG, "Connection refused error: 0x%x", event->error_handle->connect_return_code);
                // Common connect_return_code values:
                // 0x04: Bad username or password
                // 0x05: Not authorized
                // Check your broker logs for more details.
            } else {
                ESP_LOGW(TAG, "Unknown error type: 0x%x", event->error_handle->error_type);
            }
            break;
        default:
            ESP_LOGI(TAG, "Other event id:%d", event->event_id);
            break;
    }
    return ESP_OK;
}

static void mqtt_app_start(void)
{
    esp_mqtt_client_config_t mqtt_cfg = {
        .broker.address.uri = CONFIG_BROKER_URL, // e.g., "mqtt://mqtt.example.com"
        // Username/Password Authentication
        .credentials.username = CONFIG_MQTT_USERNAME,
        .credentials.authentication.password = CONFIG_MQTT_PASSWORD,
        .credentials.client_id = "esp32-auth-client-01", // Ensure this is unique if required
        // .session.disable_clean_session = false, // Default
    };

    // For MQTT over TLS (mqtts:// or port 8883), you'd also configure certificates:
    // .broker.verification.certificate = (const char *)server_cert_pem_start, // Server CA cert
    // .credentials.authentication.certificate = (const char *)client_cert_pem_start, // Client cert
    // .credentials.authentication.key = (const char *)client_key_pem_start,       // Client private key
    // More on this in Chapter 105.

    client = esp_mqtt_client_init(&mqtt_cfg);
    esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler_cb, NULL); // Pass NULL or context
    esp_mqtt_client_start(client);
}

// In your main_app function, after Wi-Fi is connected:
// mqtt_app_start();

Configuration (via idf.py menuconfig):

1. Navigate to Component config -> ESP MQTT Client.
2. Set CONFIG_BROKER_URL to your MQTT broker’s URI (e.g., mqtt://test.mosquitto.org).
   • Warning: For production and actual credential use, always use mqtts://your-broker-address (MQTT over TLS) and ensure your broker is configured for TLS. The example above uses mqtt:// for simplicity, but this sends credentials in plaintext.
3. Set CONFIG_MQTT_USERNAME to your MQTT username.
4. Set CONFIG_MQTT_PASSWORD to your MQTT password.

You can also set these directly in sdkconfig.defaults or your project’s sdkconfig file.

If your broker requires a specific client_id for authentication or ACLs, ensure .credentials.client_id is set appropriately.

2. Broker Setup (Example with Mosquitto)

If you are using Mosquitto as your broker, you’ll need to:

1. Create a password file:sudo mosquitto_passwd -c /etc/mosquitto/passwd your_username
It will prompt you to enter a password for your_username.
2. Configure Mosquitto to use this password file. Edit /etc/mosquitto/mosquitto.conf (or conf.d/default.conf):allow_anonymous false password_file /etc/mosquitto/passwd
3. Restart Mosquitto:sudo systemctl restart mosquitto

Now, Mosquitto will require username/password authentication.

3. Build and Run Instructions

1. Configure:
   • Set up your Wi-Fi credentials.
   • Set CONFIG_BROKER_URL, CONFIG_MQTT_USERNAME, and CONFIG_MQTT_PASSWORD via idf.py menuconfig as described above.
2. Build:idf.py build
3. Flash:idf.py -p /dev/YOUR_SERIAL_PORT flash
4. Monitor:idf.py -p /dev/YOUR_SERIAL_PORT monitor
5. Observe:
   • If authentication is successful, you’ll see MQTT_EVENT_CONNECTED.
   • If authentication fails, you’ll see MQTT_EVENT_ERROR with error_handle->error_type == MQTT_ERROR_TYPE_CONNECTION_REFUSED. The connect_return_code in event->error_handle will provide more specific reasons:
     • 0x04: Bad username or password.
     • 0x05: Not authorized.
   • Check your MQTT broker logs for detailed error messages regarding authentication failures.

4. Client Certificate Authentication (Configuration Snippet)

As a preview for Chapter 105, here’s how you would configure the esp-mqtt client if you were using client certificate authentication (assuming you have the server CA certificate, client certificate, and client private key embedded in your firmware, e.g., using idf.py embed-txtfiles):

// In esp_mqtt_client_config_t:

// Assume these are pointers to your embedded PEM certificate strings
extern const uint8_t server_cert_pem_start[] asm("_binary_server_root_ca_pem_start");
extern const uint8_t client_cert_pem_start[] asm("_binary_client_crt_pem_start");
extern const uint8_t client_key_pem_start[]  asm("_binary_client_key_pem_start");

// ...
    .broker.address.uri = "mqtts://secure.broker.com:8883", // Note 'mqtts' and port 8883
    .broker.verification.certificate = (const char *)server_cert_pem_start,
    .credentials.authentication.certificate = (const char *)client_cert_pem_start,
    .credentials.authentication.key = (const char *)client_key_pem_start,
    .credentials.client_id = "esp32-cert-client-01", // Often derived from cert CN
// ...

In this case, the username and password fields might be left NULL or empty if the broker authenticates solely based on the client certificate.

Variant Notes

Authentication methods themselves are protocol-defined and generally work the same across ESP32 variants. However, resource considerations and hardware features can be relevant:

• Secure Storage of Credentials:
  • NVS (Non-Volatile Storage): Usernames, passwords, or even small tokens can be stored in NVS. However, NVS is not encrypted by default.
  • Flash Encryption: All ESP32 variants support flash encryption. If enabled, data stored in flash (including NVS partitions or embedded files) is encrypted at rest, providing protection if the physical flash chip is accessed. This is crucial for storing sensitive credentials like private keys or passwords.
  • Secure Element (ATECC608A, etc.): Some ESP32 development boards or custom designs might integrate a secure element. These hardware components can securely store private keys and perform cryptographic operations, preventing key extraction. ESP32-H2 has features that can work with external secure elements. While esp-mqtt doesn’t directly interface with generic secure elements out-of-the-box for MQTT private keys, you might use esp-cryptoauthlib to retrieve a key or perform signing, then provide it to the MQTT stack if it supports external private key operations (more advanced).
  • Digital Signature Peripheral (DS): ESP32-S2, ESP32-S3, ESP32-C3, ESP32-C6, and ESP32-H2 include a Digital Signature peripheral that can help protect private keys used for signing. It can store keys in eFuses and perform signature operations without exposing the key to software. This is particularly useful for secure boot and flash encryption but can also be leveraged for application-level security if private keys need protection.
• TLS Performance (for Client Certificate Auth & Encrypted Username/Password):
  • Client certificate authentication involves TLS, which uses cryptographic operations.
  • Most ESP32 variants (ESP32, S2, S3, C3, C6, H2) have hardware acceleration for cryptographic algorithms (AES, SHA, RSA, ECC). This significantly improves TLS handshake performance and reduces CPU load compared to purely software-based crypto.
  • Variants with more processing power (e.g., dual-core ESP32, ESP32-S3) will naturally handle these operations faster.
• RAM Usage:
  • Storing certificates (server CA, client cert, client key) in RAM consumes memory. Embedding them directly in flash is common.
  • The TLS stack itself requires a significant amount of RAM for buffers and state during the handshake. This is generally manageable on all ESP32 variants but should be considered in resource-constrained applications.

In essence, while the authentication logic is standard, how securely you store credentials and the performance of cryptographic operations can vary slightly based on the ESP32 variant’s hardware capabilities and your project’s configuration (like enabled flash encryption).

Common Mistakes & Troubleshooting Tips

Exercises

1. Username/Password Authentication with Public Broker:
   • Find a public MQTT broker that supports username/password authentication (e.g., HiveMQ Cloud’s free tier, EMQX Cloud, or configure a local Mosquitto instance).
   • Obtain or create credentials (username/password).
   • Modify the example code in this chapter to connect to this broker using these credentials.
   • Verify successful connection and try publishing/subscribing.
   • Intentionally use incorrect credentials and observe the MQTT_EVENT_ERROR and connect_return_code. Correlate with broker logs if possible.
2. Broker Configuration for Authentication:
   • If you have a local Mosquitto broker, configure it to disallow anonymous connections and use a password file (as shown in the “Broker Setup” section).
   • Test connecting with your ESP32:
     • Without any credentials (should be rejected).
     • With correct credentials (should connect).
     • With incorrect credentials (should be rejected with “Bad username or password”).
3. Conceptual: Token Authentication Flow:
   • Assume you have an authentication server that issues JWTs upon successful login via an HTTPS endpoint.
   • Outline the steps your ESP32 application would need to take to:
     1. Obtain a JWT from this server.
     2. Connect to an MQTT broker using this JWT (e.g., in the password field).
     3. (Bonus) Consider how the ESP32 might handle token expiry and renewal.
   • You don’t need to write full ESP-IDF code, but describe the sequence of operations and key ESP-IDF components/libraries you might use (e.g., HTTP client for fetching the token).

Summary

• Authentication is vital for securing MQTT communication by verifying client identities.
• Username/Password: Simple, widely supported. Credentials sent in CONNECT packet. Must be used over TLS.
• Client Certificate: Strong, uses X.509 certificates during TLS handshake. More complex to manage but highly secure.
• Token-Based: Flexible, uses tokens (e.g., JWTs) often in username/password fields. Requires an external auth server.
• The esp-mqtt client in ESP-IDF supports username/password and client certificate authentication through its configuration structure.
• The MQTT broker is responsible for enforcing authentication and authorization.
• Proper error handling, especially checking connect_return_code in MQTT_EVENT_ERROR, is crucial for troubleshooting.
• ESP32 variants offer hardware features like flash encryption and crypto acceleration that can enhance the security and performance of authentication mechanisms.

Further Reading

• ESP-IDF MQTT Client Documentation:
  • ESP-MQTT Client
• MQTT v3.1.1 Specification (CONNECT packet details):
  • OASIS MQTT Version 3.1.1
• MQTT v5.0 Specification (Enhanced Authentication):
  • OASIS MQTT Version 5.0 (Section on Enhanced Authentication for more advanced scenarios)
• Mosquitto Documentation (Authentication):
  • mosquitto.conf man page (Search for password_file, allow_anonymous)
  • mosquitto_passwd man page
• HiveMQ Blog: MQTT Security Fundamentals:
  • MQTT Security Fundamentals