Chapter 109: MQTT Bridges and Proxies with ESP-IDF

Chapter Objectives

By the end of this chapter, you will be able to:

• Define MQTT bridges and MQTT proxies and understand their distinct roles in an IoT architecture.
• Explain the common use cases for MQTT bridges, such as broker federation and cloud integration.
• Describe the purposes of MQTT proxies, including load balancing, security enhancement, and protocol translation.
• Understand the basic principles of how MQTT bridges connect brokers and transfer messages.
• Recognize how an ESP32 MQTT client’s interaction might change when connecting through a proxy or to a broker that is part of a bridged system.
• Identify potential configuration considerations for ESP32 devices in such environments, especially concerning security and network endpoints.
• Appreciate that these are primarily broker-side or intermediary server configurations, with ESP32s acting as standard clients.
• Understand that the core MQTT client logic on ESP32 variants remains largely the same.

Introduction

In the previous chapters, we’ve focused on the direct communication between ESP32 MQTT clients and a single MQTT broker. This model is suitable for many applications. However, as IoT systems scale in size, geographical distribution, or complexity, a single broker might become a bottleneck or might not meet all architectural requirements. Scenarios like connecting isolated factory floor networks to a central enterprise system, linking on-premises brokers to cloud platforms, or enhancing security and scalability often necessitate more advanced MQTT network topologies.

This is where MQTT bridges and proxies come into play. An MQTT bridge allows two or more MQTT brokers to connect and share messages, effectively creating a larger, federated MQTT network. An MQTT proxy, on the other hand, acts as an intermediary between clients and a broker (or between brokers), offering services like load balancing, SSL/TLS termination, or request filtering. This chapter will explore these two important concepts, their functionalities, and how the presence of bridges or proxies might influence the way your ESP32 devices connect and communicate.

Theory

MQTT Bridge

An MQTT bridge is essentially a specialized MQTT client that connects two (or more) MQTT brokers. Its primary function is to subscribe to topics on one broker and republish the received messages to another broker, and vice-versa if configured for bidirectional communication. This allows messages to flow between distinct MQTT broker instances, which might be in different networks, geographical locations, or managed by different entities.

Analogy: Think of an MQTT bridge as an “embassy” or a “liaison office” between two separate cities (brokers). The embassy in City A listens for important announcements (messages on specific topics) in City A and relays them to its home office in City B. Similarly, it might relay messages from City B back to City A.

Key Purposes and Use Cases:

How it Works:

• A bridge is typically configured on one of the brokers (let’s call it Broker A).
• This configuration specifies the address of the remote broker (Broker B) and credentials for connecting to it.
• It defines topic mappings or patterns:
  • Which topics on Broker A should be forwarded to Broker B.
  • Which topics on Broker B (that Broker A’s bridge client subscribes to) should be re-published onto Broker A.
  • The direction of message flow (in, out, or both).
  • Optional: Topic prefixing or remapping (e.g., messages from local/sensors/# on Broker A are published to site_X/sensors/# on Broker B).
• The bridge on Broker A establishes an MQTT client connection to Broker B.
• Based on the configuration, it subscribes to specified topics on Broker A (if forwarding “out” from A) or Broker B (if forwarding “in” to A).
• When a message arrives on a subscribed topic, the bridge republishes it to the target broker on the mapped topic.

graph TB
    subgraph "Broker_A_Network [Broker A Environment (e.g., On-Premises)]"
        direction LR
        ESP32_Client_1["<b>ESP32 Client 1</b><br>Publishes to:<br><i>local/factory_A/machine_1/temp</i>"] -- Message --> Broker_A["<b>Broker A</b><br>(e.g., Mosquitto)"];
        subgraph Bridge_Config_A [Bridge on Broker A]
            direction TB
            Config["<b>Bridge Configuration:</b><br>- Connect to Broker B (Cloud)<br>- Forward: <i>local/factory_A/#</i> (from A)<br>  to <i>cloud/factory_A/#</i> (on B)<br>- Direction: out<br>- QoS: 1"]
            Bridge_Client_A["<b>Bridge Client</b><br>(Part of Broker A,<br>acts as client to Broker B)"]
        end
        Broker_A -- Monitored Topic Match --> Bridge_Client_A;
    end

    subgraph "Broker_B_Network [Broker B Environment (e.g., Cloud)]"
        direction LR
        Broker_B["<b>Broker B</b><br>(e.g., Cloud MQTT Service)"]
        Cloud_App["<b>Cloud Application</b><br>Subscribes to:<br><i>cloud/factory_A/machine_1/temp</i>"]
        Broker_B -- Delivers Message --> Cloud_App;
    end

    Bridge_Client_A -- Publishes Remapped Message<br>(e.g., to <i>cloud/factory_A/machine_1/temp</i>) --> Broker_B;

    %% Optional Bidirectional Flow (Illustrative)
    subgraph Optional_Return_Path [Optional: Data from Cloud to On-Prem]
        direction RL
        Broker_A_Subscriber["<b>Local Subscriber on Broker A</b><br>Subscribes to:<br><i>local/commands/machine_1</i>"]
        Broker_A -- Delivers Command --> Broker_A_Subscriber
        Bridge_Client_A_SubPart["<b>Bridge Client (Sub Part)</b><br>Subscribes on Broker B to:<br><i>cloud/commands/factory_A/machine_1</i><br>Republishes on Broker A to:<br><i>local/commands/machine_1</i>"]
        Broker_B -- Command Message --> Bridge_Client_A_SubPart
    end
    Bridge_Client_A_SubPart -.-> Broker_A

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,font-family:'Open Sans',color:#333;
    classDef primaryNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef successNode fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;

    class ESP32_Client_1 primaryNode;
    class Cloud_App primaryNode;
    class Broker_A_Subscriber primaryNode;
    class Broker_A processNode;
    class Broker_B processNode;
    class Bridge_Client_A processNode;
    class Bridge_Client_A_SubPart processNode;
    class Config decisionNode;

Common Bridge Implementations:

Most full-featured MQTT brokers offer bridging capabilities. For example:

• Mosquitto: Has robust bridging features configured in its mosquitto.conf file.
• HiveMQ: Offers an extension for bridging.
• EMQX: Provides powerful bridging to various MQTT services and other data systems.

Security: Connections between bridged brokers must be secured, typically using TLS/SSL and client certificate authentication or username/password credentials, just like any other MQTT client connection.

MQTT Proxy

An MQTT proxy is an intermediary server that sits in the communication path between MQTT clients and an MQTT broker, or sometimes between brokers themselves. Unlike a bridge that connects two brokers, a proxy relays traffic, often adding value or control along the way.

Analogy: Think of an MQTT proxy as a “concierge” or a “security checkpoint” at the entrance of a large hotel (the broker). All guests (clients) must pass through the concierge. The concierge might check credentials, direct guests, or provide additional services before they reach the hotel’s main services.

Key Purposes and Use Cases:

graph TB
    subgraph Client_Side [ESP32 Clients]
        direction TB
        Client1["<b>ESP32 Client 1</b><br>(Connects via TLS/SSL)"]
        Client2["<b>ESP32 Client 2</b><br>(Connects via TLS/SSL)"]
    end

    subgraph Proxy_Server [MQTT Proxy Server]
        direction TB
        Proxy["<b>MQTT Proxy</b><br>(e.g., Nginx, HAProxy, Envoy)<br>Listens on: <i>mqtts://proxy.example.com:8883</i>"]
        Proxy_TLS["1- Terminates TLS Connection"]
        Proxy_Auth["2- Authenticates Client (Optional)"]
        Proxy_Log["3- Logs Request (Optional)"]
        Proxy_Forward["4- Forwards Plain MQTT<br>to Internal Broker"]
        Proxy --> Proxy_TLS --> Proxy_Auth --> Proxy_Log --> Proxy_Forward
    end

    subgraph Broker_Side [Backend MQTT Broker]
        direction TB
        Broker["<b>MQTT Broker</b><br>(e.g., Mosquitto)<br>Listens on: <i>mqtt://broker.internal:1883</i><br>(Trusted Network, Non-TLS)"]
    end

    Client1 -- "Encrypted MQTT<br>(MQTTS)" --> Proxy;
    Client2 -- "Encrypted MQTT<br>(MQTTS)" --> Proxy;
    Proxy_Forward -- "Plain MQTT" --> Broker;

    classDef default fill:#f9f9f9,stroke:#333,stroke-width:1px,font-family:'Open Sans',color:#333;
    classDef primaryNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef checkNode fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;


    class Client1 primaryNode;
    class Client2 primaryNode;
    class Proxy processNode;
    class Proxy_TLS processNode;
    class Proxy_Auth processNode;
    class Proxy_Log processNode;
    class Proxy_Forward processNode;
    class Broker processNode;

Types of Proxies in MQTT Context:

• Forward Proxy: Clients are configured to use the proxy to reach any external broker. This is less common for dedicated IoT device deployments where the broker endpoint is usually known.
• Reverse Proxy: Clients connect to the proxy, thinking it’s the actual broker. The proxy then intelligently routes the connection to one or more backend brokers. This is the more typical model for achieving scalability, TLS offload, etc., for an MQTT service. Popular web reverse proxies like Nginx or HAProxy can often be configured to proxy MQTT (TCP) traffic, and some can even inspect MQTT for more advanced routing.

Key Differences: Bridge vs. Proxy

Practical Examples

For ESP32 developers, bridges and proxies are typically transparent infrastructure components configured by backend or network administrators. The ESP32 usually just acts as a standard MQTT client. However, its configuration or behavior might be slightly affected.

Scenario 1: ESP32 in a Bridged MQTT Environment

Imagine an ESP32 sensor node in a factory.

• It publishes temperature readings to a local Mosquitto broker (Broker A) on the topic factory_X/floor_1/machine_A/temperature.
• Broker A is configured with a bridge to a central cloud MQTT broker (Broker B).
• The bridge configuration on Broker A forwards messages from factory_X/# to the topic enterprise/sensors/factory_X/# on Broker B.
• A cloud application subscribes to enterprise/sensors/factory_X/machine_A/temperature on Broker B to receive the data.

ESP32 Code (No Change Needed):

The ESP32’s MQTT client code connects to Broker A (e.g., mqtt://local-broker-ip:1883) and publishes to factory_X/floor_1/machine_A/temperature as usual. It is unaware of the bridge.

// ESP32 Code Snippet (Conceptual - standard publish)
// Assume 'client' is an initialized and connected esp_mqtt_client_handle_t
// connected to the *local* broker (Broker A)

char topic[128];
char payload[64];
float temperature = 25.5;

snprintf(topic, sizeof(topic), "factory_X/floor_1/machine_A/temperature");
snprintf(payload, sizeof(payload), "{\"value\": %.1f, \"unit\": \"C\"}", temperature);

esp_mqtt_client_publish(client, topic, payload, 0, 1, 0); // QoS 1, Retain 0
ESP_LOGI(TAG, "Published to local broker: %s -> %s", topic, payload);

Broker A (Mosquitto) Bridge Configuration Example (Conceptual – in mosquitto.conf):

This configuration would be on the server running Broker A, not on the ESP32.

# --- Bridge to Cloud Broker (Broker B) ---
connection cloud_bridge
address cloud-broker.example.com:8883 # Address of Broker B

# Topics to forward from local (Broker A) to remote (Broker B)
# Pattern: topic <topic_pattern> <direction> <qos> [local_prefix remote_prefix]
topic factory_X/# out 1 "" enterprise/sensors/ 
# This means:
# - Messages matching "factory_X/#" on Broker A
# - Are sent "out" to Broker B
# - With QoS 1
# - Local prefix "" (no change before matching)
# - Remote prefix "enterprise/sensors/" (prepended on Broker B)
# So, local "factory_X/floor_1/temp" becomes "enterprise/sensors/factory_X/floor_1/temp" on cloud.

# Remote client ID for this bridge connection
remote_clientid bridge_client_A_to_B
start_type automatic
cleansession false # For persistent bridge connection

# Security for the bridge connection to Broker B
bridge_cafile /path/to/cloud_broker_ca.crt
bridge_certfile /path/to/bridge_client.crt
bridge_keyfile /path/to/bridge_client.key
# Or remote_username, remote_password

Implication for ESP32: None directly on its code. The data it publishes locally transparently flows to the cloud via the bridge.

Scenario 2: ESP32 Connecting Through an MQTT Proxy (TLS Termination)

Imagine an ESP32 device needs to connect securely to an MQTT service, but the service uses a proxy for TLS termination.

• The actual MQTT broker (Broker C) listens on mqtt://broker-internal-ip:1883 (non-TLS).
• An MQTT Proxy (e.g., Nginx, HAProxy, or a dedicated MQTT proxy) listens on mqtts://public-proxy.example.com:8883 (TLS).
• The proxy terminates the TLS connection from the ESP32 and forwards the plain MQTT traffic to Broker C.

ESP32 Code (Connection Configuration Change):

The ESP32’s MQTT client code needs to:

1. Connect to the proxy’s public address and port: mqtts://public-proxy.example.com:8883.
2. Be configured with the appropriate CA certificate to verify the proxy’s server certificate.

// ESP32 Code Snippet (Conceptual - client configuration)
// Assume 'proxy_ca_pem_start' points to the CA cert for the proxy server

const esp_mqtt_client_config_t mqtt_cfg = {
    .broker.address.uri = "mqtts://public-proxy.example.com:8883", // Connect to proxy
    .broker.verification.certificate = (const char *)proxy_ca_pem_start, // CA to verify proxy
    // .credentials.client_id = ...
    // .session.last_will = ...
};

esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
esp_mqtt_client_start(client);

Implication for ESP32:

• The uri in esp_mqtt_client_config_t points to the proxy.
• The certificate field in broker.verification must contain the CA certificate that signed the proxy server’s SSL/TLS certificate. If the proxy uses a self-signed certificate (not recommended for production), that self-signed cert would be the CA cert.
• The rest of the MQTT operations (publish, subscribe) remain standard.

Build Instructions (for ESP32 code)

1. Standard ESP-IDF project setup.
2. Ensure MQTT component is enabled.
3. If using TLS (for connecting to a proxy or a secure broker), embed the necessary CA certificate into your firmware (see Chapter 105: MQTT TLS Security Implementation).
4. idf.py build

Run/Flash/Observe Steps (for ESP32 code)

1. idf.py -p /dev/ttyUSB0 flash monitor
2. Observe ESP32 logs for connection success to the specified endpoint (local broker in Scenario 1, proxy in Scenario 2).
3. Use an MQTT client tool to:
   • In Scenario 1: Subscribe to the mapped topic on Broker B (cloud) to see messages from the ESP32.
   • In Scenario 2: Subscribe to the relevant topic on Broker C (or through the proxy if it allows subscriptions) to see messages from the ESP32.

Variant Notes

MQTT bridges and proxies are external infrastructure components. ESP32 devices, regardless of the specific variant (ESP32, ESP32-S2, ESP32-S3, ESP32-C3, ESP32-C6, ESP32-H2), interact with these systems as standard MQTT clients.

1. Client Behavior: The core MQTT client logic implemented using esp-mqtt in ESP-IDF is the same across all variants. The primary change is the connection endpoint URI and potentially the TLS certificates.
2. Network Interface: The underlying network connectivity (Wi-Fi for most, Ethernet for some original ESP32s, Thread/Zigbee via a gateway for ESP32-H2) doesn’t change the MQTT-level interaction with a bridge or proxy, only how the IP connection is established to that first hop (local broker or proxy).
3. Resource Considerations:
   • Connecting via a proxy that handles TLS might slightly reduce the TLS handshake load on the ESP32 if the proxy-to-broker link is non-TLS, but the initial TLS handshake with the proxy still occurs.
   • The existence of bridges or proxies does not inherently add significant resource load (CPU/memory) on the ESP32 itself, as it’s still just maintaining one MQTT connection. Any added latency is due to the extra hops in the network path.
4. ESP32 as a (Micro) Broker/Bridge (Advanced/Niche):
   • While not typical, a powerful ESP32 (like an ESP32-S3) could theoretically run a very lightweight MQTT broker. It’s also conceivable, though complex and resource-intensive for an MCU, to implement a rudimentary bridging function on an ESP32 if it were connected to two different networks (e.g., Wi-Fi and Ethernet, or Wi-Fi and ESP-NOW) and needed to relay specific MQTT messages. This is far outside standard use cases and would require custom development beyond the esp-mqtt client library. For practical bridging, dedicated server-based brokers are used.
   • An ESP32-H2 acting as a Thread Border Router or Zigbee Gateway might internally translate device communications from those protocols into MQTT messages published to a local or remote broker. This gateway function has similarities to a protocol bridge.

In general, when working with MQTT bridges and proxies, the ESP32 developer’s main concern is ensuring the client is configured with the correct endpoint URI and security credentials/certificates for its immediate connection point.

Common Mistakes & Troubleshooting Tips

Exercises

1. Conceptual System Design with a Bridge:
   • Task: You have ESP32-based weather stations deployed in three different remote agricultural fields (Field A, Field B, Field C). Each field has its own local MQTT broker due to unreliable internet to a central point. You need to collect all weather data (temperature, humidity, rainfall) in a central cloud database via a cloud MQTT broker.
   • Activity:
     1. Draw a diagram of this system, showing ESP32s, local brokers, bridges, and the cloud broker.
     2. Define a consistent topic structure for ESP32 publications (e.g., weather/[field_id]/[sensor_type]).
     3. Describe the bridge configurations needed on each local broker to forward data to the cloud broker, including potential topic remapping (e.g., so cloud topics become farm_data/[field_id]/[sensor_type]).
2. Research: Cloud IoT Platform Endpoints:
   • Task: Choose one major cloud IoT platform (AWS IoT Core, Azure IoT Hub, or Google Cloud IoT Core).
   • Activity: Research and describe how devices typically connect to its MQTT interface. Does it present a single global endpoint, or regional endpoints? Does its architecture inherently use proxy-like mechanisms for scalability and security, even if not explicitly called a “proxy” in their basic connection guides? (Focus on the device-facing MQTT endpoint).
3. Proxy Configuration Impact (Thought Exercise):
   • Scenario: An ESP32 connects to an MQTT broker via a proxy. The proxy is initially configured for pass-through (no TLS termination). Later, the administrator changes the proxy to terminate TLS (clients connect with mqtts://, proxy connects to broker with mqtt://).
   • Task: What specific changes would be required in the ESP32’s esp_mqtt_client_config_t structure to adapt to this proxy configuration change? What files or data might need to be added to the ESP32 firmware?

Summary

• MQTT Bridges connect two or more MQTT brokers, enabling message flow between them for federation, cloud integration, or network segmentation. They act as specialized MQTT clients.
• MQTT Proxies are intermediaries between clients and brokers (or brokers themselves), providing services like load balancing, TLS termination, security enforcement, or logging.
• ESP32 devices typically act as standard MQTT clients; the bridge/proxy infrastructure is usually transparent to their core application logic, apart from connection endpoint and security (certificate) configurations.
• Bridge Configuration involves defining remote broker connections and topic mappings (direction, QoS, prefixes) on one of the brokers.
• Proxy Configuration for ESP32s mainly involves setting the URI to the proxy’s address and ensuring TLS trust if the proxy handles TLS.
• These architectures can add latency but provide significant benefits in scalability, manageability, and security for larger IoT systems.
• The fundamental MQTT operations on ESP32 variants remain consistent when interacting with such systems.

Further Reading

• Mosquitto Bridging Documentation: (Excellent example of bridge configuration)
  • man mosquitto.conf (if Mosquitto installed) or search for “Mosquitto Bridging man page” online.
  • https://mosquitto.org/documentation/
• HiveMQ Article on MQTT Proxy:
  • While specific proxy products vary, general concepts are often discussed in blogs. Search for “MQTT proxy server benefits” or “using Nginx as MQTT proxy.”
• EMQX Documentation on Bridging:
  • https://www.emqx.io/docs/en/latest/data-integration/bridges.html
• Cloud Provider MQTT Documentation:
  • AWS IoT Core – Connecting MQTT devices.
  • Azure IoT Hub – MQTT support.
  • Google Cloud IoT Core – MQTT protocol.
• NGINX for TCP/UDP Load Balancing (can be adapted for MQTT):
  • https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/