Chapter 124: Building Custom Cloud Backends

Chapter Objectives

Upon completing this chapter, you will be able to:

• Understand the motivations and trade-offs for building a custom IoT cloud backend.
• Identify the core components of a typical custom IoT backend.
• Learn about common technologies used for building these components (e.g., MQTT brokers, databases, API servers).
• Configure an ESP32 to securely communicate with a custom MQTT broker.
• Send telemetry data from an ESP32 to a custom backend via MQTT and HTTP.
• Receive commands on an ESP32 from a custom backend via MQTT.
• Appreciate the security, scalability, and maintenance considerations involved.
• Design basic interactions between an ESP32 and a self-hosted infrastructure.

Introduction

In the preceding chapters, we’ve explored connecting ESP32 devices to managed IoT cloud platforms like AWS IoT Core, Azure IoT Hub, and ESP RainMaker. These platforms offer convenience, scalability, and a rich set of features, abstracting away much of the backend complexity. However, there are scenarios where building your own custom cloud backend becomes a more suitable, or even necessary, choice.

A custom cloud backend gives you complete control over your IoT infrastructure, data, security policies, and feature set. It can be tailored to specific application needs, potentially reduce costs at a very large scale, avoid vendor lock-in, or allow for unique functionalities not readily available in off-the-shelf platforms. This chapter will introduce the fundamental concepts and components involved in building such a backend and, more importantly, how your ESP32 devices would interact with it. While we won’t build a full backend here (as that’s a significant undertaking in itself), we’ll focus on the ESP32’s role in this custom ecosystem.

Theory

Why Build a Custom IoT Backend?

Opting for a custom backend is a significant decision with several potential benefits and considerable responsibilities:

Motivations:

• Full Control: Tailor every aspect of the platform, from data models and APIs to security protocols and user experience.
• Data Ownership and Privacy: Keep sensitive data entirely within your infrastructure, addressing specific regulatory or privacy concerns.
• Cost Optimization (at scale): For very large deployments, a carefully designed custom backend might offer lower operational costs compared to per-message/per-device pricing of managed services.
• Avoiding Vendor Lock-in: Freedom to choose and change technologies without being tied to a specific cloud provider’s ecosystem.
• Unique Features: Implement specialized functionalities or integrations not offered by standard platforms.
• Legacy System Integration: Easier integration with existing on-premises or proprietary systems.
• Specific Security Requirements: Implement custom or highly stringent security measures beyond what standard platforms offer.

Considerations (Challenges):

• Development Complexity and Time: Building a robust, scalable, and secure backend is a complex software engineering task.
• Infrastructure Management: You are responsible for provisioning, managing, and maintaining servers, databases, and network infrastructure (whether on-premises or using IaaS from cloud providers like AWS EC2, Google Compute Engine, Azure VMs).
• Scalability: Designing the system to handle a growing number of devices and data volume requires careful planning.
• Security: You bear full responsibility for securing the entire stack, from device communication to data at rest and APIs. This is non-trivial.
• Reliability and Availability: Ensuring high uptime and fault tolerance is crucial and requires significant effort.
• Maintenance and Operations: Ongoing monitoring, updates, patching, and troubleshooting are necessary.
• Expertise Required: Needs a team with skills in backend development, database management, network engineering, and cybersecurity.

Core Components of a Custom IoT Backend

A typical custom IoT backend, regardless of the specific technologies chosen, will usually consist of the following key components:

graph TD
    subgraph "External Entities"
        direction LR
        IoTDevices["<center><b>IoT Devices (ESP32)</b></center>"]
        Users["<center><b>Users / Applications</b><br>(Web UI, Mobile Apps, Third-party Services)</center>"]
    end

    subgraph "Custom IoT Backend Infrastructure (Self-Hosted or IaaS)"
        direction TB
        LB["<center><b>Load Balancer / API Gateway</b></center>"] --- MQTTBroker["<center><b>MQTT Broker(s)</b><br><i>(e.g., Mosquitto, EMQX)</i></center>"]
        LB --- APIServer["<center><b>Application / API Server(s)</b><br><i>(e.g., Node.js/Express, Python/Flask, Java/Spring)</i></center>"]
        
        MQTTBroker --- AuthSvc["<center><b>Authentication &<br>Authorization Service</b></center>"]
        APIServer --- AuthSvc

        MQTTBroker --- DataProcessing["<center><b>Data Processing / Rule Engine</b><br><i>(e.g., Kafka+Spark, Custom Scripts, Serverless Functions)</i></center>"]
        APIServer --- DataProcessing
        
        DataProcessing --- DBs["<center><b>Database(s)</b></center>"]
        MQTTBroker --- DBs 
        APIServer --- DBs

        subgraph "Databases"
            direction LR
            TelemetryDB["<center><b>Telemetry Database</b><br><i>(e.g., InfluxDB, TimescaleDB)</i></center>"]
            MetadataDB["<center><b>Device Metadata & State DB</b><br><i>(e.g., PostgreSQL, MongoDB)</i></center>"]
            ConfigDB["<center><b>Configuration DB</b><br><i>(e.g., PostgreSQL, Redis)</i></center>"]
        end
        DBs -.-> TelemetryDB
        DBs -.-> MetadataDB
        DBs -.-> ConfigDB

        AdminPanel["<center><b>Admin Panel / Management UI</b></center>"] --- APIServer
    end
    
    IoTDevices -- "MQTT (TLS) / HTTP(S)" --> LB
    Users -- "HTTPS / WebSockets" --> LB 

    classDef device fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6
    classDef userApp fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46
    classDef backendComponent fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF
    classDef criticalComponent fill:#FEF3C7,stroke:#D97706,stroke-width:2px,color:#92400E % MQTT Broker, API Server, Auth
    classDef dataStore fill:#E0E7FF,stroke:#4338CA,stroke-width:1px,color:#3730A3

    class IoTDevices device;
    class Users userApp;
    class AdminPanel userApp;
    class LB backendComponent
    class DataProcessing backendComponent;
    class MQTTBroker criticalComponent;
    class APIServer criticalComponent;
    class AuthSvc criticalComponent;
    class DBs dataStore;
    class TelemetryDB dataStore;
    class MetadataDB dataStore;
    class ConfigDB dataStore;

1. Message Broker:
   • Role: The primary ingestion point for data from IoT devices. It handles receiving messages from many concurrently connected devices and routing them to other backend services or subscribing clients.
   • Protocol: MQTT is the de-facto standard for IoT messaging due to its lightweight nature and publish-subscribe model. CoAP can also be an option for constrained devices.
   • Examples: Mosquitto (popular open-source), EMQX (highly scalable, feature-rich open-source), VerneMQ (open-source, focuses on performance and reliability). HiveMQ (commercial).
   • Key Features: Support for MQTT v3.1.1/v5, TLS encryption, authentication mechanisms, message persistence, clustering for scalability.
2. Database(s):
   • Role: Storing various types of data.
   • Telemetry Data: Time-series databases are optimized for handling high volumes of timestamped data from sensors (e.g., temperature, humidity readings).
     • Examples: InfluxDB, TimescaleDB (PostgreSQL extension), Prometheus.
   • Device Metadata & State: Relational (SQL) or NoSQL databases store device information (ID, type, location, firmware version), user accounts, device configurations, and last known state.
     • Examples: PostgreSQL, MySQL (SQL); MongoDB, Cassandra (NoSQL).
   • Configuration Data: Storing application settings, rules, etc.
3. Application/API Server:
   • Role: Provides the business logic of your IoT application. It exposes APIs (typically RESTful HTTP/HTTPS or WebSocket) for:
     • Device provisioning and management (registration, de-registration).
     • User authentication and management.
     • Data retrieval for dashboards and mobile applications.
     • Sending commands to devices (often by publishing to the MQTT broker).
     • Integration with third-party services.
   • Technologies: Can be built using various languages and frameworks like Node.js (with Express.js/NestJS), Python (with Flask/Django), Go, Java (with Spring Boot), Ruby on Rails.
4. Device Authentication and Authorization Service:
   • Role: Ensures that only legitimate devices can connect and that they only have access to appropriate resources or topics.
   • Mechanisms:
     • Username/Password: Simple, but less secure for IoT.
     • Access Tokens (e.g., JWT): Devices authenticate to get a token, which is then used for subsequent requests/connections.
     • Client Certificates (X.509): Each device has a unique certificate for TLS client authentication. Highly secure but more complex to manage.
     • Pre-Shared Keys (PSK): Simpler than certificates but requires secure key distribution.
   • This service often integrates closely with the MQTT broker and API server.
5. Data Processing / Rule Engine (Optional but Common):
   • Role: Processes incoming telemetry data in real-time or batch mode to trigger alerts, perform analytics, or execute predefined rules (e.g., “if temperature > 30°C, send command to turn on fan”).
   • Technologies: Apache Kafka + Spark/Flink, custom scripts, serverless functions (e.g., AWS Lambda, Google Cloud Functions if using IaaS).
6. Frontend / User Interface (UI) / Admin Dashboard:
   • Role: Allows users to interact with their devices, view data, and manage settings. An admin dashboard provides tools for managing the entire system.
   • Technologies: Web frameworks (React, Angular, Vue.js) for web UIs, native development for mobile apps.

ESP32 Interaction with a Custom Backend

The ESP32 will primarily interact with:

• The MQTT Broker: For sending telemetry and receiving real-time commands. This connection is typically long-lived.
• The API Server: For tasks like initial device registration/provisioning (if not handled by other means), fetching configuration updates, or sending bulk/less frequent data via HTTP/HTTPS.

sequenceDiagram
    participant ESP32 as ESP32 Device
    participant WiFiRouter as Wi-Fi Router / Network
    participant CustomMQTTBroker as Custom MQTT Broker (e.g., Mosquitto)
    participant BackendApp as Backend Application / Subscriber

    ESP32->>WiFiRouter: 1. Connect to Wi-Fi Network
    WiFiRouter-->>ESP32: Wi-Fi Connected, IP Assigned

    ESP32->>CustomMQTTBroker: 2. Establish MQTT Connection (TCP or TLS)<br>Client ID: "esp32-device-01"<br>(Auth: Token/Cert/UserPass if configured)
    activate CustomMQTTBroker
    CustomMQTTBroker-->>ESP32: MQTT CONNACK (Connection Acknowledged)
    deactivate CustomMQTTBroker
    Note over ESP32, CustomMQTTBroker: Secure connection established

    ESP32->>CustomMQTTBroker: 3. Subscribe to Command Topic<br>Topic: "devices/esp32-device-01/commands" (QoS 1)
    activate CustomMQTTBroker
    CustomMQTTBroker-->>ESP32: MQTT SUBACK
    deactivate CustomMQTTBroker

    loop Periodically
        ESP32->>ESP32: 4a. Read Sensor Data (e.g., temperature)
        ESP32->>CustomMQTTBroker: 4b. Publish Telemetry<br>Topic: "devices/esp32-device-01/telemetry"<br>Payload: {"temperature": 25.5} (QoS 0 or 1)
    end

    BackendApp->>CustomMQTTBroker: (Subscribed to "devices/esp32-device-01/telemetry")
    CustomMQTTBroker-->>BackendApp: Relays Telemetry Message

    BackendApp->>CustomMQTTBroker: 5. Publish Command<br>Topic: "devices/esp32-device-01/commands"<br>Payload: "LED_ON" (QoS 1)
    activate CustomMQTTBroker
    CustomMQTTBroker-->>ESP32: 6. Relays Command Message (MQTT_EVENT_DATA)
    deactivate CustomMQTTBroker
    activate ESP32
    ESP32->>ESP32: 7. Process Command (e.g., Turn LED ON)
    deactivate ESP32

Security is Paramount:

• All communication between the ESP32 and the backend must be encrypted using TLS (for MQTT this is mqtts, for HTTP it’s https).
• The ESP32 must securely store its credentials (private keys for certificates, tokens, etc.). ESP32 flash encryption and NVS (Non-Volatile Storage) are essential here.

Practical Examples

We’ll focus on the ESP32 side, assuming you have a basic MQTT broker (like Mosquitto) and a simple HTTP server running for testing.

Setting up a Local Mosquitto MQTT Broker (Brief Guide)

For testing, you can install Mosquitto on your local machine or a Raspberry Pi.

1. Installation:
   • Linux: sudo apt-get install mosquitto mosquitto-clients
   • macOS: brew install mosquitto
   • Windows: Download installer from mosquitto.org
2. Basic Configuration (mosquitto.conf):
   • By default, Mosquitto allows anonymous connections on port 1883.
   • For TLS, you’ll need to generate certificates and configure listeners for port 8883. (This is beyond this chapter’s scope but crucial for production).
   # Example minimal mosquitto.conf for anonymous access persistence true persistence_location /var/lib/mosquitto/ log_dest file /var/log/mosquitto/mosquitto.log allow_anonymous true listener 1883
3. Running Mosquitto: Usually starts automatically as a service. If not: mosquitto -c /path/to/mosquitto.conf

Warning: Using anonymous MQTT access is highly insecure and suitable only for isolated local testing. Always use authentication and TLS in any real-world scenario.

Code Snippet 1: ESP32 Connecting to Custom MQTT Broker

This example uses the esp-mqtt client to connect to an MQTT broker, publish telemetry, and subscribe to a command topic.

main/main.c:

#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"
#include "mqtt_client.h"
#include "driver/gpio.h" // For LED control

// Wi-Fi Configuration
#define WIFI_SSID CONFIG_WIFI_SSID
#define WIFI_PASS CONFIG_WIFI_PASSWORD

// Custom MQTT Broker Configuration
#define MQTT_BROKER_URI CONFIG_MQTT_BROKER_URI // e.g., "mqtt://192.168.1.100" or "mqtts://your.domain.com" for TLS
#define MQTT_DEVICE_ID CONFIG_MQTT_DEVICE_ID   // e.g., "esp32-device-01"

// For TLS, you'd embed the server's root CA certificate
// extern const uint8_t server_root_ca_pem_start[] asm("_binary_server_root_ca_pem_start");
// extern const uint8_t server_root_ca_pem_end[]   asm("_binary_server_root_ca_pem_end");

#define LED_GPIO CONFIG_LED_GPIO

static const char *TAG = "CUSTOM_MQTT_EXAMPLE";
static EventGroupHandle_t wifi_event_group;
const int WIFI_CONNECTED_BIT = BIT0;
static esp_mqtt_client_handle_t mqtt_client_handle = NULL;
static bool led_state = false;

static void wifi_event_handler(void* arg, esp_event_base_t event_base, int32_t event_id, void* event_data) {
    // (Identical to previous chapters - connect, handle disconnect, got_ip)
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        ESP_LOGI(TAG, "Disconnected from Wi-Fi. Retrying...");
        // esp_mqtt_client_stop(mqtt_client_handle); // Stop MQTT client before Wi-Fi reconnect might be needed
        esp_wifi_connect();
        xEventGroupClearBits(wifi_event_group, WIFI_CONNECTED_BIT);
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        ESP_LOGI(TAG, "Got IP address: " IPSTR, IP2STR(&event->ip_info.ip));
        xEventGroupSetBits(wifi_event_group, WIFI_CONNECTED_BIT);
        if (mqtt_client_handle) {
            esp_mqtt_client_start(mqtt_client_handle); // Start MQTT client once IP is obtained
        }
    }
}

void wifi_init_sta(void) {
    // (Identical to previous chapters)
    wifi_event_group = xEventGroupCreate();
    ESP_ERROR_CHECK(esp_netif_init()); ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta(); wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    esp_event_handler_instance_t instance_any_id, instance_got_ip;
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &wifi_event_handler, NULL, &instance_any_id));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &wifi_event_handler, NULL, &instance_got_ip));
    wifi_config_t wifi_config = { .sta = { .ssid = WIFI_SSID, .password = WIFI_PASS, .threshold.authmode = WIFI_AUTH_WPA2_PSK, }, };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA)); ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_STA, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start()); ESP_LOGI(TAG, "wifi_init_sta finished.");
    xEventGroupWaitBits(wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdFALSE, portMAX_DELAY);
    ESP_LOGI(TAG, "Connected to AP SSID:%s", WIFI_SSID);
}

static void mqtt_event_handler_cb(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data) {
    esp_mqtt_event_handle_t event = event_data;
    // esp_mqtt_client_handle_t client = event->client; // mqtt_client_handle can be used
    char topic_buffer[128];
    char data_buffer[128];

    switch ((esp_mqtt_event_id_t)event_id) {
        case MQTT_EVENT_CONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED to %s", MQTT_BROKER_URI);
            // Subscribe to command topic
            snprintf(topic_buffer, sizeof(topic_buffer), "devices/%s/commands", MQTT_DEVICE_ID);
            int msg_id = esp_mqtt_client_subscribe(mqtt_client_handle, topic_buffer, 1); // QoS 1
            ESP_LOGI(TAG, "Sent subscribe successful for topic %s, msg_id=%d", topic_buffer, msg_id);
            break;
        case MQTT_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
            break;
        case MQTT_EVENT_SUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_UNSUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_UNSUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_PUBLISHED:
            ESP_LOGI(TAG, "MQTT_EVENT_PUBLISHED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_DATA:
            ESP_LOGI(TAG, "MQTT_EVENT_DATA");
            // Copy topic and data to ensure null termination and prevent buffer overflows
            strncpy(topic_buffer, event->topic, event->topic_len < sizeof(topic_buffer) -1 ? event->topic_len : sizeof(topic_buffer) -1);
            topic_buffer[event->topic_len < sizeof(topic_buffer) -1 ? event->topic_len : sizeof(topic_buffer) -1] = '\0';
            
            strncpy(data_buffer, event->data, event->data_len < sizeof(data_buffer) -1 ? event->data_len : sizeof(data_buffer) -1);
            data_buffer[event->data_len < sizeof(data_buffer) -1 ? event->data_len : sizeof(data_buffer) -1] = '\0';

            ESP_LOGI(TAG, "TOPIC=%.*s", event->topic_len, event->topic);
            ESP_LOGI(TAG, "DATA=%.*s", event->data_len, event->data);

            char expected_command_topic[128];
            snprintf(expected_command_topic, sizeof(expected_command_topic), "devices/%s/commands", MQTT_DEVICE_ID);
            if (strcmp(topic_buffer, expected_command_topic) == 0) {
                if (strcmp(data_buffer, "LED_ON") == 0) {
                    ESP_LOGI(TAG, "Command: LED ON");
                    led_state = true;
                    gpio_set_level(LED_GPIO, 1);
                } else if (strcmp(data_buffer, "LED_OFF") == 0) {
                    ESP_LOGI(TAG, "Command: LED OFF");
                    led_state = false;
                    gpio_set_level(LED_GPIO, 0);
                } else {
                    ESP_LOGW(TAG, "Unknown command: %s", data_buffer);
                }
            }
            break;
        case MQTT_EVENT_ERROR:
            ESP_LOGE(TAG, "MQTT_EVENT_ERROR");
            if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
                ESP_LOGE(TAG, "Last error code reported from esp-tls: 0x%x", event->error_handle->esp_tls_last_esp_err);
                ESP_LOGE(TAG, "Last error code reported from tls stack: 0x%x", event->error_handle->esp_tls_stack_err);
            } else if (event->error_handle->error_type == MQTT_ERROR_TYPE_CONNECTION_REFUSED) {
                ESP_LOGE(TAG, "Connection refused error: 0x%x", event->error_handle->connect_return_code);
            } else {
                ESP_LOGW(TAG, "Other error type: 0x%x", event->error_handle->error_type);
            }
            break;
        default:
            ESP_LOGI(TAG, "Other event id:%d", event->event_id);
            break;
    }
}

static void mqtt_app_start(void) {
    const esp_mqtt_client_config_t mqtt_cfg = {
        .broker.address.uri = MQTT_BROKER_URI,
        // For TLS:
        // .broker.verification.certificate = (const char *)server_root_ca_pem_start,
        // .broker.verification.certificate_len = server_root_ca_pem_end - server_root_ca_pem_start,
        // For username/password auth:
        // .credentials.username = "my_device_user",
        // .credentials.authentication.password = "my_device_password",
        .credentials.client_id = MQTT_DEVICE_ID, // Must be unique per client connecting to the broker
    };

    mqtt_client_handle = esp_mqtt_client_init(&mqtt_cfg);
    if (mqtt_client_handle == NULL) {
        ESP_LOGE(TAG, "Failed to init MQTT client");
        return;
    }
    esp_mqtt_client_register_event(mqtt_client_handle, ESP_EVENT_ANY_ID, mqtt_event_handler_cb, NULL);
    // esp_mqtt_client_start(mqtt_client_handle); // Start is called after Wi-Fi connects
}

void app_main(void) {
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    // Configure LED GPIO
    gpio_reset_pin(LED_GPIO);
    gpio_set_direction(LED_GPIO, GPIO_MODE_OUTPUT);
    gpio_set_level(LED_GPIO, led_state);


    wifi_init_sta(); // This will block until Wi-Fi is connected

    mqtt_app_start(); // Initialize MQTT client, it will start upon IP_EVENT_STA_GOT_IP

    // Main loop to publish telemetry periodically
    char telemetry_topic[128];
    snprintf(telemetry_topic, sizeof(telemetry_topic), "devices/%s/telemetry", MQTT_DEVICE_ID);
    int msg_count = 0;

    while (1) {
        if (xEventGroupGetBits(wifi_event_group) & WIFI_CONNECTED_BIT) { // Check if Wi-Fi is connected
            char telemetry_payload[128];
            float temperature = 20.0 + ((float)esp_random() / (float)UINT32_MAX) * 10.0; // Simulated
            snprintf(telemetry_payload, sizeof(telemetry_payload),
                     "{\"messageId\":%d, \"deviceId\":\"%s\", \"temperature\":%.2f, \"ledState\":%s}",
                     msg_count++, MQTT_DEVICE_ID, temperature, led_state ? "true" : "false");

            // Only publish if MQTT client is started and connected (implicitly handled by esp_mqtt_client_publish)
            int msg_id = esp_mqtt_client_publish(mqtt_client_handle, telemetry_topic, telemetry_payload, 0, 0, 0); // QoS 0
            if (msg_id != -1) {
                ESP_LOGI(TAG, "Sent telemetry publish successful, msg_id=%d, payload: %s", msg_id, telemetry_payload);
            } else {
                ESP_LOGW(TAG, "Failed to publish telemetry (MQTT client might not be connected yet or broker down).");
            }
        } else {
            ESP_LOGW(TAG, "Wi-Fi not connected. Skipping telemetry publish.");
        }
        vTaskDelay(pdMS_TO_TICKS(15000)); // Publish every 15 seconds
    }
}

main/Kconfig.projbuild (Create this file if it doesn’t exist):

menu "Example Configuration"
    config WIFI_SSID
        string "Wi-Fi SSID"
        default "YourSSID"

    config WIFI_PASSWORD
        string "Wi-Fi Password"
        default "YourPassword"
    
    config MQTT_BROKER_URI
        string "MQTT Broker URI"
        default "mqtt://192.168.1.100" # Replace with your broker's IP/hostname
        help
            URI of the MQTT broker.
            For non-TLS: mqtt://host:port
            For TLS: mqtts://host:port
            For WS: ws://host:port/ws
            For WSS: wss://host:port/ws

    config MQTT_DEVICE_ID
        string "MQTT Device ID (Client ID)"
        default "esp32-custom-01"
        help
            Unique client ID for this device.

    config LED_GPIO
        int "LED GPIO Pin"
        default 2
        help
            GPIO pin connected to an LED for command testing.

endmenu

Component File for CA Certificate (Optional, for TLS):

If using TLS with a self-signed certificate for your broker or a private CA:

1. Create a directory main/certs.
2. Place your broker’s CA certificate file (e.g., server_root_ca.pem) in main/certs.
3. Modify main/CMakeLists.txt:idf_component_register(SRCS "main.c" INCLUDE_DIRS "." EMBED_FILES "certs/server_root_ca.pem") # The symbol will be _binary_server_root_ca_pem_start

Code Snippet 2: ESP32 Sending Data via HTTP POST

This snippet shows how to send data to a hypothetical HTTP API endpoint.

(Requires esp_http_client component, usually included by default).

// Add to includes in main.c
#include "esp_http_client.h"

// ... (other parts of main.c)

esp_err_t http_post_data(const char *url, const char *json_payload) {
    esp_http_client_config_t config = {
        .url = url,
        .method = HTTP_METHOD_POST,
        .event_handler = NULL, // Add an event handler for more detailed response/error handling
        // For HTTPS:
        // .cert_pem = server_api_ca_pem_start, // If using custom CA for API server
        // .skip_cert_common_name_check = true, // If CN doesn't match URL (dev only)
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);
    if (client == NULL) {
        ESP_LOGE(TAG, "Failed to initialise HTTP client");
        return ESP_FAIL;
    }

    esp_http_client_set_header(client, "Content-Type", "application/json");
    esp_http_client_set_post_field(client, json_payload, strlen(json_payload));

    esp_err_t err = esp_http_client_perform(client);
    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTP POST Status = %d, content_length = %"PRId64,
                 esp_http_client_get_status_code(client),
                 esp_http_client_get_content_length(client));
        // You can read response data here if needed
        // char response_buffer[1024] = {0};
        // int read_len = esp_http_client_read_response(client, response_buffer, sizeof(response_buffer)-1);
        // if (read_len > 0) {
        //     ESP_LOGI(TAG, "HTTP Response: %s", response_buffer);
        // }
    } else {
        ESP_LOGE(TAG, "HTTP POST request failed: %s", esp_err_to_name(err));
    }

    esp_http_client_cleanup(client);
    return err;
}

// Example usage in your main loop or a task:
// if (xEventGroupGetBits(wifi_event_group) & WIFI_CONNECTED_BIT) {
//     char http_payload[100];
//     snprintf(http_payload, sizeof(http_payload), "{\"sensor\":\"temp\", \"value\":25.5, \"deviceId\":\"%s\"}", MQTT_DEVICE_ID);
//     http_post_data("http://your-api-server.com/api/data", http_payload); // Replace with your API URL
// }

Build Instructions

1. Save all files.
2. Open ESP-IDF Terminal in VS Code.
3. Configure:idf.py menuconfig.
   • Set Wi-Fi SSID and Password.
   • Set MQTT Broker URI (e.g., mqtt://<your_broker_ip>).
   • Set MQTT Device ID.
   • Set LED GPIO.
4. Build: idf.py build

Run/Flash/Observe Steps

1. Ensure your custom MQTT broker is running and accessible from your ESP32’s network.
2. Connect ESP32 device.
3. Flash: idf.py -p /dev/ttyUSB0 flash monitor (adjust port).
4. Observe Logs on ESP32:
   • Wi-Fi connection.
   • MQTT connection attempts and status.
   • Published telemetry messages.
5. Monitor MQTT Broker:
   • Use an MQTT client tool (like mqttx, MQTT Explorer, or mosquitto_sub) to subscribe to the telemetry topic (e.g., devices/esp32-custom-01/telemetry) and see the data.
   • Publish a command (e.g., LED_ON or LED_OFF) to the command topic (e.g., devices/esp32-custom-01/commands) and observe the ESP32’s LED and logs.
6. For HTTP POST: Ensure your simple HTTP server is running and check its logs for incoming data from the ESP32.

Variant Notes

• ESP32, ESP32-S2, ESP32-S3, ESP32-C3, ESP32-C6: All these Wi-Fi enabled variants are well-suited for connecting to custom backends using MQTT and HTTP.
  • TLS Resource Usage: Implementing TLS for secure communication will consume additional RAM and flash, and increase CPU load for cryptographic operations. This is more critical on resource-constrained variants like the ESP32-C3. Always enable relevant mbedTLS configurations (e.g., for specific ciphers, X.509 certificate parsing).
  • JSON Processing: Parsing and generating JSON payloads also consumes resources. For very constrained scenarios, consider more compact binary formats like Protocol Buffers or MessagePack (covered in Chapter 119), though this adds complexity to your backend.
• ESP32-H2: Lacks built-in Wi-Fi. To connect to a custom cloud backend, it would typically require:
  • An IP-based Thread/Zigbee border router that bridges its 802.15.4 network to your LAN/internet.
  • A gateway device (e.g., another ESP32 with Wi-Fi) that receives data from the ESP32-H2 (via BLE, ESP-NOW, or 802.15.4) and then relays it to the custom backend. The ESP32-H2 itself would run a client for the local communication protocol.

Common Mistakes & Troubleshooting Tips

Exercises

1. Secure MQTT Setup:
   • Configure your local Mosquitto broker to use TLS and basic username/password authentication.
   • Modify the ESP32 MQTT example to connect securely using these credentials and the necessary CA certificate.
2. Bi-directional MQTT Control:
   • Expand the ESP32 example: add another controllable element (e.g., a relay or a PWM output for dimming the LED).
   • Define new MQTT command topics/payloads for this element.
   • Implement the ESP32 logic to respond to these commands and also report the new element’s status via telemetry.
3. HTTP Data Reception on ESP32:
   • Implement a simple HTTP GET endpoint on your ESP32 (using esp_http_server, covered in Chapter 112).
   • From your custom backend (e.g., a Python script), send a GET request to this ESP32 endpoint to retrieve its current status or trigger an action.
4. Custom API Integration:
   • Set up a very simple API endpoint using Python Flask or Node.js Express on your computer that accepts JSON POST requests.
   • Modify the ESP32 HTTP example to send its simulated temperature data to this local API endpoint. The API should print the received data.
5. Topic Design and Database Schema:
   • Consider a scenario: a smart building with multiple rooms, each having temperature sensors, light switches, and occupancy sensors.
   • Design an MQTT topic hierarchy for this scenario.
   • Outline a basic database schema (e.g., for PostgreSQL or InfluxDB) to store the telemetry and device metadata. Justify your choices.

Summary

• Building a custom IoT backend offers maximum control and flexibility but comes with significant development, management, and security responsibilities.
• Core components include an MQTT broker, database(s), an application/API server, and authentication services.
• ESP32 devices typically interact with custom backends via MQTT (for real-time data/commands) and HTTP/HTTPS (for configuration, less frequent data).
• Security is paramount: always use TLS, secure credential storage on the ESP32, and robust authentication mechanisms.
• Careful planning for scalability, reliability, and maintenance is essential for a production-grade custom backend.
• Wi-Fi enabled ESP32 variants can readily connect to custom backends, while non-Wi-Fi variants like ESP32-H2 require gateway solutions.

Further Reading

• MQTT Protocol:
  • MQTT Official Website: https://mqtt.org/
  • HiveMQ MQTT Essentials: https://www.hivemq.com/mqtt-essentials/
• Open-Source MQTT Brokers:
  • Mosquitto: https://mosquitto.org/
  • EMQX: https://www.emqx.io/
• Time-Series Databases:
  • InfluxDB: https://www.influxdata.com/
  • TimescaleDB: https://www.timescale.com/
• Backend Frameworks (Examples):
  • Node.js with Express.js: https://expressjs.com/
  • Python with Flask: https://flask.palletsprojects.com/
• ESP-IDF Documentation:
  • MQTT Client: https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-reference/protocols/mqtt.html
  • HTTP Client: https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-reference/protocols/esp_http_client.html
  • NVS (Non-Volatile Storage): https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/api-reference/storage/nvs_flash.html
  • Flash Encryption: https://docs.espressif.com/projects/esp-idf/en/v5.4/esp32/security/flash-encryption.html