Leave a Comment / By /

Chapter 125: Implementing Device Shadows and Digital Twins

Chapter Objectives

Upon completing this chapter, you will be able to:

  • Understand the concept of a Device Shadow and its importance in IoT applications.
  • Differentiate between a Device Shadow and a more comprehensive Digital Twin.
  • Explain the typical structure of a Device Shadow document (reported, desired, delta states).
  • Understand how devices interact with shadows using protocols like MQTT.
  • Implement ESP32 firmware to report its state to a Device Shadow.
  • Implement ESP32 firmware to subscribe to and act upon desired state changes from a shadow.
  • Recognize the benefits of using shadows for handling intermittent connectivity and state synchronization.
  • Appreciate the role of shadows in enabling applications to interact with devices even when they are offline.
  • Identify common patterns and best practices for working with Device Shadows.

Introduction

In our exploration of IoT cloud connectivity, we’ve seen how ESP32 devices can send telemetry to and receive commands from various cloud platforms. However, real-world IoT devices often face challenges like intermittent network connectivity. How can an application reliably get the last known state of a device, or set a new desired state, if the device is temporarily offline? This is where the concept of a Device Shadow (or a similar mechanism like a Device Twin) becomes invaluable.

A Device Shadow is essentially a persistent, virtual representation of your device’s state, stored and managed in the cloud. It acts as an intermediary, decoupling the device from the applications that interact with it. This chapter will delve into the theory behind Device Shadows and the broader concept of Digital Twins, and then demonstrate how an ESP32 device can effectively utilize this pattern for robust state management and synchronization with the cloud.

Theory

What is a Device Shadow?

A Device Shadow is a JSON document stored in the cloud that reflects the current (reported) and intended (desired) state of a physical IoT device. Think of it as a “cloud copy” or a “virtual counterpart” of your device’s key attributes.

Key Purposes and Benefits:

  1. Handling Intermittent Connectivity: Devices might not always be online. Applications can read the device’s last reported state from the shadow or set a desired state in the shadow. When the device reconnects, it can retrieve the desired state and update its actual state accordingly.
  2. Decoupling Devices and Applications: Applications don’t need to communicate directly with the device in real-time for every state query or update. They interact with the shadow, and the cloud platform handles synchronization with the actual device when it’s available.
  3. State Persistence: The shadow stores the device’s state even if the device loses power or reboots.
  4. Simplified Application Logic: Application developers can build experiences assuming the device state is always accessible via the shadow.
  5. Remote Configuration and Control: Desired states can be set in the shadow, and the device will pick them up and apply them when it connects.

Device Shadow Structure

Most Device Shadow implementations (like AWS IoT Device Shadow or Azure IoT Hub Device Twin) use a JSON document with a structure typically including:

  • state: An object containing the core state information.
    • reported: An object where the device writes its current actual state. For example, { "temperature": 25.5, "led_on": true }. Applications read this to know the device’s last known status.
    • desired: An object where applications write the state they want the device to be in. For example, { "led_on": false, "telemetry_interval": 60 }. The device reads this and attempts to conform to it.
    • delta: (Often automatically generated by the cloud platform) An object representing the difference between the desired and reported states. This is useful for the device to quickly see what changes it needs to apply. For example, if desired: {"led_on": false} and reported: {"led_on": true}, the delta might be {"led_on": false}.
  • metadata: Information about the state properties, such as timestamps of when they were last updated.
  • version: A version number that increments with each update, used for optimistic locking and conflict resolution.

Example Shadow Document:

JSON
{
  "state": {
    "desired": {
      "led_on": false,
      "brightness": 75,
      "telemetry_interval": 300
    },
    "reported": {
      "led_on": true,
      "brightness": 50,
      "firmware_version": "v1.2.0",
      "ip_address": "192.168.1.105",
      "telemetry_interval": 600
    },
    "delta": { // This would be generated by the platform
      "led_on": false,
      "brightness": 75,
      "telemetry_interval": 300
    }
  },
  "metadata": {
    "desired": {
      "led_on": { "timestamp": 1678886400 },
      "brightness": { "timestamp": 1678886400 },
      "telemetry_interval": { "timestamp": 1678886400 }
    },
    "reported": {
      "led_on": { "timestamp": 1678886300 },
      "brightness": { "timestamp": 1678886300 },
      "firmware_version": { "timestamp": 1678886000 },
      "ip_address": { "timestamp": 1678886000 },
      "telemetry_interval": { "timestamp": 1678886000 }
    }
  },
  "version": 15,
  "timestamp": 1678886405
}

Device Shadow Interaction (Typically via MQTT)

Devices and applications usually interact with the shadow service using a publish-subscribe protocol like MQTT, over specific, well-defined topics. For example, AWS IoT uses topics like:

  • Device Reporting State:
    • Publish to: $aws/things/{thingName}/shadow/update
    • Payload: { "state": { "reported": { "your_param": "value" } } }
  • Getting Current Shadow State:
    • Publish an empty message to: $aws/things/{thingName}/shadow/get
    • Receive state on: $aws/things/{thingName}/shadow/get/accepted or /rejected
  • Receiving Desired State Changes (Delta):
    • Subscribe to: $aws/things/{thingName}/shadow/update/delta
    • When the desired state changes and differs from reported, the platform publishes the delta here.
  • Notifying Cloud of Applied Desired State:
    • After applying a desired change, the device should update its reported state. This clears the delta.
    • Optionally, it can also clear the desired value by publishing null to it:{ “state”: { “desired”: { “your_param”: null } } }
sequenceDiagram
    participant App as Cloud Application / User
    participant ShadowSvc as Device Shadow Service (Cloud)
    participant ESP32 as ESP32 Device

    App->>ShadowSvc: 1. Update Desired State<br><tt>{"state": {"desired": {"led_on": false}}}</tt><br>(Publishes to shadow update topic)
    activate ShadowSvc
    ShadowSvc->>ShadowSvc: 2. Stores Desired State<br>Calculates Delta
    ShadowSvc-->>ESP32: 3. Publish Delta State<br>Topic: <tt>.../shadow/update/delta</tt><br>Payload: <tt>{"state": {"led_on": false}, "version": X}</tt>
    deactivate ShadowSvc
    activate ESP32

    ESP32->>ESP32: 4. Receives Delta
    ESP32->>ESP32: 5. Applies Changes to Hardware<br>(e.g., turns LED OFF)
    ESP32-->>ShadowSvc: 6. Update Reported State<br>Topic: <tt>.../shadow/update</tt><br>Payload: <tt>{"state": {"reported": {"led_on": false}}, "version": X}</tt>
    deactivate ESP32
    activate ShadowSvc

    ShadowSvc->>ShadowSvc: 7. Stores Reported State<br>Delta is now resolved (desired == reported)
    ShadowSvc-->>App: 8. (Optional) Notify App of State Change<br>(e.g., via subscription to <tt>.../shadow/update/accepted</tt>)
    deactivate ShadowSvc

    App->>ShadowSvc: 9. (Later) Get Current State<br>(Publishes to <tt>.../shadow/get</tt>)
    activate ShadowSvc
    ShadowSvc-->>App: 10. Returns Full Shadow Document<br>Topic: <tt>.../shadow/get/accepted</tt><br>Payload: <tt>{"state": {"reported": ..., "desired": ...}, ...}</tt>
    deactivate ShadowSvc

Device Shadow vs. Digital Twin

While “Device Shadow” and “Device Twin” are often used interchangeably, especially in services like Azure IoT Hub Device Twin, a Digital Twin is generally a broader and more comprehensive concept.

  • Device Shadow: Primarily focused on the state synchronization (current and desired) of a device. It’s a lightweight, data-centric representation.
  • Digital Twin: A dynamic, virtual representation of a physical asset, process, or system. It can include:
    • The device’s state (like a shadow).
    • Models: Physical models, behavioral models, simulation models.
    • Analytics: Data analysis performed on the twin’s data.
    • Relationships: Connections to other twins, systems, or processes.
    • Lifecycle Data: Manufacturing data, maintenance history, etc.
    • It aims to provide a holistic view and allow for complex simulations, predictions, and optimizations.

A Device Shadow can be considered a foundational component or a specific implementation pattern within a larger Digital Twin strategy. For many IoT applications focused on state management and remote control, the functionality provided by a “Device Shadow” is often what’s practically implemented and sufficient.

Feature / Aspect Device Shadow Digital Twin (Broader Concept)
Primary Focus State synchronization (reported, desired, delta) of a device. Lightweight data representation. Holistic, dynamic virtual representation of a physical asset, process, or system. Can include much more than just state.
Data Content Typically JSON document with device state attributes, metadata, and version. Can include state, physical models, behavioral models, simulation data, analytics results, lifecycle data (maintenance history, manufacturing details), relationships to other twins/systems.
Key Purpose Handle intermittent connectivity, decouple devices from applications, enable remote control and configuration through state persistence. Simulation, prediction, optimization, advanced analytics, understanding complex interactions, “what-if” scenarios, operational efficiency improvements.
Implementation Complexity Relatively simpler to implement; many IoT platforms offer this as a standard feature (e.g., AWS IoT Device Shadow, Azure IoT Hub Device Twin for state). Can be significantly more complex, requiring integration of various data sources, modeling tools, and analytical engines.
Scope Often focused on a single device’s current and intended operational parameters. Can represent individual assets, complex systems of assets, or even entire processes or environments.
Analogy A device’s “status board” or “cloud memory” for its settings and readings. A “living, breathing, virtual replica” used for deep understanding and interaction.
Relationship A Device Shadow can be considered a key component or a specific implementation pattern within a Digital Twin. A Digital Twin often incorporates Device Shadow functionality for its state management aspects.

Platforms like AWS IoT Core use the term “Device Shadow,” while Azure IoT Hub uses “Device Twin” for a similar state management purpose but also provides features that lean towards the broader Digital Twin concept. ESP RainMaker also implements a shadow-like mechanism for state persistence and synchronization.

Practical Examples

This example will demonstrate how an ESP32 can interact with a generic Device Shadow mechanism using MQTT. We’ll simulate the shadow interaction by defining MQTT topics and payload structures similar to common platforms. This allows us to focus on the ESP32 logic.

Assume we have an MQTT broker and a backend service (or are using a platform like AWS/Azure that provides shadow services) that manages the shadow document and exposes the following topics for a device named esp32-smart-light:

  • Update Reported State: devices/esp32-smart-light/shadow/update (Device publishes here)
  • Receive Desired/Delta State: devices/esp32-smart-light/shadow/update/delta (Device subscribes here)
  • Get Current Shadow: devices/esp32-smart-light/shadow/get (Device publishes to trigger)
  • Receive Full Shadow on Get: devices/esp32-smart-light/shadow/get/accepted (Device subscribes to get response)

Prerequisites

  1. ESP-IDF v5.x environment with VS Code.
  2. An MQTT broker accessible to the ESP32 (e.g., a local Mosquitto, or a cloud MQTT service).
  3. An MQTT client tool (MQTTX, MQTT Explorer) for simulating cloud/application interactions with the shadow.
  4. Wi-Fi credentials.

Code Snippet: ESP32 Interacting with a Device Shadow

main/main.c:

C
#include <stdio.h>
#include <string.h>
#include <cJSON.h> // For parsing and creating JSON
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"
#include "mqtt_client.h"
#include "driver/gpio.h"

// Configuration
#define WIFI_SSID CONFIG_WIFI_SSID
#define WIFI_PASS CONFIG_WIFI_PASSWORD
#define MQTT_BROKER_URI CONFIG_MQTT_BROKER_URI
#define DEVICE_ID CONFIG_DEVICE_ID // e.g., "esp32-smart-light"
#define LED_GPIO CONFIG_LED_GPIO

static const char *TAG = "SHADOW_EXAMPLE";
static EventGroupHandle_t wifi_event_group;
const int WIFI_CONNECTED_BIT = BIT0;
static esp_mqtt_client_handle_t mqtt_client_handle = NULL;

// Device state variables
static bool g_led_state = false; // Actual hardware state
static int g_brightness = 50;   // Actual hardware state
static int g_telemetry_interval_sec = 60; // Actual operational parameter

// Shadow topics
char shadow_update_topic[128];
char shadow_delta_topic[128];
char shadow_get_topic[128];
char shadow_get_accepted_topic[128];

// Forward declaration
static void report_current_state_to_shadow(void);

static void wifi_event_handler(void* arg, esp_event_base_t event_base, int32_t event_id, void* event_data) {
    // (Identical to previous chapters)
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        ESP_LOGI(TAG, "Disconnected from Wi-Fi. Retrying...");
        esp_wifi_connect();
        xEventGroupClearBits(wifi_event_group, WIFI_CONNECTED_BIT);
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        ESP_LOGI(TAG, "Got IP address: " IPSTR, IP2STR(&event->ip_info.ip));
        xEventGroupSetBits(wifi_event_group, WIFI_CONNECTED_BIT);
        if (mqtt_client_handle) {
            esp_mqtt_client_start(mqtt_client_handle);
        }
    }
}

void wifi_init_sta(void) {
    // (Identical to previous chapters)
    wifi_event_group = xEventGroupCreate();
    ESP_ERROR_CHECK(esp_netif_init()); ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta(); wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    esp_event_handler_instance_t instance_any_id, instance_got_ip;
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &wifi_event_handler, NULL, &instance_any_id));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &wifi_event_handler, NULL, &instance_got_ip));
    wifi_config_t wifi_config = { .sta = { .ssid = WIFI_SSID, .password = WIFI_PASS, .threshold.authmode = WIFI_AUTH_WPA2_PSK, }, };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA)); ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_STA, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start()); ESP_LOGI(TAG, "wifi_init_sta finished.");
    xEventGroupWaitBits(wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdFALSE, portMAX_DELAY);
    ESP_LOGI(TAG, "Connected to AP SSID:%s", WIFI_SSID);
}

// Function to apply changes from desired/delta state
static void apply_desired_state(cJSON *state_obj) {
    bool state_changed = false;
    cJSON *led_item = cJSON_GetObjectItem(state_obj, "led_on");
    if (cJSON_IsBool(led_item)) {
        bool desired_led_state = cJSON_IsTrue(led_item);
        if (g_led_state != desired_led_state) {
            g_led_state = desired_led_state;
            gpio_set_level(LED_GPIO, g_led_state);
            ESP_LOGI(TAG, "Applied desired state: LED is now %s", g_led_state ? "ON" : "OFF");
            state_changed = true;
        }
    }

    cJSON *brightness_item = cJSON_GetObjectItem(state_obj, "brightness");
    if (cJSON_IsNumber(brightness_item)) {
        int desired_brightness = brightness_item->valueint;
        if (desired_brightness >= 0 && desired_brightness <= 100) {
            if (g_brightness != desired_brightness) {
                g_brightness = desired_brightness;
                // Add actual PWM control for brightness here if applicable
                ESP_LOGI(TAG, "Applied desired state: Brightness is now %d%%", g_brightness);
                state_changed = true;
            }
        }
    }
    
    cJSON *interval_item = cJSON_GetObjectItem(state_obj, "telemetry_interval");
    if (cJSON_IsNumber(interval_item)) {
        int desired_interval = interval_item->valueint;
        if (desired_interval > 0) {
            if (g_telemetry_interval_sec != desired_interval) {
                g_telemetry_interval_sec = desired_interval;
                ESP_LOGI(TAG, "Applied desired state: Telemetry interval is now %d seconds", g_telemetry_interval_sec);
                // Adjust actual telemetry task timing here
                state_changed = true;
            }
        }
    }

    if (state_changed) {
        report_current_state_to_shadow(); // Report back the new actual state
    }
}

static void mqtt_event_handler_cb(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data) {
    esp_mqtt_event_handle_t event = event_data;
    cJSON *json_payload = NULL;

    switch ((esp_mqtt_event_id_t)event_id) {
        case MQTT_EVENT_CONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
            // Subscribe to delta topic and get/accepted topic
            esp_mqtt_client_subscribe(mqtt_client_handle, shadow_delta_topic, 1);
            ESP_LOGI(TAG, "Subscribed to %s", shadow_delta_topic);
            esp_mqtt_client_subscribe(mqtt_client_handle, shadow_get_accepted_topic, 1);
            ESP_LOGI(TAG, "Subscribed to %s", shadow_get_accepted_topic);

            // Request the full shadow document on connection to sync up
            esp_mqtt_client_publish(mqtt_client_handle, shadow_get_topic, "{}", 0, 1, 0); 
            ESP_LOGI(TAG, "Published to %s to get current shadow", shadow_get_topic);
            break;
        case MQTT_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
            break;
        case MQTT_EVENT_SUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_DATA:
            ESP_LOGI(TAG, "MQTT_EVENT_DATA from topic: %.*s", event->topic_len, event->topic);
            ESP_LOGD(TAG, "Data: %.*s", event->data_len, event->data);

            // Null-terminate the data to parse as string
            char *data_str = malloc(event->data_len + 1);
            if (!data_str) {
                ESP_LOGE(TAG, "Failed to allocate memory for MQTT data");
                break;
            }
            memcpy(data_str, event->data, event->data_len);
            data_str[event->data_len] = '\0';
            
            json_payload = cJSON_Parse(data_str);
            free(data_str);

            if (json_payload == NULL) {
                ESP_LOGE(TAG, "Failed to parse JSON payload: %s", cJSON_GetErrorPtr());
                break;
            }

            if (strncmp(event->topic, shadow_delta_topic, event->topic_len) == 0) {
                ESP_LOGI(TAG, "Received DELTA update");
                cJSON *state_node = cJSON_GetObjectItem(json_payload, "state"); // Delta usually contains the state directly
                if (state_node) {
                    apply_desired_state(state_node);
                } else {
                    ESP_LOGW(TAG, "Delta message does not contain 'state' object");
                }
            } else if (strncmp(event->topic, shadow_get_accepted_topic, event->topic_len) == 0) {
                ESP_LOGI(TAG, "Received full shadow document from GET/ACCEPTED");
                cJSON *state_node = cJSON_GetObjectItem(json_payload, "state");
                if (state_node) {
                    cJSON *desired_node = cJSON_GetObjectItem(state_node, "desired");
                    if (desired_node) {
                        ESP_LOGI(TAG, "Processing 'desired' state from full shadow");
                        apply_desired_state(desired_node);
                    }
                    // Optionally, also sync reported if needed, though device is source of truth for reported
                    cJSON *reported_node = cJSON_GetObjectItem(state_node, "reported");
                    if(reported_node){
                        // This is where you might reconcile if cloud has an older reported state
                        // For simplicity, we assume device's g_ variables are the truth for reported.
                        // On initial connection, it's good to report current state regardless.
                        ESP_LOGI(TAG, "Reporting current state after GET/ACCEPTED processing");
                        report_current_state_to_shadow();
                    } else {
                         ESP_LOGI(TAG, "No 'reported' state in full shadow, reporting current state.");
                         report_current_state_to_shadow(); // Report if no reported state exists yet
                    }
                } else {
                     ESP_LOGW(TAG, "GET/ACCEPTED message does not contain 'state' object");
                }
            }
            cJSON_Delete(json_payload);
            break;
        case MQTT_EVENT_ERROR:
            ESP_LOGE(TAG, "MQTT_EVENT_ERROR type %d", event->error_handle->error_type);
            break;
        default:
            ESP_LOGI(TAG, "Other MQTT event id:%d", event->event_id);
            break;
    }
}

static void report_current_state_to_shadow(void) {
    if (!mqtt_client_handle || !(xEventGroupGetBits(wifi_event_group) & WIFI_CONNECTED_BIT)) {
        ESP_LOGW(TAG, "Cannot report state: MQTT not connected or Wi-Fi down.");
        return;
    }

    cJSON *root = cJSON_CreateObject();
    cJSON *state = cJSON_CreateObject();
    cJSON *reported = cJSON_CreateObject();

    cJSON_AddItemToObject(root, "state", state);
    cJSON_AddItemToObject(state, "reported", reported);

    cJSON_AddBoolToObject(reported, "led_on", g_led_state);
    cJSON_AddNumberToObject(reported, "brightness", g_brightness);
    cJSON_AddNumberToObject(reported, "telemetry_interval", g_telemetry_interval_sec);
    cJSON_AddStringToObject(reported, "firmware_version", "v0.1.0"); // Example static value

    char *json_string = cJSON_PrintUnformatted(root);
    if (json_string) {
        esp_mqtt_client_publish(mqtt_client_handle, shadow_update_topic, json_string, 0, 1, 0);
        ESP_LOGI(TAG, "Reported state to %s: %s", shadow_update_topic, json_string);
        free(json_string);
    } else {
        ESP_LOGE(TAG, "Failed to create JSON for reporting state.");
    }
    cJSON_Delete(root);
}


static void mqtt_app_start(void) {
    // Construct topic strings
    snprintf(shadow_update_topic, sizeof(shadow_update_topic), "devices/%s/shadow/update", DEVICE_ID);
    snprintf(shadow_delta_topic, sizeof(shadow_delta_topic), "devices/%s/shadow/update/delta", DEVICE_ID);
    snprintf(shadow_get_topic, sizeof(shadow_get_topic), "devices/%s/shadow/get", DEVICE_ID);
    snprintf(shadow_get_accepted_topic, sizeof(shadow_get_accepted_topic), "devices/%s/shadow/get/accepted", DEVICE_ID);

    const esp_mqtt_client_config_t mqtt_cfg = {
        .broker.address.uri = MQTT_BROKER_URI,
        .credentials.client_id = DEVICE_ID, // Or a unique client ID if DEVICE_ID is just for topics
        // Add .broker.verification.certificate for TLS if needed
    };

    mqtt_client_handle = esp_mqtt_client_init(&mqtt_cfg);
    esp_mqtt_client_register_event(mqtt_client_handle, ESP_EVENT_ANY_ID, mqtt_event_handler_cb, NULL);
    // MQTT client start is handled by Wi-Fi event handler once IP is obtained
}

void app_main(void) {
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    gpio_reset_pin(LED_GPIO);
    gpio_set_direction(LED_GPIO, GPIO_MODE_OUTPUT);
    gpio_set_level(LED_GPIO, g_led_state); // Set initial physical state

    wifi_init_sta();
    mqtt_app_start();

    // Periodically report state (can also be event-driven on state change)
    while (1) {
        vTaskDelay(pdMS_TO_TICKS(g_telemetry_interval_sec * 1000)); // Use dynamic interval
        ESP_LOGI(TAG, "Periodic state report due (current interval %ds).", g_telemetry_interval_sec);
        report_current_state_to_shadow();
    }
}


graph TD
    A["<center><b>Start: app_main()</b></center>"] --> B["<center>1. Init NVS, Wi-Fi, GPIO</center>"];
    B --> C["<center>2. <tt>wifi_init_sta()</tt><br>(Connect to Wi-Fi)</center>"];
    C --> D["<center>3. <tt>mqtt_app_start()</tt><br>(Init MQTT Client, Define Shadow Topics)</center>"];
    
    subgraph "MQTT Event Handler (mqtt_event_handler_cb)"
        EvConnected["<center><b>MQTT_EVENT_CONNECTED</b></center>"] --> SubDelta["<center>Subscribe to Shadow Delta Topic<br><tt>.../shadow/update/delta</tt></center>"];
        SubDelta --> SubGetAccepted["<center>Subscribe to Shadow Get Accepted Topic<br><tt>.../shadow/get/accepted</tt></center>"];
        SubGetAccepted --> PubGet["<center>Publish to Shadow Get Topic<br><tt>.../shadow/get</tt><br>(Request full shadow)</center>"];
        
        EvData["<center><b>MQTT_EVENT_DATA</b></center>"] --> CheckTopic{"<center>Topic Match?</center>"};
        CheckTopic -- Delta Topic --> ParseDelta["<center>Parse Delta JSON</center>"] --> ApplyDesired["<center>Call <tt>apply_desired_state()</tt></center>"];
        CheckTopic -- Get/Accepted Topic --> ParseFullShadow["<center>Parse Full Shadow JSON</center>"] --> ProcessDesiredFull["<center>Process 'desired' section<br>(Call <tt>apply_desired_state()</tt>)</center>"];
        ProcessDesiredFull --> ReportInitial["<center>Optionally, call <tt>report_current_state_to_shadow()</tt></center>"];
    end

    D -- Triggers on Connection --> EvConnected;
    D -- Receives Data --> EvData;

    subgraph "apply_desired_state(json_delta)"
        direction TB
        ApplyStart["<center><b>Start apply_desired_state</b></center>"] --> CompareLed["<center>Compare <tt>delta.led_on</tt> with <tt>g_led_state</tt></center>"];
        CompareLed -- Different --> UpdateLed["<center>Update <tt>g_led_state</tt><br>Set GPIO</center>"] --> SetChangedFlag["<center>state_changed = true</center>"];
        CompareLed -- Same --> CompareBrightness;
        UpdateLed --> CompareBrightness;
        CompareBrightness["<center>Compare <tt>delta.brightness</tt> with <tt>g_brightness</tt></center>"] -- Different --> UpdateBrightness["<center>Update <tt>g_brightness</tt><br>(Set PWM)</center>"] --> SetChangedFlag2["<center>state_changed = true</center>"];
        CompareBrightness -- Same --> CheckChanged;
        UpdateBrightness --> CheckChanged;
        SetChangedFlag --> CompareBrightness;
        SetChangedFlag2 --> CheckChanged;
        CheckChanged{"<center>state_changed?</center>"} -- Yes --> ReportBack["<center>Call <tt>report_current_state_to_shadow()</tt></center>"];
        CheckChanged -- No --> ApplyEnd["<center><b>End apply_desired_state</b></center>"];
        ReportBack --> ApplyEnd;
    end
    ApplyDesired -.-> ApplyStart;
    
    subgraph "ESP32 Main Loop (Periodic Reporting)"
        direction TB
        LoopStart["<center><b>Loop Start (while(1))</b></center>"] --> DelayTask["<center>Delay for <tt>g_telemetry_interval_sec</tt></center>"];
        DelayTask --> ReportPeriodic["<center>Call <tt>report_current_state_to_shadow()</tt></center>"];
        ReportPeriodic --> LoopStart;
    end
    D --> LoopStart;

    classDef startNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef eventNode fill:#A7F3D0,stroke:#047857,stroke-width:1px,color:#064E3B;
    classDef functionCall fill:#FBCFE8,stroke:#DB2777,stroke-width:1px,color:#9D174D;

    class A startNode;
    class B,C,D,SubDelta,SubGetAccepted,PubGet,ParseDelta,ParseFullShadow,ProcessDesiredFull,ReportInitial,UpdateLed,UpdateBrightness,DelayTask,ReportPeriodic processNode;
    class CheckTopic,CompareLed,CompareBrightness,CheckChanged decisionNode;
    class EvConnected,EvData eventNode;
    class ApplyDesired,ReportBack,ApplyStart,ApplyEnd,SetChangedFlag,SetChangedFlag2 functionCall;
    class LoopStart processNode;

main/Kconfig.projbuild (or add to existing):

Plaintext
menu "Shadow Example Configuration"
    config WIFI_SSID
        string "Wi-Fi SSID"
        default "YourSSID"

    config WIFI_PASSWORD
        string "Wi-Fi Password"
        default "YourPassword"
    
    config MQTT_BROKER_URI
        string "MQTT Broker URI"
        default "mqtt://192.168.1.100" # Replace with your broker's IP/hostname

    config DEVICE_ID
        string "Device ID for Shadow Topics"
        default "esp32-smart-light"

    config LED_GPIO
        int "LED GPIO Pin"
        default 2
endmenu

main/CMakeLists.txt:

Make sure cJSON is linked. It’s usually available as a component in ESP-IDF. If not explicitly linked by esp-mqtt or another component, you might need to add REQUIRES cJSON or PRIV_REQUIRES cJSON to your idf_component_register. However, esp-idf-cjson is typically a default component.

Build Instructions

  1. Save all files.
  2. Open ESP-IDF Terminal in VS Code.
  3. Configure: idf.py menuconfig. Set Wi-Fi, MQTT Broker URI, Device ID, and LED GPIO.
  4. Build: idf.py build

Run/Flash/Observe Steps

  1. Ensure your MQTT broker is running.
  2. Connect ESP32 device.
  3. Flash: idf.py -p /dev/ttyUSB0 flash monitor (adjust port).
  4. Observe ESP32 Logs:
    • Wi-Fi connection.
    • MQTT connection, subscription to delta topic.
    • Publication to shadow/get topic.
    • Initial state report to shadow/update.
  5. Simulate Cloud/Application Interaction (using MQTTX or similar tool):
    • Connect your MQTT tool to the same broker.
    • Subscribe to devices/esp32-smart-light/shadow/update to see reported states from the ESP32.
    • To change desired state: Publish a JSON payload to devices/esp32-smart-light/shadow/update/delta (or to a topic that your backend would translate into a delta/desired update for the device). For example, to turn LED off and set brightness:{ "state": { "led_on": false, "brightness": 30, "telemetry_interval": 15 }, "timestamp": 1678888000 }
      (Note: Real shadow services might expect you to publish to shadow/update with a desired section, and they generate the delta. For this generic example, publishing directly to a delta-like topic that the device subscribes to simplifies simulation).
    • Observe ESP32 logs: It should receive the delta, apply changes (log messages, LED state change), and then publish its new reported state.
    • Verify the new reported state in your MQTT tool.

Variant Notes

  • ESP32, ESP32-S2, ESP32-S3, ESP32-C3, ESP32-C6: All Wi-Fi enabled variants can implement Device Shadow interactions.
    • Resource Usage: JSON parsing (cJSON library) and generation, along with MQTT and TLS (if used), consume RAM and flash. For complex shadow documents or frequent updates on highly constrained devices (e.g., ESP32-C3 with minimal flash/RAM), optimize JSON handling and monitor resources.
    • The cJSON library is efficient but dynamic memory allocation for parsing/creating JSON objects needs careful management to avoid fragmentation or out-of-memory errors. Consider pre-allocating buffers or using cJSON’s buffered functions if memory is tight.
  • ESP32-H2: Lacks built-in Wi-Fi. To participate in a Device Shadow system, it would require a gateway architecture as described in previous chapters. The gateway device would manage the shadow interaction with the cloud, while the ESP32-H2 would communicate its state to/from the gateway using a local protocol (BLE, Thread, Zigbee). The gateway would then translate these into shadow updates.

Common Mistakes & Troubleshooting Tips

Mistake / Issue Symptom(s) Troubleshooting / Solution
JSON Parsing/Generation Errors Device fails to process delta/desired state, or cloud fails to parse reported state. cJSON_Parse() returns NULL, cJSON_PrintUnformatted() returns NULL. Errors in device or cloud logs related to JSON. Validate JSON structure against expected format (e.g., {"state":{"reported":{...}}}).
Check for typos, missing commas/braces, incorrect data types.
Use cJSON_GetErrorPtr() after parsing failures. Ensure proper memory management with cJSON (cJSON_Delete()).
Log raw JSON strings on ESP32 before publishing and upon reception.
Incorrect MQTT Topics for Shadow Device doesn’t receive delta updates, GET requests fail, or reported state isn’t reflected in the cloud. Verify exact topic strings: .../shadow/update, .../shadow/update/delta, .../shadow/get, .../shadow/get/accepted.
Ensure {thingName} or {deviceId} is correctly substituted.
Check MQTT client subscriptions and publications.
State Synchronization Loops (Thrashing) Device repeatedly applies a desired state, reports it, cloud sends same delta again. Constant MQTT traffic related to the same state change. Ensure device only reports its state after successfully applying the change to hardware/internal state.
If a desired state cannot be achieved, device should report its actual current state, not the failed desired state.
Implement robust logic to compare current state with desired state before applying changes to avoid unnecessary actions.
Check if cloud platform requires explicit clearing of desired state (e.g., publishing "desired": {"param": null}) once applied.
Ignoring Offline Desired State Changes Device reconnects after being offline but doesn’t pick up desired state changes made by an app during its offline period. On (re)connection to MQTT broker, device must explicitly fetch the current full shadow (e.g., publish to .../shadow/get) or ensure it’s subscribed to .../shadow/update/delta before assuming its local state is authoritative.
Process the received full shadow’s ‘desired’ section or any pending deltas.
Mishandling Shadow Versions (If platform uses versioning) Shadow updates are rejected by the cloud due to version mismatch. Device tries to update with an old version number. When updating the shadow, include the latest version number received from the cloud (e.g., from /get/accepted or /update/delta messages).
If an update is rejected due to version conflict, re-fetch the shadow, re-apply changes if necessary, and retry the update with the new version.
Reported State Not Clearing Delta Device applies a desired change and reports its new state, but the cloud continues to send a delta for that change. Ensure the keys and values in the ‘reported’ state exactly match what was in the ‘desired’ state that was applied. Case sensitivity matters.
Some platforms might require the ‘desired’ section to be explicitly nulled out by the device after applying, as mentioned above.
ESP32 Resource Issues (RAM/Stack) Crashes or instability when parsing large shadow documents or handling frequent updates. cJSON allocation failures. Optimize JSON document size if possible. Keep shadow state focused on essential parameters.
Monitor heap and stack usage. Increase task stack for MQTT/shadow handling if needed.
Consider cJSON_Minify or ensure unformatted JSON is used for transmission to save space.
No Initial State Reported Device connects, but applications see no ‘reported’ state until the device experiences its first change or periodic report. Upon successful connection and processing of any initial ‘desired’ state (from GET), the device should immediately report its current full state to populate the ‘reported’ section of the shadow.

Exercises

  1. Shadow Versioning (Conceptual):
    • Research how AWS IoT Device Shadow or Azure IoT Hub Device Twin use version numbers in their shadow documents.
    • Explain in a short paragraph how versioning helps prevent conflicts when multiple actors (device, applications) try to update the shadow simultaneously.
  2. Error Reporting via Shadow:
    • Modify the ESP32 example. If the device receives an invalid brightness value (e.g., > 100) in the desired state, instead of just ignoring it, report an error back via the reported state. For example: {"reported": {"last_error": "Invalid brightness value received"}}.
  3. Partial Shadow Updates:
    • Currently, report_current_state_to_shadow() sends all known reported parameters. Modify it so that if only g_led_state changes, it only includes led_on in the reported section of the shadow update. (Hint: Build the JSON dynamically based on what changed).
  4. Simulate Intermittent Connectivity:
    • In your ESP32 code, add a mechanism to temporarily stop and then restart the MQTT client (simulating network loss and recovery).
    • While the ESP32 is “offline” (MQTT client stopped), use your MQTT tool to publish a new desired state to the shadow (via the delta topic or by updating the full shadow on the broker if you have a way to simulate the backend).
    • Observe if the ESP32 correctly picks up and applies the desired state changes when it “reconnects” (MQTT client restarted and shadow/get is processed).

Summary

  • A Device Shadow is a cloud-persisted JSON document representing a device’s reported (actual) and desired (intended) states.
  • Shadows decouple devices from applications, handle intermittent connectivity, and provide a consistent way to interact with device state.
  • The delta state highlights differences between desired and reported, guiding the device on necessary changes.
  • Digital Twins are a broader concept, potentially including models and analytics, with shadows being a key state management component.
  • Devices typically use MQTT to publish reported states and subscribe to desired/delta state changes on specific topics.
  • On connection, devices should synchronize with the latest shadow state. After applying desired changes, they must report their new actual state.
  • Careful JSON handling, topic management, and state synchronization logic are crucial for robust shadow implementation on the ESP32.

Further Reading

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top