Chapter 120: AWS IoT Core Integration for ESP32

Chapter Objectives

By the end of this chapter, you will be able to:

• Understand the role and architecture of AWS IoT Core in IoT solutions.
• Set up and configure an “Thing” (device) in AWS IoT Core.
• Manage device certificates and security policies for secure communication.
• Connect an ESP32 device to the AWS IoT Core MQTT broker using TLS.
• Publish telemetry data from an ESP32 to AWS IoT Core.
• Subscribe to and receive commands from AWS IoT Core on an ESP32.
• Utilize the AWS IoT Device Shadow service to report and synchronize device state.
• Integrate the esp-aws-iot component into an ESP-IDF project for AWS connectivity.
• Recognize key considerations for deploying ESP32 devices with AWS IoT Core.

Introduction

As we venture further into building sophisticated IoT solutions, the need to connect our embedded devices to robust cloud platforms becomes increasingly apparent. Cloud services offer scalability, extensive data storage, powerful processing capabilities, advanced analytics, and seamless integration with other web services. Amazon Web Services (AWS) IoT Core is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices.

In previous chapters, we’ve explored various communication protocols, including MQTT. AWS IoT Core heavily relies on MQTT for device communication, making it a natural fit for our ESP32 projects. This chapter will guide you through the process of registering your ESP32 as a “Thing” in AWS IoT Core, establishing secure communication using X.509 certificates, and exchanging data using MQTT and the Device Shadow service. By leveraging AWS IoT Core, your ESP32 applications can become part of larger, more intelligent, and globally accessible IoT ecosystems.

Theory

AWS IoT Core Overview

AWS IoT Core is a central component in the AWS IoT suite of services. It acts as a gateway, allowing devices to connect to the AWS cloud and routing messages between devices and AWS cloud services. Its key functionalities include:

• Message Broker: A highly scalable publish/subscribe message broker that supports MQTT (standard and over WebSockets) and HTTPS. This is the primary way devices communicate.
• Device Registry (Thing Registry): Allows you to register and manage your devices (referred to as “Things”). Each Thing can have attributes, certificates, and be associated with Thing Types and Thing Groups.
• Security and Identity Service: Provides mutual authentication and encryption at all points of connection, so that data is never exchanged between devices and AWS IoT Core without proven identity. This is primarily achieved using X.509 certificates.
• Rules Engine: Enables you to build IoT applications that gather, process, analyze, and act on data generated by connected devices. You can configure rules to route messages to other AWS services like Lambda, S3, DynamoDB, Kinesis, SNS, etc., based on message content.
• Device Shadow Service: Maintains a persistent, virtual representation (a “shadow”) of each connected device. The shadow stores the last reported state and desired future state of the device, even when the device is offline. Applications can interact with the shadow to get the device’s current state or set a desired state.

graph TD
    subgraph "External World"
        direction LR
        D1["<center><b>ESP32 Devices</b><br>(Things)</center>"]
        D2["<center><b>Other Clients</b><br>(Web/Mobile Apps)</center>"]
    end

    subgraph "AWS IoT Core"
        direction TB
        MB["<center><b>Message Broker</b><br>(MQTT, HTTPS)</center>"]
        REG["<center><b>Device Registry</b><br>(Thing Registry)</center>"]
        SEC["<center><b>Security & Identity</b><br>(X.509 Certs, Policies)</center>"]
        RE["<center><b>Rules Engine</b></center>"]
        DS["<center><b>Device Shadow Service</b></center>"]

        MB --- REG
        MB --- SEC
        MB --- RE
        MB --- DS
    end

    subgraph "Other AWS Services"
        direction RL
        S3["<center><b>Amazon S3</b><br>(Storage)</center>"]
        LAM["<center><b>AWS Lambda</b><br>(Compute)</center>"]
        DDB["<center><b>Amazon DynamoDB</b><br>(NoSQL Database)</center>"]
        KIN["<center><b>Amazon Kinesis</b><br>(Data Streams)</center>"]
        SNS["<center><b>Amazon SNS</b><br>(Notifications)</center>"]
        OtherS["<center><b>... more services</b></center>"]
    end

    D1 -- "MQTT/TLS (X.509)" --> MB
    D2 -- "HTTPS/MQTT over WebSockets" --> MB

    RE --> S3
    RE --> LAM
    RE --> DDB
    RE --> KIN
    RE --> SNS
    RE --> OtherS

    classDef device fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6
    classDef iotCoreComponent fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF
    classDef awsService fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46
    classDef centralBroker fill:#FEF3C7,stroke:#D97706,stroke-width:2px,color:#92400E

    class D1,D2 device;
    class MB centralBroker;
    class REG,SEC,RE,DS iotCoreComponent;
    class S3,LAM,DDB,KIN,SNS,OtherS awsService;

Key Concepts

• Thing: A representation of your physical device (e.g., an ESP32) in AWS IoT Core. Each Thing has a unique name (Thing Name) and can have attributes.
• X.509 Certificates: Used for authenticating devices to AWS IoT Core. Each device needs a unique certificate and private key. AWS IoT Core also has its own root CA certificates that the device must trust. Communication is secured using TLS.
• Policies: JSON documents that define the permissions for a device (identified by its certificate). Policies control what actions a device can perform, such as connecting to the broker, publishing to specific MQTT topics, or subscribing to topics. They are attached to certificates.
• MQTT (Message Queuing Telemetry Transport): The primary communication protocol used by devices to interact with AWS IoT Core. It’s a lightweight publish/subscribe protocol ideal for constrained devices.
  • Topics: Devices publish messages to topics (e.g., device/esp32-001/temperature) and subscribe to topics to receive messages. AWS IoT Core uses a hierarchical topic structure.
  • QoS (Quality of Service): MQTT supports QoS levels 0 (at most once) and 1 (at least once) with AWS IoT Core. QoS 2 (exactly once) is not supported for device-to-broker communication.

• Device Shadow:
  • A JSON document that stores and retrieves current state information for a device.It has a reported section (what the device last reported its state to be) and a desired section (what the application or user wants the device’s state to be).When the desired state differs from the reported state, a delta is generated. Devices can subscribe to this delta to know they need to change their state.Shadows are accessed via specific MQTT topics (e.g., $aws/things/YourThingName/shadow/update, $aws/things/YourThingName/shadow/get, $aws/things/YourThingName/shadow/update/delta).

graph TB
    subgraph "Device (ESP32)"
        direction TB
        DevApp["<center><b>Device Application</b></center>"]
        DevState["<center><b>Actual Device State</b><br>(e.g., LED: ON, temp: 25C)</center>"]
    end

    subgraph "AWS IoT Core"
        direction TB
        Shadow["<center><b>Device Shadow Document</b><br>(JSON)</center>"]
        ShadowReported["<center><b>Reported State</b><br><tt>{ \led\: \ON\, \temp\: 25 }</tt></center>"]
        ShadowDesired["<center><b>Desired State</b><br><tt>{ \led\: \OFF\, \interval\: 60 }</tt></center>"]
        ShadowDelta["<center><b>Delta</b><br><tt>{ \led\: \OFF\, \interval\: 60 }</tt></center>"]
        Shadow -- contains --> ShadowReported
        Shadow -- contains --> ShadowDesired
        Shadow -- generates --> ShadowDelta
    end

    subgraph "Cloud/User Application"
        direction TB
        CloudApp["<center><b>Cloud Application / User</b></center>"]
    end

    DevApp -- "1- Publishes current state" --> ShadowReported;
    CloudApp -- "2- Updates desired state" --> ShadowDesired;
    ShadowDelta -- "3- Notifies device of changes" --> DevApp;
    DevApp -- "4- Acts on delta" --> DevState;
    DevApp -- "5- Publishes new reported state" --> ShadowReported;
    CloudApp -- "Can read current/desired state" --> Shadow;

    classDef device fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef shadow fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef cloud fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46;
    classDef stateData fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;

    class DevApp,DevState device;
    class Shadow shadow;
    class ShadowReported,ShadowDesired,ShadowDelta stateData;
    class CloudApp cloud;

• AWS IoT Endpoint: Each AWS account has a unique, regional endpoint (a URL) that devices connect to for AWS IoT Core services.

ESP-IDF esp-aws-iot Component

ESP-IDF provides an esp-aws-iot component that simplifies integration with AWS IoT Core. This component is based on the AWS IoT Device SDK for Embedded C. It provides APIs for:

• Establishing a secure MQTT connection using TLS and X.509 certificates.
• Publishing and subscribing to MQTT messages.
• Interacting with the Device Shadow service.

You’ll typically configure this component with your device certificates, private key, root CA certificate, and AWS IoT endpoint.

Practical Examples

The following examples will guide you through connecting an ESP32 to AWS IoT Core.

Prerequisites

1. AWS Account: You need an active AWS account.
2. AWS CLI (Optional but Recommended): The AWS Command Line Interface can be helpful for managing IoT resources, though most setup can be done via the AWS Management Console.
3. ESP-IDF v5.x Environment: Properly set up with VS Code.
4. OpenSSL (or similar tool): Sometimes needed for certificate format verification or conversion, though AWS provides certificates in the correct format.

Step 1: Setting up AWS IoT Core

• Navigate to AWS IoT Core Console:
  • Log in to your AWS Management Console.
  • Select your desired region (e.g., us-east-1, eu-west-2). This is important as your IoT endpoint is region-specific.
  • Search for “IoT Core” and navigate to the service.
• Create a Policy:
  • In the IoT Core console, go to “Secure” -> “Policies”.
  • Click “Create policy”.
  • Policy name: e.g., ESP32_Device_Policy.
  • Policy document (JSON): Add statements to allow necessary actions. For a basic device that connects, publishes, subscribes, and receives:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:client/${iot:ClientId}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": [
        "arn:aws:iot:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:topic/*/${iot:ClientId}/*",
        "arn:aws:iot:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:topic/$aws/things/${iot:ClientId}/shadow/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:topicfilter/*/${iot:ClientId}/*",
        "arn:aws:iot:YOUR_AWS_REGION:YOUR_AWS_ACCOUNT_ID:topicfilter/$aws/things/${iot:ClientId}/shadow/*"
      ]
    }
  ]
}


Important: Replace YOUR_AWS_REGION and YOUR_AWS_ACCOUNT_ID with your actual values. The ${iot:ClientId} variable allows the policy to be generic for any device whose client ID matches its Thing Name. This policy is quite permissive for development; for production, restrict topics more granularly.

• Click “Create”.
• Create a Thing and Certificates:
  • Go to “Manage” -> “Things”.
  • Click “Create things”.
  • Select “Create single thing”. Click “Next”.
  • Thing name: e.g., MyESP32_Device_01. This will also be used as the MQTT Client ID. Click “Next”.
  • Device certificate: Choose “Auto-generate a new certificate (recommended)”. Click “Next”.
  • Policies: Select the policy you created (e.g., ESP32_Device_Policy). Click “Create thing”.
  • Download Certificates and Keys: This is a critical step. You must download these files now, as the private key won’t be available later:
    • Device certificate: (e.g., xxxxxxxxxx-certificate.pem.crt)
    • Private key: (e.g., xxxxxxxxxx-private.pem.key)
    • Root CA certificate: Download the “Amazon Root CA 1” (or the appropriate current root CA for your region/endpoint – check AWS docs for the latest recommendations, often it’s the Amazon Root CA 1, 2, 3 or Starfield). You can find these under “Secure” -> “Certificates” -> “CA certificates” or linked from the “Connect a device” wizard. A common one is “AmazonRootCA1.pem”.
  • Click “Done”.
• Get Your AWS IoT Endpoint:
  • In the IoT Core console, go to “Settings” (usually in the bottom-left navigation pane).
  • Your Endpoint will be listed here (e.g., xxxxxxxxxxxxxx-ats.iot.YOUR_AWS_REGION.amazonaws.com). Note this down.

Step 2: ESP32 Project Setup

• Create a new ESP-IDF Project.
• Embed Certificates and Key:
  • Create a directory in your main component, e.g., main/certs.
  • Copy the downloaded device certificate, private key, and Amazon Root CA 1 certificate into this main/certs directory. Rename them for convenience, e.g.:
    • device.pem.crt
    • private.pem.key
    • aws_root_ca.pem
  • In your main/CMakeLists.txt, add the following to embed these files into the firmware:

idf_component_register(...
                       REQUIRES esp-aws-iot ...) # Ensure esp-aws-iot is required

target_add_binary_data(${COMPONENT_TARGET} "certs/device.pem.crt" TEXT 이름 aws_iot_certificate_pem_start)
target_add_binary_data(${COMPONENT_TARGET} "certs/private.pem.key" TEXT 이름 aws_iot_private_key_pem_start)
target_add_binary_data(${COMPONENT_TARGET} "certs/aws_root_ca.pem" TEXT 이름 aws_iot_root_ca_pem_start)

• 
This makes the content of these files available as extern const uint8_t aws_iot_certificate_pem_start[] etc. in your C code.
• Configure esp-aws-iot via menuconfig:
  • Run idf.py menuconfig.
  • Navigate to Component config -> Amazon Web Services IoT Platform.
  • AWS IoT Endpoint Hostname: Enter your AWS IoT endpoint.
  • AWS IoT MQTT Port: Usually 8883 (for TLS) or 443 (MQTT over WSS). 8883 is common for device SDKs.
  • Client ID: Set this to your Thing Name (e.g., MyESP32_Device_01).
  • You might also need to adjust MQTT Tx/Rx buffer sizes or keep-alive settings depending on your application.
  Tip: Ensure your ESP32 has its system time synchronized (e.g., using SNTP, covered in Chapter 98), as TLS connections require accurate time for certificate validation.

Example 1: Publishing Telemetry to AWS IoT Core

This example simulates publishing temperature data.

graph TD
    A["<center><b>Start: ESP32 app_main</b></center>"] --> B["<center>1. Initialize NVS, Netif, Event Loop</center>"];
    B --> C["<center>2. Connect to Wi-Fi<br><tt>example_connect()</tt></center>"];
    C --> D["<center>3. Create <tt>aws_publisher_task</tt></center>"];
    
    subgraph "aws_publisher_task"
        direction TB
        D1["<center><b>4. Initialize MQTT Client Params</b><br><tt>IoT_Client_Init_Params</tt><br>(Endpoint, Port, Certs)</center>"] --> D2["<center><b>5. <tt>aws_iot_mqtt_init()</tt></b></center>"];
        D2 -- Success --> D3["<center><b>6. Set MQTT Connect Params</b><br><tt>IoT_Client_Connect_Params</tt><br>(ClientID, KeepAlive)</center>"];
        D3 --> D4{"<center><b>7. <tt>aws_iot_mqtt_connect()</tt></b></center>"};
        D4 -- Success --> D5["<center><b>8. <tt>aws_iot_mqtt_autoreconnect_set_status(true)</tt></b></center>"];
        D5 --> D6["<center><b>Loop:</b></center>"];
        D6 --> D7{"<center><b>9. <tt>aws_iot_mqtt_yield()</tt></b><br><i>(Handle reconnects)</i></center>"};
        D7 -- Connected --> D8["<center><b>10. Simulate/Read Sensor Data</b><br>(e.g., temperature)</center>"];
        D8 --> D9["<center><b>11. Format Payload (JSON)</b><br><tt>{\temperature\: 25.5}</tt></center>"];
        D9 --> D10["<center><b>12. Set Publish Params</b><br><tt>IoT_Publish_Message_Params</tt><br>(QoS1, Payload)</center>"];
        D10 --> D11{"<center><b>13. <tt>aws_iot_mqtt_publish()</tt></b><br>To topic: <tt>device/CLIENT_ID/data</tt></center>"};
        D11 -- Success/Timeout --> D12["<center><b>14. Log Publish Status</b></center>"];
        D12 --> D13["<center><b>15. Delay (e.g., 10s)</b></center>"];
        D13 --> D6;
        D4 -- Failure --> D_Fail["<center><b>Handle Connection Error</b><br>Log, Disconnect, Exit Task</center>"];
        D2 -- Failure --> D_Fail;
        D11 -- Error --> D_FailPublish["<center><b>Log Publish Error</b><br><i>(Loop may continue or exit based on error)</i></center>"] --> D6;
    end
    
    A --> F["<center><b>AWS IoT Core</b><br>(MQTT Test Client Subscribed<br>to <tt>device/CLIENT_ID/data</tt>)</center>"];
    D11 -.->|MQTT Message| F;

    classDef startNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef cloudNode fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46;
    classDef errorNode fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef loopNode fill:#E0E7FF,stroke:#4338CA,stroke-width:1px,color:#3730A3;


    class A,D1 startNode;
    class B,C,D,D2,D3,D5,D8,D9,D10,D12,D13 processNode;
    class D4,D7,D11 decisionNode;
    class F cloudNode;
    class D_Fail errorNode;
    class D_FailPublish errorNode;
    class D6 loopNode;

main/aws_iot_publisher_main.c:

#include <stdio.h>
#include <string.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <freertos/event_groups.h>
#include "esp_log.h"
#include "nvs_flash.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_netif.h"
#include "esp_system.h" // For esp_random
#include "protocol_examples_common.h" // For Wi-Fi connection helper (if used)

#include "aws_iot_config.h" // From menuconfig
#include "aws_iot_log.h"
#include "aws_iot_version.h"
#include "aws_iot_mqtt_client_interface.h"
#include "aws_iot_shadow_interface.h" // Though not used in this specific example

static const char *TAG = "AWS_PUB";

// Embedded certificate data (declared by CMake target_add_binary_data)
extern const uint8_t aws_iot_certificate_pem_start[] asm("_binary_certs_device_pem_crt_start");
extern const uint8_t aws_iot_certificate_pem_end[]   asm("_binary_certs_device_pem_crt_end");
extern const uint8_t aws_iot_private_key_pem_start[] asm("_binary_certs_private_pem_key_start");
extern const uint8_t aws_iot_private_key_pem_end[]   asm("_binary_certs_private_pem_key_end");
extern const uint8_t aws_iot_root_ca_pem_start[]     asm("_binary_certs_aws_root_ca_pem_start");
extern const uint8_t aws_iot_root_ca_pem_end[]       asm("_binary_certs_aws_root_ca_pem_end");

AWS_IoT_Client mqtt_client;

static void aws_iot_mqtt_event_handler(AWS_IoT_Client *pClient, void *data,
                                     AWS_IoT_MQTT_Event_type_t event_type,
                                     AWS_IoT_MQTT_Event_data_t *pData) {
    switch (event_type) {
        case AWS_MQTT_EVENT_CONNECT:
            ESP_LOGI(TAG, "MQTT Connected");
            break;
        case AWS_MQTT_EVENT_DISCONNECT:
            ESP_LOGW(TAG, "MQTT Disconnected");
            break;
        case AWS_MQTT_EVENT_SUBSCRIBE_TIMEOUT:
            ESP_LOGW(TAG, "MQTT Subscribe Timeout");
            break;
        case AWS_MQTT_EVENT_PUBLISH_TIMEOUT:
            ESP_LOGW(TAG, "MQTT Publish Timeout");
            break;
        default:
            ESP_LOGD(TAG, "MQTT event %d", event_type);
    }
}

static void aws_publisher_task(void *param) {
    IoT_Error_t rc = FAILURE;

    // Initialize MQTT client parameters
    IoT_Client_Init_Params mqttInitParams = iotClientInitParamsDefault;
    mqttInitParams.enableAutoReconnect = false; // Recommended to be true
    mqttInitParams.pHostURL = CONFIG_AWS_IOT_MQTT_HOST; // From menuconfig
    mqttInitParams.port = CONFIG_AWS_IOT_MQTT_PORT;   // From menuconfig

    mqttInitParams.pRootCALocation = (const char *)aws_iot_root_ca_pem_start;
    mqttInitParams.pDeviceCertLocation = (const char *)aws_iot_certificate_pem_start;
    mqttInitParams.pDevicePrivateKeyLocation = (const char *)aws_iot_private_key_pem_start;

    mqttInitParams.mqttCommandTimeout_ms = 20000;
    mqttInitParams.tlsHandshakeTimeout_ms = 5000;
    mqttInitParams.isSSLHostnameVerify = true;
    mqttInitParams.disconnectHandler = NULL; // Can set a specific disconnect handler
    mqttInitParams.disconnectHandlerData = NULL;

    rc = aws_iot_mqtt_init(&mqtt_client, &mqttInitParams);
    if (SUCCESS != rc) {
        ESP_LOGE(TAG, "aws_iot_mqtt_init returned error : %d ", rc);
        goto exit_task;
    }

    // Connect parameters
    IoT_Client_Connect_Params connectParams = iotClientConnectParamsDefault;
    connectParams.keepAliveIntervalInSec = 600; // Increased keep-alive
    connectParams.isCleanSession = true;
    connectParams.MQTTVersion = MQTT_3_1_1;
    connectParams.pClientID = CONFIG_AWS_IOT_CLIENT_ID; // From menuconfig
    connectParams.clientIDLen = (uint16_t)strlen(CONFIG_AWS_IOT_CLIENT_ID);
    connectParams.isWillMsgPresent = false;

    ESP_LOGI(TAG, "Connecting to AWS IoT...");
    rc = aws_iot_mqtt_connect(&mqtt_client, &connectParams);
    if (SUCCESS != rc) {
        ESP_LOGE(TAG, "Error(%d) connecting to %s:%d", rc, mqttInitParams.pHostURL, mqttInitParams.port);
        goto exit_task;
    } else {
         ESP_LOGI(TAG, "Connected to AWS IoT successfully!");
    }
    /*
     * Enable Auto Reconnect functionality. Minimum and Maximum time of Exponential backoff are set in aws_iot_config.h
     * #AWS_IOT_MQTT_MIN_RECONNECT_WAIT_INTERVAL
     * #AWS_IOT_MQTT_MAX_RECONNECT_WAIT_INTERVAL
     */
    rc = aws_iot_mqtt_autoreconnect_set_status(&mqtt_client, true);
    if (SUCCESS != rc) {
        ESP_LOGE(TAG, "Unable to set Auto Reconnect to true - %d", rc);
        goto exit_task;
    }

    char topic_str[100];
    snprintf(topic_str, sizeof(topic_str), "device/%s/data", CONFIG_AWS_IOT_CLIENT_ID);
    ESP_LOGI(TAG, "Publishing to topic: %s", topic_str);

    IoT_Publish_Message_Params paramsQOS1;
    paramsQOS1.qos = QOS1;
    paramsQOS1.isRetained = 0;

    char cPayload[100];

    while ((NETWORK_ATTEMPTING_RECONNECT == rc || NETWORK_RECONNECTED == rc || SUCCESS == rc)) {
        //Max time the yield function will wait for read messages
        rc = aws_iot_mqtt_yield(&mqtt_client, 100);
        if (NETWORK_ATTEMPTING_RECONNECT == rc) {
            // If the client is attempting to reconnect we skip the rest of the loop.
            ESP_LOGI(TAG, "Attempting to reconnect...");
            vTaskDelay(pdMS_TO_TICKS(1000)); // Wait before next yield
            continue;
        }
        if (NETWORK_RECONNECTED == rc) {
             ESP_LOGI(TAG, "Network reconnected.");
        }


        float temperature = 20.0 + ((float)esp_random() / (float)UINT32_MAX) * 10.0; // Simulate 20-30 C
        int len = snprintf(cPayload, sizeof(cPayload), "{\"temperature\": %.2f, \"clientId\": \"%s\"}",
                           temperature, CONFIG_AWS_IOT_CLIENT_ID);
        paramsQOS1.payload = (void *)cPayload;
        paramsQOS1.payloadLen = len;

        rc = aws_iot_mqtt_publish(&mqtt_client, topic_str, (uint16_t)strlen(topic_str), &paramsQOS1);
        if (rc == MQTT_REQUEST_TIMEOUT_ERROR) {
            ESP_LOGW(TAG, "QOS1 publish ack not received.");
            rc = SUCCESS; // Reset to continue
        } else if (SUCCESS != rc) {
            ESP_LOGE(TAG, "Error publishing: %d", rc);
        } else {
            ESP_LOGI(TAG, "Published: %s", cPayload);
        }

        vTaskDelay(pdMS_TO_TICKS(10000)); // Publish every 10 seconds
    }

exit_task:
    ESP_LOGE(TAG, "Connection error or task exiting: %d!", rc);
    aws_iot_mqtt_disconnect(&mqtt_client);
    aws_iot_mqtt_free(&mqtt_client);
    vTaskDelete(NULL);
}

void app_main(void) {
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        ESP_ERROR_CHECK(nvs_flash_erase());
        ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());

    /* This helper function configures Wi-Fi or Ethernet, as selected in menuconfig.
     * Read "Establishing Wi-Fi or Ethernet Connection" section in
     * examples/protocols/README.md for more information about this function.
     */
    ESP_ERROR_CHECK(example_connect()); // From protocol_examples_common

    xTaskCreate(&aws_publisher_task, "aws_publisher_task", 9216, NULL, 5, NULL);
}

Note: You’ll need to add protocol_examples_common.c and its header to your project for example_connect(), or implement your own Wi-Fi connection logic (as shown in earlier chapters). Ensure Wi-Fi SSID/Password are set in menuconfig -> Example Connection Configuration.

Build, Flash, and Observe:

1. idf.py menuconfig (configure Wi-Fi, AWS IoT Endpoint, Client ID).
2. idf.py build.
3. idf.py -p /dev/ttyUSB0 flash monitor.
4. In AWS IoT Core Console:
   • Go to “Test” -> “MQTT test client”.
   • In “Subscribe to a topic”, enter device/MyESP32_Device_01/data (or device/+/data to catch all).
   • Click “Subscribe”.
   • You should see JSON messages with temperature data appearing every 10 seconds.

Example 2: Subscribing to Commands

Modify the previous example to also subscribe to a command topic.

Changes to aws_publisher_task (add before the while loop):

    // ... (after successful connect and autoreconnect_set_status) ...

    char sub_topic_str[100];
    snprintf(sub_topic_str, sizeof(sub_topic_str), "device/%s/command", CONFIG_AWS_IOT_CLIENT_ID);
    ESP_LOGI(TAG, "Subscribing to topic: %s", sub_topic_str);

    // MQTT message callback for subscribed topics
    void iot_subscribe_callback_handler(AWS_IoT_Client *pClient, char *topicName, uint16_t topicNameLen,
                                        IoT_Publish_Message_Params *params, void *pData) {
        ESP_LOGI(TAG, "Received message on topic '%.*s': '%.*s'", topicNameLen, topicName, (int)params->payloadLen, (char *)params->payload);
        // Example: Toggle an LED or perform an action based on payload
        if (strncmp((char*)params->payload, "LED_ON", params->payloadLen) == 0) {
            ESP_LOGI(TAG, "Command: Turn LED ON");
            // Add GPIO logic here
        } else if (strncmp((char*)params->payload, "LED_OFF", params->payloadLen) == 0) {
            ESP_LOGI(TAG, "Command: Turn LED OFF");
            // Add GPIO logic here
        }
    }

    rc = aws_iot_mqtt_subscribe(&mqtt_client, sub_topic_str, (uint16_t)strlen(sub_topic_str), QOS1, iot_subscribe_callback_handler, NULL);
    if (SUCCESS != rc) {
        ESP_LOGE(TAG, "Error subscribing : %d ", rc);
        // Handle error, maybe disconnect and exit
    } else {
        ESP_LOGI(TAG, "Subscribed to %s successfully", sub_topic_str);
    }

    // ... (rest of the while loop for publishing) ...

graph TD
    A["<center><b>ESP32: After MQTT Connect & Autoreconnect Setup</b></center>"] --> B["<center><b>1. Define Subscription Topic</b><br><tt>device/CLIENT_ID/command</tt></center>"];
    B --> C["<center><b>2. Define Callback Function</b><br><tt>iot_subscribe_callback_handler()</tt></center>"];
    C --> D{"<center><b>3. <tt>aws_iot_mqtt_subscribe()</tt></b><br>To command topic with callback</center>"};
    D -- Success --> E["<center><b>Subscription Active</b></center>"];
    D -- Failure --> F["<center><b>Handle Subscription Error</b></center>"];

    subgraph "ESP32: Main Loop (aws_iot_mqtt_yield)"
        direction LR
        Y["<center><b><tt>aws_iot_mqtt_yield()</tt></b></center>"]
    end
    
    E --> Y;

    subgraph "AWS IoT Core / MQTT Test Client"
        direction TB
        P["<center><b>Publisher (e.g., MQTT Test Client)</b></center>"] --> P1["<center><b>Publish Command</b><br>To: <tt>device/CLIENT_ID/command</tt><br>Payload: <tt>\LED_ON\</tt></center>"];
    end
    
    P1 -.->|MQTT Message| Y;
    
    Y -- Message for subscribed topic --> CB["<center><b><tt>iot_subscribe_callback_handler()</tt> Triggered</b></center>"];
    CB --> CB1["<center><b>Parse Payload</b><br>(e.g., <tt>\LED_ON\</tt>)</center>"];
    CB1 --> CB2["<center><b>Perform Action</b><br>(e.g., Control LED)</center>"];
    CB2 --> CB3["<center><b>Log Received Command</b></center>"];
    
    classDef startNode fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef processNode fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decisionNode fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef cloudNode fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46;
    classDef errorNode fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef callbackNode fill:#A7F3D0,stroke:#047857,stroke-width:1px,color:#064E3B;


    class A startNode;
    class B,C,E,Y,CB1,CB2,CB3 processNode;
    class D decisionNode;
    class P,P1 cloudNode;
    class F errorNode;
    class CB callbackNode;

Build, Flash, and Observe:

1. Reflash the ESP32.
2. In AWS IoT Core MQTT test client:
   • Go to “Publish to a topic”.
   • Topic name: device/MyESP32_Device_01/command.
   • Message payload (JSON or plain text): LED_ON or LED_OFF.
   • Click “Publish”.
   • Observe the ESP32’s serial monitor; it should log the received command.

Example 3: Using Device Shadow

The Device Shadow allows persistent state management.

main/aws_iot_shadow_main.c (Illustrative Snippets – requires more setup):

The esp-aws-iot SDK provides functions to interact with the shadow. This involves:

1. Initializing the shadow client: aws_iot_shadow_init().
2. Setting up shadow parameters: ShadowInitParameters_t.
3. Connecting the shadow client: aws_iot_shadow_connect().
4. Registering delta callbacks: aws_iot_shadow_register_delta().
5. Updating reported state: aws_iot_shadow_update().
6. Getting current shadow: aws_iot_shadow_get().
7. Yielding for shadow messages: aws_iot_shadow_yield().

// --- Illustrative Shadow Snippets ---
// (Full example requires careful integration with MQTT client and task structure)

// In your main task, after MQTT connection:
// ShadowInitParameters_t sp = ShadowInitParametersDefault;
// sp.pHost = CONFIG_AWS_IOT_MQTT_HOST;
// sp.port = CONFIG_AWS_IOT_MQTT_PORT;
// sp.pClientCRT = (const char *)aws_iot_certificate_pem_start;
// sp.pClientKey = (const char *)aws_iot_private_key_pem_start;
// sp.pRootCA = (const char *)aws_iot_root_ca_pem_start;
// sp.enableAutoReconnect = false; // Or true
// sp.disconnectHandler = NULL;

// rc = aws_iot_shadow_init(&mqtt_client, &sp); // Note: mqtt_client is also used by shadow
// if (SUCCESS != rc) { ESP_LOGE(TAG, "Shadow init error %d", rc); }

// ShadowConnectParameters_t scp = ShadowConnectParametersDefault;
// scp.pMyThingName = CONFIG_AWS_IOT_CLIENT_ID;
// scp.pMqttClientId = CONFIG_AWS_IOT_CLIENT_ID;
// scp.mqttClientIdLen = (uint16_t)strlen(CONFIG_AWS_IOT_CLIENT_ID);

// rc = aws_iot_shadow_connect(&mqtt_client, &scp);
// if (SUCCESS != rc) { ESP_LOGE(TAG, "Shadow connect error %d", rc); }

// // Callback for shadow delta
// void shadow_delta_callback(const char *pJsonString, uint32_t JsonStringDataLen, jsonStruct_t *pContext) {
//     ESP_LOGI(TAG, "Received shadow delta: %.*s", JsonStringDataLen, pJsonString);
//     // Parse JSON (pJsonString) to get desired state and act on it
//     // Example: if "desired":{"led_state":"ON"} is in delta, turn LED on.
//     // Then, report the new actual state.
// }

// jsonStruct_t delta_cb_data; // User data for callback
// rc = aws_iot_shadow_register_delta(&mqtt_client, &delta_cb_data, shadow_delta_callback, 10);
// if (SUCCESS != rc) { ESP_LOGE(TAG, "Shadow register delta error %d", rc); }


// // To report state:
// char reported_payload[100];
// snprintf(reported_payload, sizeof(reported_payload), "{\"state\":{\"reported\":{\"led_state\":\"ON\", \"uptime\":%lu}}}", (unsigned long)(esp_timer_get_time()/1000000));
// IoT_Publish_Message_Params_t params_update;
// params_update.qos = QOS1;
// params_update.payload = reported_payload;
// params_update.payloadLen = strlen(reported_payload);
// params_update.isRetained = 0;
// rc = aws_iot_shadow_update(&mqtt_client, CONFIG_AWS_IOT_CLIENT_ID, reported_payload, strlen(reported_payload), NULL, NULL, 10);
// if (SUCCESS != rc) { ESP_LOGE(TAG, "Shadow update error %d", rc); }


// // In the main loop:
// // aws_iot_shadow_yield(&mqtt_client, 200); // Process shadow messages

Note: A full shadow example is more complex and involves careful JSON parsing and state management. The esp-idf/examples/protocols/aws_iot/shadow example is a good reference.

To test shadow:

1. After ESP32 reports its state, go to your Thing in AWS IoT Console -> “Device Shadow”.
2. Edit the shadow document, add/modify a desired state (e.g., {"desired": {"led_state": "OFF"}}).
3. The ESP32 (if subscribed to delta) should receive this and act. Then it should update its reported state.

Variant Notes

• All Wi-Fi Enabled ESP32 Variants (ESP32, ESP32-S2, ESP32-S3, ESP32-C3, ESP32-C6):
  • These variants fully support AWS IoT Core integration using the esp-aws-iot component over Wi-Fi.
  • Resource Usage:
    • Flash: Storing certificates, the root CA, and the AWS IoT SDK adds to firmware size (can be tens to over a hundred KB).
    • RAM: TLS connections require significant RAM for buffers (typically 30-50KB or more during handshake, less once established). The MQTT client and shadow client also consume RAM. This needs to be budgeted, especially on variants with less RAM like ESP32-C3.
    • Performance: TLS handshake and encryption/decryption add CPU load. Hardware acceleration for cryptographic operations on newer ESP32 variants (e.g., ESP32-S2, S3, C3, C6) helps significantly.
• ESP32-H2 (Thread/Zigbee & BLE):
  • Direct AWS IoT Core connection via MQTT/TLS requires an IP network. If the ESP32-H2 is part of a Thread network, it would need an IP-capable border router to bridge Thread to Wi-Fi/Ethernet to reach the internet and AWS IoT Core.
  • Alternatively, an ESP32-H2 device might communicate with a gateway device (e.g., an ESP32 Wi-Fi variant) using BLE or another local protocol, and the gateway then relays messages to AWS IoT Core.
  • The esp-aws-iot SDK itself is IP-based.

Common Mistakes & Troubleshooting Tips

Exercises

1. Real Sensor Telemetry:
   • Modify Example 1. Instead of simulated temperature, read data from a physical sensor connected to your ESP32 (e.g., DHT11 for temperature/humidity, or an analog sensor via ADC).
   • Publish this real sensor data to AWS IoT Core.
2. Bi-directional Control with Acknowledgment:
   • Extend Example 2. When the ESP32 receives a command (e.g., “SET_THRESHOLD:30”), it should:
     • Perform an action (e.g., update an internal variable).
     • Publish an acknowledgment message back to a different topic (e.g., device/MyESP32_Device_01/command_ack) indicating success or failure and the new state.
   • Monitor this acknowledgment topic in the AWS IoT MQTT test client.
3. Advanced Device Shadow Interaction:
   • Create a more complex desired state in the Device Shadow (e.g., { "desired": { "config": { "update_interval_sec": 60, "logging_level": "INFO" }, "led_color": "blue" } }).
   • Implement logic on the ESP32 to parse the delta for these nested properties.
   • When the ESP32 applies these changes, it should report back the full new configuration and LED color in its reported state.
4. AWS IoT Rule to Lambda and SNS:
   • On AWS: Create a simple AWS Lambda function (e.g., in Python or Node.js) that can be triggered by an AWS IoT Rule.
   • On AWS: Create an SNS (Simple Notification Service) topic and subscribe your email address to it.
   • On AWS: Create an AWS IoT Rule that:
     • Listens to the MQTT topic your ESP32 publishes telemetry to (e.g., device/+/data).
     • Filters for a specific condition (e.g., temperature > 28.0).
     • If the condition is met, the rule action should trigger your Lambda function.
   • Modify the Lambda function to publish a message to your SNS topic (which then sends you an email).
   • Test by having your ESP32 publish a high temperature value. You should receive an email notification. (This exercise focuses more on the AWS cloud side but demonstrates end-to-end integration.)

Summary

• AWS IoT Core provides a scalable and secure platform for connecting ESP32 devices to the cloud.
• Key components include the Message Broker (MQTT), Device Registry (Things), Security Service (Certificates, Policies), Rules Engine, and Device Shadow.
• ESP32 devices authenticate using X.509 certificates and are authorized by IoT Policies.
• The esp-aws-iot component in ESP-IDF simplifies MQTT communication and Device Shadow interaction.
• Certificates and keys must be securely stored on/accessible by the ESP32, often embedded in firmware.
• Accurate system time (via SNTP) is crucial for TLS connections.
• Device Shadow allows for persistent state management (desired vs. reported) even when devices are offline.
• Careful configuration of AWS resources (Things, Policies, Endpoints) and ESP32 project settings is essential for successful integration.

Further Reading

• AWS IoT Core Developer Guide: https://docs.aws.amazon.com/iot/latest/developerguide/what-is-aws-iot.html
• AWS IoT Device SDK for Embedded C: https://github.com/aws/aws-iot-device-sdk-embedded-C (The esp-aws-iot component is based on this)
• ESP-IDF esp-aws-iot Component Documentation: Check the ESP-IDF Programming Guide for this component.
  • Example path: esp-idf/components/esp-aws-iot/README.md
  • Official Docs: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_aws_iot.html (Adjust version/target)
• AWS IoT Core MQTT Topics: https://docs.aws.amazon.com/iot/latest/developerguide/topics.html
• AWS IoT Device Shadow Service: https://docs.aws.amazon.com/iot/latest/developerguide/iot-device-shadows.html