Chapter 94: HTTPS with TLS/SSL in ESP-IDF

Chapter Objectives

After completing this chapter, you will be able to:

• Understand the fundamental need for secure communication in IoT applications.
• Grasp the basics of TLS/SSL, including its goals: confidentiality, integrity, and authentication.
• Learn about Public Key Infrastructure (PKI), digital certificates (X.509), and the role of Certificate Authorities (CAs).
• Configure the ESP-IDF esp_http_client to make HTTPS requests.
• Understand and implement server certificate verification on the ESP32.
• Embed CA root certificates in your ESP32 firmware for trusting servers.
• Discuss alternative methods for managing certificates.
• Implement secure HTTPS GET and POST requests to web servers.
• Be aware of the performance implications and resource considerations of using TLS on ESP32 devices.
• Troubleshoot common issues related to HTTPS connections.

Introduction

In Chapter 93, we explored how the ESP32 can act as an HTTP client, enabling it to interact with web servers and APIs. While HTTP is functional, it transmits data in plaintext, making it vulnerable to eavesdropping and modification. For any application that handles sensitive information—such as personal data, credentials, financial transactions, or critical control commands for IoT devices—unsecured communication is unacceptable.

graph LR
    subgraph "HTTPS Communication (Secure)"
        direction LR
        HTTPS_ESP32["<br><b>ESP32</b><br><i>Client</i>"]
        HTTPS_DATA["<br>Encrypted Data<br>(Confidential & Integrity-Protected)<br>🔒"]
        HTTPS_TUNNEL["<br>(Secure TLS/SSL Tunnel)<br><i>Protected Channel</i>"]
        HTTPS_SERVER["<br><b>Web Server</b>"]

        HTTPS_ESP32 -- "Sends/Receives" --> HTTPS_DATA
        HTTPS_DATA -- "Transmitted Via" --> HTTPS_TUNNEL
        HTTPS_TUNNEL -- "Reaches" --> HTTPS_SERVER
    end
    subgraph "HTTP Communication (Insecure)"
        direction LR
        HTTP_ESP32["<br><b>ESP32</b><br><i>Client</i>"]
        HTTP_DATA["<br>Plaintext Data<br>(e.g., credentials, sensor readings)<br>🔓"]
        HTTP_INTERNET["<br>(Internet/Network)<br><i>Vulnerable to Eavesdropping</i>"]
        HTTP_SERVER["<br><b>Web Server</b>"]

        HTTP_ESP32 -- "Sends/Receives" --> HTTP_DATA
        HTTP_DATA -- "Transmitted Over" --> HTTP_INTERNET
        HTTP_INTERNET -- "Reaches" --> HTTP_SERVER
    end



    %% Styling
    classDef default fill:#transparent,stroke:#4B5563,stroke-width:1px,color:#1F2937;
    classDef primary fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6
    classDef process fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF
    classDef danger fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B
    classDef success fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46

    class HTTP_ESP32,HTTPS_ESP32 primary
    class HTTP_SERVER,HTTPS_SERVER primary
    class HTTP_DATA,HTTP_INTERNET danger
    class HTTPS_DATA,HTTPS_TUNNEL success

This is where HTTPS (HTTP Secure) comes into play. HTTPS is essentially the HTTP protocol layered on top of a secure communication channel provided by TLS (Transport Layer Security) or its predecessor SSL (Secure Sockets Layer). TLS/SSL encrypts the data exchanged between the client (your ESP32) and the server, ensuring confidentiality. It also provides mechanisms for data integrity (ensuring data hasn’t been tampered with) and server authentication (verifying that you are talking to the legitimate server and not an imposter).

This chapter will guide you through the principles of TLS/SSL and how to implement HTTPS client functionality on your ESP32 using the esp_http_client component. We will focus on enabling your ESP32 to securely connect to servers, verify their identities using digital certificates, and exchange data over an encrypted channel. Mastering HTTPS is a cornerstone of developing robust and trustworthy IoT solutions.

Theory

What is TLS/SSL?

Transport Layer Security (TLS) is a cryptographic protocol designed to provide secure communication over a computer network. It is the successor to Secure Sockets Layer (SSL). Although the term SSL is still widely used, modern secure connections almost exclusively use TLS. For simplicity, we’ll often use “TLS” or “TLS/SSL”.

The primary goals of TLS are:

1. Confidentiality: Achieved through encryption. Data exchanged between the client and server is encrypted, making it unreadable to any third party who might intercept it.
2. Integrity: Achieved through message authentication codes (MACs). TLS ensures that the data has not been altered or corrupted during transmission.
3. Authentication: Primarily server authentication. TLS allows the client to verify the identity of the server it is connecting to, preventing man-in-the-middle (MITM) attacks. Client authentication (where the server also verifies the client’s identity) is also possible but less common for typical web browsing or simple API access.

TLS Handshake

Before any application data can be exchanged securely, the client and server perform a TLS handshake. This is a complex process, but a simplified overview includes:

1. Client Hello: The client initiates the handshake by sending a “ClientHello” message to the server. This message includes the TLS versions the client supports, a list of cipher suites (cryptographic algorithms for encryption, key exchange, and MACs) it can use, and a random number (client random).
2. Server Hello: The server responds with a “ServerHello” message, selecting the TLS version and cipher suite from the client’s list, and providing its own random number (server random).
3. Server Certificate: The server sends its digital certificate (and potentially intermediate certificates) to the client. This certificate contains the server’s public key and information about its identity, signed by a Certificate Authority (CA).
4. Server Key Exchange (Optional): Depending on the cipher suite, the server might send additional information needed for key exchange.
5. Server Hello Done: The server indicates it has finished sending its initial handshake messages.
6. Client Certificate Verification: The client verifies the server’s certificate (see PKI section below). This is a critical step for security.
7. Client Key Exchange: The client generates a “premaster secret” (a key component for the session keys), encrypts it with the server’s public key (obtained from the server’s certificate), and sends it to the server. Only the server, with its corresponding private key, can decrypt this.
8. Change Cipher Spec: Both client and server send “ChangeCipherSpec” messages, indicating that subsequent messages will be encrypted using the newly negotiated keys and algorithms.
9. Finished: Both client and server send “Finished” messages, which are encrypted and MAC’d with the new session keys. These messages confirm that the handshake was successful and that both parties have derived the same session keys.

Once the handshake is complete, the client and server can exchange encrypted application data (e.g., HTTP requests and responses).

sequenceDiagram


    participant C as Client (ESP32)
    participant S as Server

    C->>S: 1. ClientHello <br> (TLS versions, Cipher Suites, Client Random)
    activate S
    S-->>C: 2. ServerHello <br> (Selected TLS version, Cipher Suite, Server Random)
    S-->>C: 3. Server Certificate <br> (Server's Public Key, CA Signature)
    S-->>C: 4. ServerKeyExchange (Optional, e.g., for DHE/ECDHE)
    S-->>C: 5. ServerHelloDone
    deactivate S

    activate C
    C-->>C: 6. Verify Server Certificate <br> (Check signature, validity, trust chain, hostname)
    C->>S: 7. ClientKeyExchange <br> (Premaster Secret, encrypted with Server's Public Key)
    C->>S: 8. ChangeCipherSpec <br> (Future messages will be encrypted)
    activate S
    C->>S: 9. Finished (Encrypted Handshake Verification)
    deactivate C

    S-->>C: 10. ChangeCipherSpec <br> (Future messages will be encrypted)
    S-->>C: 11. Finished (Encrypted Handshake Verification)
    deactivate S

    loop Encrypted Application Data Exchange
        C->>S: HTTPS Request (Encrypted)
        S-->>C: HTTPS Response (Encrypted)
    end

Public Key Infrastructure (PKI)

PKI is a set of roles, policies, hardware, software, and procedures needed to create, manage, distribute, use, store, and revoke digital certificates and manage public-key encryption. It’s the framework that enables trust in TLS/SSL.

Asymmetric vs. Symmetric Encryption

• Symmetric Encryption: Uses the same key for both encryption and decryption. It’s fast but requires a secure way to share the key initially. TLS uses symmetric encryption for the bulk data transfer after the handshake.
• Asymmetric Encryption (Public-Key Cryptography): Uses a pair of keys: a public key (which can be shared widely) and a private key (kept secret). Data encrypted with the public key can only be decrypted with the corresponding private key, and vice-versa. TLS uses asymmetric encryption during the handshake (e.g., for the server to prove its identity and for the client to securely send the premaster secret).

Digital Certificates (X.509)

An X.509 digital certificate is an electronic document that uses a digital signature to bind a public key with an identity — information such as the name of a person or an organization, their address, and so forth.

Key information in a server certificate includes:

Certificate Authorities (CAs)

A Certificate Authority is a trusted entity that issues digital certificates. CAs validate the identity of entities (like websites) before issuing them a certificate. Browsers and operating systems (and thus our ESP32 applications) come with a pre-installed list of root CA certificates. These root CAs are globally trusted.

Chain of Trust

Often, server certificates are not directly signed by a root CA. Instead, they are signed by an intermediate CA, which in turn is signed by another intermediate CA or a root CA. This forms a certificate chain (or path).

Server Certificate (e.g., www.example.com) <- Signed by Intermediate CA 1

Intermediate CA 1 Certificate <- Signed by Intermediate CA 2 (or Root CA)

Intermediate CA 2 Certificate <- Signed by Root CA

Root CA Certificate (Self-signed, trusted by client)

To verify a server’s certificate, the client (ESP32) must be able to trace this chain back to a root CA certificate that it trusts (i.e., one that is embedded in its firmware or otherwise made available to it).

graph TD
    RootCA["<br><b>Root CA Certificate</b><br>(e.g., ISRG Root X1)<br><i>Self-Signed</i><br>🛡️ Trusted by Client (ESP32)"]
    IntermediateCA1["<br><b>Intermediate CA Certificate 1</b><br>(e.g., Let's Encrypt R3)"]
    ServerCert["<br><b>Server Certificate</b><br>(e.g., www.example.com)"]

    subgraph "Trust Anchor on Client (ESP32)"
        direction LR
        ClientTrustStore["<br>Client's Trust Store<br>(Contains Root CA Certs)"]
    end
    
    ServerCert -- "<i>Signed by</i><br>(Issuer: Intermediate CA 1)" --> IntermediateCA1
    IntermediateCA1 -- "<i>Signed by</i><br>(Issuer: Root CA)" --> RootCA
    RootCA -. "<i>Verified Against</i>" .-> ClientTrustStore


    %% Optional: Another Intermediate CA
    %% IntermediateCA2["<br><b>Intermediate CA Certificate 2</b><br>(e.g., Another Intermediate)"]
    %% IntermediateCA1 -- "<i>Signed by</i><br>(Issuer: Intermediate CA 2)" --> IntermediateCA2
    %% IntermediateCA2 -- "<i>Signed by</i><br>(Issuer: Root CA)" --> RootCA

    %% Styling
    classDef default fill:#transparent,stroke:#4B5563,color:#1F2937;
    classDef root fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46 
    classDef intermediate fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E 
    classDef server_cert fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF 
    classDef trust_store fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6 

    class RootCA root
    class IntermediateCA1 intermediate
    %% class IntermediateCA2 intermediate
    class ServerCert server_cert
    class ClientTrustStore trust_store

Server Authentication by the Client (ESP32)

When the ESP32 (as an HTTPS client) receives the server’s certificate during the TLS handshake, it performs several checks:

If all these checks pass, the server is considered authentic.

ESP-IDF, esp_http_client, and mbedTLS

ESP-IDF uses the mbedTLS library (a popular open-source SSL/TLS library) to provide TLS functionality. The esp_http_client component, when configured for HTTPS, leverages esp_tls (an abstraction layer over mbedTLS) to establish secure connections.

graph TD
    subgraph "User Application Layer"
        App["<br><b>Your ESP32 Application</b><br>(e.g., Sensor Data Upload, API Call)"]
    end

    subgraph "ESP-IDF HTTP Client Layer"
        HTTPClient["<br><b>esp_http_client</b><br>(Manages HTTP/HTTPS requests/responses,<br>headers, methods, etc.)"]
    end

    subgraph "ESP-IDF TLS Abstraction Layer"
        ESPTLS["<br><b>esp_tls</b><br>(Provides a unified API for TLS operations,<br>abstracts underlying TLS library)"]
    end

    subgraph "Core TLS/SSL Library"
        MBEDTLS["<br><b>mbedTLS Library</b><br>(Handles TLS handshake, encryption/decryption,<br>certificate parsing, cryptographic functions)"]
    end

    subgraph "Network Stack & Drivers"
        TCPIP["<br><b>LWIP TCP/IP Stack</b><br>(Manages network connections, sockets)"]
        WiFiDrv["<br><b>Wi-Fi Driver & Hardware</b><br>(Physical layer communication)"]
    end

    App -- "Uses API of" --> HTTPClient
    HTTPClient -- "<i>For HTTPS, uses</i>" --> ESPTLS
    ESPTLS -- "<i>Wraps/Uses</i>" --> MBEDTLS
    MBEDTLS -- "<i>Relies on</i>" --> TCPIP
    TCPIP -- "<i>Interfaces with</i>" --> WiFiDrv

    %% Styling
    classDef default fill:#transparent,stroke:#4B5563,color:#1F2937;
    classDef app fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6 
    classDef http fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF 
    classDef tls_abs fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E 
    classDef core_tls fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46 
    classDef network fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B 

    class App app
    class HTTPClient http
    class ESPTLS tls_abs
    class MBEDTLS core_tls
    class TCPIP,WiFiDrv network

You typically don’t interact with mbedTLS directly when using esp_http_client for basic HTTPS, but it’s working under the hood.

Practical Examples

The core change from HTTP to HTTPS using esp_http_client involves:

1. Changing the URL scheme from http:// to https://.
2. Setting the transport_type in esp_http_client_config_t to HTTP_TRANSPORT_OVER_SSL.
3. Providing the CA root certificate(s) needed to verify the server via the cert_pem field in the configuration or using esp_crt_bundle_attach for a common bundle of CAs.

Obtaining CA Root Certificates

To verify a server’s certificate, your ESP32 needs the root CA certificate that signed the server’s certificate chain.

• Browser Method:
  1. Open the HTTPS website in your browser (e.g., Chrome, Firefox).
  2. Click the padlock icon in the address bar.
  3. View certificate details. Look for the certificate chain or path.
  4. Identify the root CA at the top of the chain.
  5. You can often download or export the root CA certificate in PEM format from the browser’s certificate manager or find it on the CA’s website (e.g., Let’s Encrypt, DigiCert, Sectigo).
• OpenSSL Command Line:openssl s_client -connect example.com:443 -showcerts </dev/null
This command will display the server’s certificate chain. The last certificate in the chain is usually the root CA (or an intermediate that you might need to trace further). Copy the PEM block (—–BEGIN CERTIFICATE—– … —–END CERTIFICATE—–) for the required root CA.

PEM Format:

A CA certificate in PEM (Privacy Enhanced Mail) format looks like this:

-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG
A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv
... (many lines of Base64 encoded data) ...
b8ravHNjkOR/ez4KkEUDHYzT7pG9yS+R/0YgiycMO1L2E+xKj2stLAlO9wY0Jud4
JwBSW2fE0P0gP19i8F7PbI8D9p20dgA=
-----END CERTIFICATE-----

You will embed this string into your ESP32 code.

Example 1: Simple HTTPS GET Request

This example makes an HTTPS GET request to https://www.howsmyssl.com/a/check, which returns JSON data about the client’s TLS capabilities. We’ll use its root CA certificate. (As of my last update, howsmyssl.com might use Let’s Encrypt or another common CA. You should verify the current CA). For this example, let’s assume we need the “ISRG Root X1” certificate from Let’s Encrypt.

Project Setup:

1. Create a new ESP-IDF project: idf.py create-project https_get_example
2. Navigate into the project: cd https_get_example
3. Replace main/main.c.
4. Configure Wi-Fi in menuconfig.
5. Enable CONFIG_MBEDTLS_CERTIFICATE_BUNDLE in menuconfig (Component config -> mbedTLS -> Certificate Bundle) and select “Provide bundle” or “Default bundle” for easier setup with common CAs. If using a specific cert_pem, this might not be strictly necessary but the bundle is good practice.

main/main.c:

#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"

#include "esp_http_client.h"
#include "esp_netif.h"
#include "esp_crt_bundle.h" // For esp_crt_bundle_attach

#define WIFI_SSID      CONFIG_EXAMPLE_WIFI_SSID
#define WIFI_PASS      CONFIG_EXAMPLE_WIFI_PASSWORD
#define MAX_FAILURES   10

#define MAX_HTTP_OUTPUT_BUFFER 2048

static const char *TAG = "https_get_example";

// Wi-Fi event group and event_handler, wifi_init_sta (same as Chapter 93)
static EventGroupHandle_t s_wifi_event_group;
#define WIFI_CONNECTED_BIT BIT0
#define WIFI_FAIL_BIT      BIT1
static int s_retry_num = 0;

static void event_handler(void* arg, esp_event_base_t event_base,
                                int32_t event_id, void* event_data)
{
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        if (s_retry_num < MAX_FAILURES) {
            esp_wifi_connect();
            s_retry_num++;
            ESP_LOGI(TAG, "retry to connect to the AP");
        } else {
            xEventGroupSetBits(s_wifi_event_group, WIFI_FAIL_BIT);
        }
        ESP_LOGI(TAG,"connect to the AP fail");
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        ESP_LOGI(TAG, "got ip:" IPSTR, IP2STR(&event->ip_info.ip));
        s_retry_num = 0;
        xEventGroupSetBits(s_wifi_event_group, WIFI_CONNECTED_BIT);
    }
}

void wifi_init_sta(void)
{
    s_wifi_event_group = xEventGroupCreate();
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta();
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    esp_event_handler_instance_t instance_any_id;
    esp_event_handler_instance_t instance_got_ip;
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &event_handler, NULL, &instance_any_id));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &event_handler, NULL, &instance_got_ip));
    wifi_config_t wifi_config = {
        .sta = { .ssid = WIFI_SSID, .password = WIFI_PASS, .threshold.authmode = WIFI_AUTH_WPA2_PSK },
    };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA) );
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config) );
    ESP_ERROR_CHECK(esp_wifi_start() );
    ESP_LOGI(TAG, "wifi_init_sta finished.");
    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT | WIFI_FAIL_BIT, pdFALSE, pdFALSE, portMAX_DELAY);
    if (bits & WIFI_CONNECTED_BIT) { ESP_LOGI(TAG, "connected to ap SSID:%s", WIFI_SSID); }
    else if (bits & WIFI_FAIL_BIT) { ESP_LOGI(TAG, "Failed to connect to SSID:%s", WIFI_SSID); }
    else { ESP_LOGE(TAG, "UNEXPECTED EVENT"); }
}

// HTTP event handler (same as Chapter 93, Example 1)
esp_err_t _http_event_handler(esp_http_client_event_t *evt)
{
    static char *output_buffer;  // Buffer to store response of http request
    static int output_len;       // Stores number of bytes read
    switch(evt->event_id) {
        case HTTP_EVENT_ERROR:
            ESP_LOGD(TAG, "HTTP_EVENT_ERROR");
            break;
        case HTTP_EVENT_ON_CONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_CONNECTED");
            break;
        case HTTP_EVENT_HEADER_SENT:
            ESP_LOGD(TAG, "HTTP_EVENT_HEADER_SENT");
            break;
        case HTTP_EVENT_ON_HEADER:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_HEADER, key=%s, value=%s", evt->header_key, evt->header_value);
            break;
        case HTTP_EVENT_ON_DATA:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
            // If user_data is not NULL, it means buffer is passed from user, append data to it.
            // Ensure buffer is large enough.
            if (evt->user_data) {
                // Check if buffer is large enough
                char *user_buffer = (char *)evt->user_data;
                if (output_len + evt->data_len < MAX_HTTP_OUTPUT_BUFFER) {
                    memcpy(user_buffer + output_len, evt->data, evt->data_len);
                    output_len += evt->data_len;
                } else {
                    ESP_LOGW(TAG, "Output buffer too small for all data");
                    // Handle buffer overflow, e.g. by processing chunk by chunk or truncating
                }
            } else { // Dynamic allocation (less safe for embedded if not careful)
                if (output_buffer == NULL) {
                    // Consider esp_http_client_get_content_length(evt->client) if available
                    // For this example, let's assume a fixed max or chunked processing
                    output_buffer = (char *) malloc(MAX_HTTP_OUTPUT_BUFFER); // Example fixed size
                    output_len = 0;
                    if (output_buffer == NULL) {
                        ESP_LOGE(TAG, "Failed to allocate memory for output buffer");
                        return ESP_FAIL;
                    }
                }
                 if (output_len + evt->data_len < MAX_HTTP_OUTPUT_BUFFER) {
                    memcpy(output_buffer + output_len, evt->data, evt->data_len);
                    output_len += evt->data_len;
                }
            }
            break;
        case HTTP_EVENT_ON_FINISH:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_FINISH");
            char *buffer_to_log = output_buffer;
            if (evt->user_data) { // If user_data was used, log from there
                buffer_to_log = (char *)evt->user_data;
                buffer_to_log[output_len] = '\0'; // Null-terminate
            } else if (output_buffer) {
                 output_buffer[output_len] = '\0'; // Null-terminate
            }

            if (buffer_to_log) {
                ESP_LOGI(TAG, "Received data (len %d):\n%.*s", output_len, output_len, buffer_to_log);
            }
            
            if (output_buffer && !evt->user_data) { // Free only if dynamically allocated
                free(output_buffer);
                output_buffer = NULL;
            }
            output_len = 0; // Reset for next request if client is reused
            break;
        case HTTP_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "HTTP_EVENT_DISCONNECTED");
            int mbedtls_err = 0;
            esp_err_t err = esp_tls_get_and_clear_last_error((esp_tls_error_handle_t)evt->data, &mbedtls_err, NULL);
            if (err != 0) {
                 ESP_LOGI(TAG, "Last esp_tls error: 0x%x", err);
                 ESP_LOGI(TAG, "Last mbedtls error: 0x%x", mbedtls_err);
            }
            if (output_buffer && !evt->user_data) {
                free(output_buffer);
                output_buffer = NULL;
            }
            output_len = 0;
            break;
        case HTTP_EVENT_REDIRECT:
            ESP_LOGD(TAG, "HTTP_EVENT_REDIRECT");
            esp_http_client_set_redirection(evt->client);
            break;
    }
    return ESP_OK;
}


// You would get this from the CA's website or by inspecting the server's certificate chain.
// Example: ISRG Root X1 (Let's Encrypt)
// IMPORTANT: This certificate might expire or change. Always use a current one.
// For production, consider using the ESP-IDF certificate bundle feature.
const char *howsmyssl_root_cert_pem_start asm("_binary_howsmyssl_pem_start");
const char *howsmyssl_root_cert_pem_end asm("_binary_howsmyssl_pem_end");
// To use this, create a file named "howsmyssl.pem" in your main directory
// and add its content. Then, in CMakeLists.txt (main), add:
// target_add_binary_data(${COMPONENT_TARGET} "howsmyssl.pem" TEXT)


static void https_get_task(void *pvParameters)
{
    xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdTRUE, portMAX_DELAY);
    ESP_LOGI(TAG, "Connected to Wi-Fi, starting HTTPS GET task.");

    char local_response_buffer[MAX_HTTP_OUTPUT_BUFFER] = {0};

    esp_http_client_config_t config = {
        .url = "https://www.howsmyssl.com/a/check",
        .event_handler = _http_event_handler,
        .user_data = local_response_buffer,
        .transport_type = HTTP_TRANSPORT_OVER_SSL,  // Specify HTTPS
        //.cert_pem = howsmyssl_root_cert_pem_start, // Use this if embedding a specific cert
        .crt_bundle_attach = esp_crt_bundle_attach, // Use ESP-IDF's certificate bundle
                                                    // Requires CONFIG_MBEDTLS_CERTIFICATE_BUNDLE
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);

    esp_err_t err = esp_http_client_perform(client);
    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTPS GET Status = %d, content_length = %lld",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
    } else {
        ESP_LOGE(TAG, "HTTPS GET request failed: %s (0x%x)", esp_err_to_name(err), err);
    }

    esp_http_client_cleanup(client);
    vTaskDelete(NULL);
}

void app_main(void)
{
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    wifi_init_sta();
    
    // Ensure SNTP is initialized for time synchronization, which is crucial for certificate validation
    // See Chapter 98 for SNTP details. For this example, we assume time might be set or
    // certificate check might be lenient if time is not perfectly synced yet on first boot.
    // esp_sntp_config_t sntp_config = ESP_NETIF_SNTP_DEFAULT_CONFIG("pool.ntp.org");
    // esp_netif_sntp_init(&sntp_config);
    // if (esp_netif_sntp_sync_wait(pdMS_TO_TICKS(10000)) != ESP_OK) {
    //    ESP_LOGW(TAG, "Failed to sync SNTP time!");
    // }


    xTaskCreate(&https_get_task, "https_get_task", 8192 * 2, NULL, 5, NULL); // Increased stack for TLS
}

main/CMakeLists.txt:

Add the following line if you are embedding a specific certificate named howsmyssl.pem:

idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS ".")
# Add this line if you create howsmyssl.pem in the main directory:
# target_add_binary_data(${COMPONENT_TARGET} "howsmyssl.pem" TEXT)

If you are using esp_crt_bundle_attach, you don’t need target_add_binary_data.

Build and Flash Instructions:

1. Set target, configure Wi-Fi.
2. Enable Certificate Bundle: idf.py menuconfig -> Component config -> mbedTLS -> Certificate Bundle -> Check [*] Provide bundle of common CA certificates. Save and exit.
3. Build: idf.py build
4. Flash: idf.py -p /dev/ttyUSB0 flash monitor

Observe:

The ESP32 will connect to Wi-Fi and attempt the HTTPS GET request. The response from howsmyssl.com (JSON detailing TLS status) should be printed. If there are certificate verification errors, they will be logged.

Important Note on cert_pem vs crt_bundle_attach:

• Using config.cert_pem = specific_ca_pem_string; requires you to manage and embed individual CA certificates.
• Using config.crt_bundle_attach = esp_crt_bundle_attach; (and enabling CONFIG_MBEDTLS_CERTIFICATE_BUNDLE in menuconfig) tells esp_http_client to use a pre-compiled bundle of common CA certificates provided by ESP-IDF. This is often more convenient for general-purpose HTTPS to public servers, as it includes many common CAs.

Example 2: HTTPS POST Request

This example sends JSON data via an HTTPS POST request to https://httpbin.org/post.

Project Setup: (Similar to HTTPS GET example, new project https_post_example)

main/main.c:

(Wi-Fi init and event handler can be reused from Example 1. The main changes are in the https_post_task and esp_http_client_config_t.)

#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"

#include "esp_http_client.h"
#include "esp_netif.h"
#include "esp_crt_bundle.h"

#define WIFI_SSID      CONFIG_EXAMPLE_WIFI_SSID
#define WIFI_PASS      CONFIG_EXAMPLE_WIFI_PASSWORD
#define MAX_FAILURES   10
#define MAX_HTTP_OUTPUT_BUFFER 2048
static const char *TAG = "https_post_example";

// Wi-Fi related code (event_handler, wifi_init_sta, s_wifi_event_group etc.)
// is identical to Example 1. For brevity, it's not repeated here.
// Please copy it from the HTTPS GET example.
// ... (Copy Wi-Fi init and event handler here) ...
static EventGroupHandle_t s_wifi_event_group;
#define WIFI_CONNECTED_BIT BIT0
#define WIFI_FAIL_BIT      BIT1
static int s_retry_num = 0;

static void event_handler(void* arg, esp_event_base_t event_base,
                                int32_t event_id, void* event_data)
{
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        if (s_retry_num < MAX_FAILURES) {
            esp_wifi_connect();
            s_retry_num++;
            ESP_LOGI(TAG, "retry to connect to the AP");
        } else {
            xEventGroupSetBits(s_wifi_event_group, WIFI_FAIL_BIT);
        }
        ESP_LOGI(TAG,"connect to the AP fail");
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        ESP_LOGI(TAG, "got ip:" IPSTR, IP2STR(&event->ip_info.ip));
        s_retry_num = 0;
        xEventGroupSetBits(s_wifi_event_group, WIFI_CONNECTED_BIT);
    }
}

void wifi_init_sta(void)
{
    s_wifi_event_group = xEventGroupCreate();
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta();
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    esp_event_handler_instance_t instance_any_id;
    esp_event_handler_instance_t instance_got_ip;
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &event_handler, NULL, &instance_any_id));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &event_handler, NULL, &instance_got_ip));
    wifi_config_t wifi_config = {
        .sta = { .ssid = WIFI_SSID, .password = WIFI_PASS, .threshold.authmode = WIFI_AUTH_WPA2_PSK },
    };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA) );
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config) );
    ESP_ERROR_CHECK(esp_wifi_start() );
    ESP_LOGI(TAG, "wifi_init_sta finished.");
    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT | WIFI_FAIL_BIT, pdFALSE, pdFALSE, portMAX_DELAY);
    if (bits & WIFI_CONNECTED_BIT) { ESP_LOGI(TAG, "connected to ap SSID:%s", WIFI_SSID); }
    else if (bits & WIFI_FAIL_BIT) { ESP_LOGI(TAG, "Failed to connect to SSID:%s", WIFI_SSID); }
    else { ESP_LOGE(TAG, "UNEXPECTED EVENT"); }
}


// HTTP event handler (can be the same as Example 1 or a simplified one for POST response)
esp_err_t _http_event_handler_for_post(esp_http_client_event_t *evt) {
    static char response_data[MAX_HTTP_OUTPUT_BUFFER];
    static int response_data_len = 0;

    switch(evt->event_id) {
        case HTTP_EVENT_ERROR:
            ESP_LOGD(TAG, "HTTP_EVENT_ERROR");
            break;
        case HTTP_EVENT_ON_CONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_CONNECTED");
            response_data_len = 0; // Reset buffer length for new connection
            break;
        case HTTP_EVENT_ON_HEADER:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_HEADER, key=%s, value=%s", evt->header_key, evt->header_value);
            break;
        case HTTP_EVENT_ON_DATA:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
            if (response_data_len + evt->data_len < MAX_HTTP_OUTPUT_BUFFER) {
                memcpy(response_data + response_data_len, evt->data, evt->data_len);
                response_data_len += evt->data_len;
            } else {
                ESP_LOGW(TAG, "Response buffer overflow");
            }
            break;
        case HTTP_EVENT_ON_FINISH:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_FINISH");
            response_data[response_data_len] = '\0'; // Null-terminate
            ESP_LOGI(TAG, "Received HTTPS POST Response (len %d):\n%s", response_data_len, response_data);
            response_data_len = 0; // Reset for next potential request
            break;
        case HTTP_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "HTTP_EVENT_DISCONNECTED");
            int mbedtls_err = 0;
            esp_err_t err = esp_tls_get_and_clear_last_error((esp_tls_error_handle_t)evt->data, &mbedtls_err, NULL);
            if (err != 0) {
                 ESP_LOGI(TAG, "Last esp_tls error: 0x%x", err);
                 ESP_LOGI(TAG, "Last mbedtls error: 0x%x", mbedtls_err);
            }
            response_data_len = 0;
            break;
        case HTTP_EVENT_REDIRECT:
            ESP_LOGD(TAG, "HTTP_EVENT_REDIRECT");
            esp_http_client_set_redirection(evt->client);
            break;
    }
    return ESP_OK;
}


static void https_post_task(void *pvParameters)
{
    xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdTRUE, portMAX_DELAY);
    ESP_LOGI(TAG, "Connected to Wi-Fi, starting HTTPS POST task.");

    esp_http_client_config_t config = {
        .url = "https://httpbin.org/post",
        .method = HTTP_METHOD_POST,
        .event_handler = _http_event_handler_for_post,
        .transport_type = HTTP_TRANSPORT_OVER_SSL,
        .crt_bundle_attach = esp_crt_bundle_attach, // Use ESP-IDF's certificate bundle
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);

    const char *post_data = "{\"device\":\"ESP32\", \"value\":42, \"secure\":true}";
    esp_http_client_set_header(client, "Content-Type", "application/json");
    esp_http_client_set_post_field(client, post_data, strlen(post_data));

    ESP_LOGI(TAG, "Performing HTTPS POST to %s", config.url);
    ESP_LOGI(TAG, "POST data: %s", post_data);

    esp_err_t err = esp_http_client_perform(client);
    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTPS POST Status = %d, content_length = %lld",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
    } else {
        ESP_LOGE(TAG, "HTTPS POST request failed: %s (0x%x)", esp_err_to_name(err), err);
    }

    esp_http_client_cleanup(client);
    vTaskDelete(NULL);
}

void app_main(void)
{
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    wifi_init_sta();
    // SNTP init recommended for time sync

    xTaskCreate(&https_post_task, "https_post_task", 8192 * 2, NULL, 5, NULL); // Increased stack
}

Build and Flash Instructions: (Same as HTTPS GET example, ensure Certificate Bundle is enabled in menuconfig)

Observe:

The ESP32 will send a secure POST request. The response from httpbin.org/post (echoing your request details) will be printed.

Certificate Handling Alternatives

Instead of embedding certificates directly in code or using the global bundle:

Variant Notes

• Hardware Cryptographic Acceleration:
  • ESP32, ESP32-S2, ESP32-S3: These variants have dedicated hardware accelerators for cryptographic operations like AES, SHA, RSA, and ECC. This significantly speeds up TLS handshakes and data encryption/decryption compared to pure software implementations.
  • ESP32-C3: Early versions relied more on software for some crypto, but later ROM versions and IDF improvements have optimized this. It generally performs well for typical IoT use cases.
  • ESP32-C6, ESP32-H2: These newer RISC-V based chips also include hardware crypto support, offering good TLS performance.
  • The impact of hardware acceleration is most noticeable during the TLS handshake, which is computationally intensive.
• Memory Usage (RAM and Flash):
  • mbedTLS Library: The TLS library itself adds to the firmware size (flash).
  • Session Buffers: During a TLS session, mbedTLS requires RAM for input/output buffers, certificate parsing, and session state. This can be significant (tens of KBs).
  • Task Stack Size: Tasks performing HTTPS operations often require larger stack sizes (e.g., 8KB or more, as seen in examples) compared to plain HTTP or other tasks, due to the depth of function calls within mbedTLS.
  • Certificate Storage: Embedding certificates directly increases firmware size. Storing them in NVS/SPIFFS uses flash partition space. The ESP-IDF certificate bundle also adds to firmware size.
  • Resource-constrained variants (e.g., those with less RAM) need careful management. Consider optimizing mbedTLS configuration in menuconfig if memory is extremely tight (Component config -> mbedTLS -> Memory allocation strategy / Tweaks).
• Performance Impact:
  • Latency: The TLS handshake adds noticeable latency to the initial connection setup (can be hundreds of milliseconds to seconds, depending on server, network, and ESP32 variant).
  • Throughput: Encryption and decryption of data add CPU overhead, which can slightly reduce maximum data throughput compared to plain TCP or HTTP, though often network bandwidth is the bottleneck for typical IoT payloads.

For most common IoT applications (sending sensor data, receiving commands), all ESP32 variants are capable of performing HTTPS, but developers should be mindful of the increased resource usage and handshake latency.

Memory Effects:

Performance Implications:

Common Mistakes & Troubleshooting Tips

Tip: When debugging HTTPS issues, temporarily setting skip_common_name_check = true or skip_cert_common_name_check = true (depending on IDF version, check esp_http_client_config_t fields) in the config can help isolate if the problem is CN mismatch vs. other trust issues. Never leave this enabled in production!

Exercises

1. Secure Weather API Client:
   • Modify Exercise 1 from Chapter 93 (Fetch and Parse Weather Data) to use HTTPS.
   • Find a weather API that supports HTTPS (most do).
   • Obtain and configure the necessary CA certificate (or use the ESP-IDF certificate bundle).
   • Ensure the data is fetched securely and parsed.
2. Securely POST ESP32 Health Data:
   • Modify Exercise 2 from Chapter 93 (POST ESP32 Chip Information).
   • Change the endpoint to https://httpbin.org/post.
   • Ensure the chip information is POSTed over HTTPS.
   • Verify the response from httpbin.org.
3. Certificate Pinning (Conceptual Research & Discussion):
   • Research the concept of “certificate pinning” or “public key pinning.”
   • Discuss its advantages (enhanced security against compromised CAs) and disadvantages (maintenance overhead if pinned certs change).
   • How might you implement a basic form of certificate pinning with esp_http_client? (Hint: You might use a specific server certificate via cert_pem instead of just a root CA, or check the public key of the received certificate). This is more of a research and design exercise.
4. Error Analysis from esp_tls_get_and_clear_last_error:
   • Intentionally cause an HTTPS connection to fail (e.g., by using a wrong CA certificate for cert_pem, or connecting to a server whose certificate is expired or uses a self-signed cert without providing the CA).
   • In the HTTP_EVENT_DISCONNECTED or HTTP_EVENT_ERROR handler, use esp_tls_get_and_clear_last_error() to get the specific esp_tls error and the underlying mbedTLS error code.
   • Look up the mbedTLS error code (e.g., in mbedtls/error.h or online) to understand the precise reason for the failure. Log these error codes.

Summary

• HTTPS (HTTP Secure) encrypts HTTP communication using TLS/SSL, providing confidentiality, integrity, and server authentication.
• TLS/SSL relies on a handshake process to negotiate security parameters and establish shared secret keys.
• PKI and X.509 digital certificates are fundamental for authentication. Clients verify server certificates against a list of trusted Root CAs.
• The ESP-IDF esp_http_client supports HTTPS by setting transport_type = HTTP_TRANSPORT_OVER_SSL and providing server CA certificate information (via cert_pem or crt_bundle_attach).
• Accurate system time on the ESP32 is crucial for validating certificate validity periods.
• Hardware cryptographic acceleration in many ESP32 variants improves TLS performance.
• TLS operations increase RAM, flash, and task stack requirements compared to plain HTTP.
• Troubleshooting HTTPS often involves checking certificates, CAs, system time, and interpreting TLS-specific error codes.

Further Reading

• ESP-IDF Programming Guide – esp_tls:
  • esp_tls documentation (This is the underlying TLS abstraction used by esp_http_client)
• ESP-IDF Programming Guide – Certificate Bundle:
  • Certificate Bundle documentation
• mbedTLS Knowledge Base:
  • mbedTLS Official Documentation
  • Common mbedTLS X.509 error codes
• Let’s Encrypt Documentation: (A popular free CA)
  • Let’s Encrypt – Chain of Trust
• MDN Web Docs – TLS:
  • Transport Layer Security (TLS)