Chapter 111: HTTP/HTTPS RESTful Client Implementation

Chapter Objectives

By the end of this chapter, you will be able to:

• Understand the principles of RESTful APIs.
• Implement HTTP GET requests to retrieve data from REST APIs.
• Implement HTTP POST requests to send data to REST APIs.
• Handle JSON data in the context of REST API communication.
• Securely communicate with REST APIs using HTTPS.
• Understand considerations for different ESP32 variants when implementing RESTful clients.
• Troubleshoot common issues related to HTTP/HTTPS RESTful client implementation.

Introduction

In the rapidly expanding world of the Internet of Things (IoT), devices often need to communicate with web services to send data, receive commands, or access information. Representational State Transfer (REST) has become a de facto standard for designing networked applications, offering a stateless, client-server architecture that is simple, scalable, and widely adopted. RESTful APIs, which adhere to REST principles, allow devices like the ESP32 to interact with a vast array of cloud platforms, databases, and third-party services using standard HTTP/HTTPS methods.

This chapter builds upon your knowledge of HTTP and HTTPS (covered in Chapters 93 and 94) and focuses on how to implement a RESTful client on your ESP32. You will learn how to consume REST APIs, enabling your ESP32 projects to fetch data from online sources, report sensor readings to cloud services, and interact with other web-connected systems. Mastering RESTful client implementation is a crucial skill for developing sophisticated IoT applications.

Prerequisite Note: This chapter assumes you have a working Wi-Fi connection on your ESP32. Please refer to Volume 2: “Connectivity Fundamentals – WiFi” (Chapters 26-50) for detailed instructions on establishing a Wi-Fi connection.

Theory

What is HTTP/HTTPS?

Before diving into REST, let’s briefly revisit HTTP (Hypertext Transfer Protocol) and HTTPS (HTTP Secure).

• HTTP is the foundation of data communication for the World Wide Web. It’s an application-layer protocol that defines how messages are formatted and transmitted, and what actions web servers and browsers should take in response to various commands. It operates on a client-server model, where the client (e.g., your ESP32) sends a request to a server, and the server sends back a response.
• HTTPS is the secure version of HTTP. It encrypts the communication between the client and server using TLS/SSL protocols (Transport Layer Security/Secure Sockets Layer), ensuring data confidentiality, integrity, and server authentication. This is crucial for sensitive data exchange.

Chapter 93 and Chapter 94 provided a detailed look into implementing basic HTTP and HTTPS clients. Here, we leverage that understanding for RESTful communication.

What is a RESTful API?

REST (Representational State Transfer) is an architectural style for designing networked applications. It’s not a protocol or a standard itself but a set of constraints that, when applied, lead to scalable, stateless, and cacheable web services. APIs that adhere to these constraints are called RESTful APIs.

Key principles of REST:

1. Client-Server Architecture: The client (e.g., ESP32) is responsible for the user interface and user experience, while the server is responsible for processing requests, managing resources, and storing data. This separation of concerns improves portability and scalability.
2. Statelessness: Each request from a client to a server must contain all the information needed to understand and process the request. The server does not store any client context (state) between requests. If session state is required, it’s managed by the client. This enhances reliability and scalability.
3. Cacheability: Responses from the server can be explicitly marked as cacheable or non-cacheable. This allows clients or intermediaries to reuse response data for equivalent later requests, improving performance and efficiency.
4. Uniform Interface: This is a core principle that simplifies and decouples the architecture. It consists of several sub-constraints:
   • Resource Identification: Resources (e.g., a sensor reading, a user profile) are identified by Uniform Resource Identifiers (URIs), typically URLs. For example, /sensors/temperature/1 could represent a specific temperature sensor.
   • Manipulation of Resources Through Representations: Clients interact with resources by exchanging representations of these resources. A common representation format is JSON (JavaScript Object Notation), but XML or plain text can also be used. For instance, when you request a sensor’s state, you get a JSON object representing that state.
   • Self-descriptive Messages: Each message (request or response) includes enough information to describe how to process it. This is often achieved using standard HTTP methods, status codes, and media types (e.g., Content-Type: application/json).
   • Hypermedia as the Engine of Application State (HATEOAS): This (optional but recommended) constraint means that responses from the server should include links (hypermedia) that guide the client on how to discover other available actions or related resources.
5. Layered System: A client cannot ordinarily tell whether it is connected directly to the end server or to an intermediary along the way (like a load balancer or proxy). This allows for better scalability and security.

Common HTTP Methods in REST APIs:

RESTful APIs use standard HTTP methods (verbs) to perform operations on resources:

• GET: Retrieves a representation of a resource. (e.g., GET /users/123 to fetch user 123’s details). This is a safe and idempotent operation (making the same GET request multiple times has the same effect as making it once).
• POST: Creates a new resource. Often used to submit data to be processed (e.g., POST /users with user details in the request body to create a new user). Not idempotent.
• PUT: Updates an existing resource or creates it if it doesn’t exist. The entire representation of the resource is typically sent in the request body (e.g., PUT /users/123 with updated user details). Idempotent.
• DELETE: Removes a resource (e.g., DELETE /users/123 to delete user 123). Idempotent.
• PATCH: Applies partial modifications to a resource (e.g., PATCH /users/123 with only the fields to be updated). Not necessarily idempotent.
• HEAD: Similar to GET, but only retrieves the headers of the response, not the body. Useful for checking if a resource exists or getting metadata.
• OPTIONS: Retrieves the supported HTTP methods and other communication options for a resource or server.

Typical RESTful interaction flow using HTTP methods.

sequenceDiagram
    participant C as ESP32 Application
    participant API as API Endpoint
    participant BL as Business Logic
    participant DB as Database/Resources

    Note over C, DB: GET Request - Retrieve Temperature Data
    C->>+API: 1. HTTP GET /sensors/temp
    API->>+BL: 2. Request Resource
    BL->>+DB: 3. Retrieve Data
    DB-->>-BL: 4. Data
    BL-->>-API: 5. Prepare Response
    API-->>-C: 6. HTTP 200 OK<br/>{"value": 25.5}

    Note over C, DB: POST Request - Send New Temperature Data
    C->>+API: 7. HTTP POST /sensors/temp<br/>{"value": 26.1}
    API->>+BL: 8. Process Data
    BL->>+DB: 9. Store/Update Data
    DB-->>-BL: 10. Acknowledge
    BL-->>-API: 11. Prepare Response
    API-->>-C: 12. HTTP 201 Created<br/>{"id": "new_reading_id"}

    Note over C, DB: PUT Request - Update Configuration
    C->>+API: 13. HTTP PUT /sensors/config/1<br/>{"threshold": 30}
    API->>+BL: 14. Process Update
    BL->>+DB: 15. Update Resource
    DB-->>-BL: 16. Acknowledge
    BL-->>-API: 17. Prepare Response
    API-->>-C: 18. HTTP 200 OK<br/>{"threshold": 30}

    Note over C, DB: DELETE Request - Remove Old Data
    C->>+API: 19. HTTP DELETE /sensors/temp/old_id
    API->>+BL: 20. Request Deletion
    BL->>+DB: 21. Delete Resource
    DB-->>-BL: 22. Acknowledge
    BL-->>-API: 23. Prepare Response
    API-->>-C: 24. HTTP 204 No Content

JSON: The Lingua Franca of REST APIs

While REST doesn’t mandate a specific data format, JSON (JavaScript Object Notation) has become the most popular choice for representing resources in RESTful APIs due to its simplicity, human-readability, and ease of parsing by machines.

A JSON object consists of key-value pairs, similar to a dictionary or map in many programming languages.

Example JSON:

{
  "sensorId": "temp001",
  "temperature": 25.5,
  "unit": "Celsius",
  "timestamp": "2025-05-29T12:00:00Z",
  "active": true,
  "location": {
    "room": "living_room",
    "coordinates": [38.9637, 35.2433]
  },
  "tags": ["indoor", "ambient"]
}

When your ESP32 interacts with a REST API, it will often send JSON in the body of POST or PUT requests and receive JSON in the body of responses to GET requests. You’ll need a JSON parsing library (like cJSON, which is included in ESP-IDF) to work with this data.

ESP-IDF esp_http_client API for RESTful Operations

The ESP-IDF provides the esp_http_client component, which is a versatile HTTP/HTTPS client library. We introduced it in Chapter 93. Now, let’s focus on its application for RESTful services.

Key steps for using esp_http_client:

1. Configuration (esp_http_client_config_t):This structure is used to set up the HTTP client. Important fields for REST include:
   • url: The full URL of the API endpoint.
   • host, path: Alternatively, specify host and path separately.
   • method: The HTTP method (e.g., HTTP_METHOD_GET, HTTP_METHOD_POST).
   • event_handler: A callback function to handle HTTP events (connection, headers received, data received, disconnection, errors).
   • user_data: Custom data to pass to the event handler.
   • timeout_ms: Connection timeout.
   • crt_bundle_attach or cert_pem: For HTTPS, to provide CA certificates for server verification (see Chapter 94).
   • transport_type: HTTP_TRANSPORT_OVER_TCP (for HTTP) or HTTP_TRANSPORT_OVER_SSL (for HTTPS).
2. Initialization:esp_http_client_handle_t client = esp_http_client_init(&config);
3. Setting Request Headers (Optional but often needed for REST):esp_http_client_set_header(client, “Content-Type”, “application/json”);esp_http_client_set_header(client, “Authorization”, “Bearer your_api_token”);
4. Setting Request Body (for POST, PUT, PATCH):esp_http_client_set_post_field(client, json_payload_string, strlen(json_payload_string));
5. Performing the Request:esp_err_t err = esp_http_client_perform(client);This is a blocking call. For non-blocking/asynchronous operations, you’d manage connections and data transfer more manually with esp_http_client_open(), esp_http_client_write(), esp_http_client_read(), etc. However, esp_http_client_perform() is often sufficient for many REST client tasks.
6. Getting Response Information (if esp_http_client_perform was successful):
   • int status_code = esp_http_client_get_status_code(client);
   • int64_t content_length = esp_http_client_get_content_length(client);
   • Response body data is typically handled within the HTTP_EVENT_ON_DATA event in your event handler.
7. Cleanup:esp_http_client_cleanup(client);

%%{init: {"flowchart": {"htmlLabels": true}} }%%
graph TD
    A[Start: Configure Request] -- "esp_http_client_config_t config;" --> B("Initialize Client <br> esp_http_client_handle_t client = <br> esp_http_client_init(&config)");
    B -- Optional --> C{Need Custom Headers?};
    C -- Yes --> D["Set Headers <br> esp_http_client_set_header(...)"] -- HTTP Methods --> E;
    C -- No --> E{Method is POST/PUT/PATCH?};
    E -- Yes --> F["Set POST Data (Body) <br> esp_http_client_set_post_field(...)"] -- Perform Request --> G;
    E -- No (GET/DELETE etc.) --> G["Perform Request <br> esp_err_t err = <br> esp_http_client_perform(client)"];
    
    G -- Request Execution --> H{"Request Successful? <br> (err == ESP_OK)"};
    H -- Yes --> I["Get Response Info <br> esp_http_client_get_status_code() <br> esp_http_client_get_content_length()"];
    I -- Response Data Handled in Event Handler --> J["Cleanup Client <br> esp_http_client_cleanup(client)"];
    H -- No (Error) --> K["Handle Error <br> Log esp_err_to_name(err)"];
    K -- After Error Handling --> J;
    J --> L[End];

    subgraph "Event Handler (_http_event_handler)"
        direction TB
        EV_START[Event Triggered] --> EV_SWITCH{"Switch (evt->event_id)"};
        EV_SWITCH -- HTTP_EVENT_ON_CONNECTED --> EV_CON(Handle Connection);
        EV_SWITCH -- HTTP_EVENT_ON_HEADER --> EV_HEAD(Handle Header <br> evt->header_key, evt->header_value);
        EV_SWITCH -- HTTP_EVENT_ON_DATA --> EV_DATA(Handle Data Chunk <br> Accumulate/Process evt->data);
        EV_SWITCH -- HTTP_EVENT_ON_FINISH --> EV_FIN(Finalize Response <br> Process accumulated data, free buffer);
        EV_SWITCH -- HTTP_EVENT_ERROR --> EV_ERR(Handle HTTP Error);
        EV_SWITCH -- HTTP_EVENT_DISCONNECTED --> EV_DISC(Handle Disconnection <br> Free resources);
        EV_CON --> EV_NEXT;
        EV_HEAD --> EV_NEXT;
        EV_DATA --> EV_NEXT;
        EV_FIN --> EV_NEXT;
        EV_ERR --> EV_NEXT;
        EV_DISC --> EV_NEXT[Return ESP_OK];
    end

    G -.-> EV_START;


    classDef startEnd fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef process fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decision fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef io fill:#D1FAE5,stroke:#059669,stroke-width:1px,color:#065F46;
    classDef error fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef eventHandler fill:#E0E7FF,stroke:#4338CA,stroke-width:1px,color:#3730A3;

    class A,L startEnd;
    class B,D,F,G,I,J process;
    class C,E,H decision;
    class K error;
    class EV_START,EV_SWITCH,EV_CON,EV_HEAD,EV_DATA,EV_FIN,EV_ERR,EV_DISC,EV_NEXT eventHandler;

Summary of Fields:

Event Handling:

The event handler is crucial for processing responses, especially the body content, which might arrive in chunks.

%%{init: {"flowchart": {"htmlLabels": true}} }%%
graph TD
    Start["HTTP Event Received <br> (esp_http_client_event_t *evt)"] --> EventSwitch{"switch(evt->event_id)"};

    EventSwitch -- "HTTP_EVENT_ERROR" --> Error["Log Error <br> Potentially free resources"];
    EventSwitch -- "HTTP_EVENT_ON_CONNECTED" --> Connected["Log \"Connected\""];
    EventSwitch -- "HTTP_EVENT_HEADER_SENT" --> HeaderSent["Log \"Header Sent\""];
    EventSwitch -- "HTTP_EVENT_ON_HEADER" --> OnHeader["Log Header: key, value"];
    EventSwitch -- "HTTP_EVENT_ON_DATA" --> OnData{"Process Data Chunk (evt->data_len)"};
    EventSwitch -- "HTTP_EVENT_ON_FINISH" --> OnFinish["Log \Finish\ <br> Process complete output_buffer <br> Free output_buffer"];
    EventSwitch -- "HTTP_EVENT_DISCONNECTED" --> Disconnected["Log \Disconnected\ <br> Free any remaining resources <br> Signal completion if needed"];
    EventSwitch -- "HTTP_EVENT_REDIRECT" --> Redirect["Log \Redirect\ <br> esp_http_client_set_redirection(evt->client)"];

    OnData --> CheckChunked{"Is response chunked? <br> esp_http_client_is_chunked_response()"};
    CheckChunked -- "Yes (Chunked)" --> ChunkedData["Handle Chunked Data <br> (May involve clearing buffer per chunk or specific chunk processing)"];
    CheckChunked -- "No (Not Chunked)" --> AccumulateData{"Append evt->data to output_buffer <br> (malloc/realloc output_buffer)"};
    AccumulateData -- "Success" --> DataAppended["output_buffer contains new data"];
    AccumulateData -- "Allocation Failed" --> AllocError["Log \Memory Allocation Failed\ <br> return ESP_FAIL"];
    
    ChunkedData --> EndEventProcessing;
    DataAppended --> EndEventProcessing;
    AllocError --> EndEventProcessing;
    Error --> EndEventProcessing;
    Connected --> EndEventProcessing;
    HeaderSent --> EndEventProcessing;
    OnHeader --> EndEventProcessing;
    OnFinish --> EndEventProcessing;
    Disconnected --> EndEventProcessing;
    Redirect --> EndEventProcessing;

    EndEventProcessing["Return ESP_OK"] --> End["Event Processed"];

    classDef startEnd fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef process fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef decision fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef check fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;
    classDef success fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;

    class Start startEnd;
    class End startEnd;
    class EventSwitch decision;
    class OnData decision;
    class CheckChunked decision;
    class Error process;
    class Connected process;
    class HeaderSent process;
    class OnHeader process;
    class AccumulateData process;
    class DataAppended process;
    class AllocError process;
    class ChunkedData process;
    class OnFinish process;
    class Disconnected process;
    class Redirect process;
    class EndEventProcessing success;

esp_err_t _http_event_handler(esp_http_client_event_t *evt) {
    static char *output_buffer;  // Buffer to store response of http request from event handler
    static int output_len;       // Stores number of bytes read
    switch(evt->event_id) {
        case HTTP_EVENT_ERROR:
            ESP_LOGD(TAG, "HTTP_EVENT_ERROR");
            break;
        case HTTP_EVENT_ON_CONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_CONNECTED");
            break;
        case HTTP_EVENT_HEADER_SENT:
            ESP_LOGD(TAG, "HTTP_EVENT_HEADER_SENT");
            break;
        case HTTP_EVENT_ON_HEADER:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_HEADER, key=%s, value=%s", evt->header_key, evt->header_value);
            break;
        case HTTP_EVENT_ON_DATA:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
            // Clean_up data buffer if we are expecting more data
            if (output_buffer != NULL && esp_http_client_is_chunked_response(evt->client)) {
                free(output_buffer);
                output_buffer = NULL;
                output_len = 0;
            }

            if (!esp_http_client_is_chunked_response(evt->client)) {
                // If user_data buffer is configured, copy the response data into it
                if (evt->user_data) {
                    memcpy(evt->user_data + output_len, evt->data, evt->data_len);
                } else {
                    // Allocate buffer to store response data
                    if (output_buffer == NULL) {
                        // We are not appending to an existing buffer
                        output_buffer = (char *) malloc(evt->data_len + 1); // +1 for null terminator
                        if (output_buffer == NULL) {
                            ESP_LOGE(TAG, "Failed to allocate memory for output buffer");
                            return ESP_FAIL;
                        }
                        output_len = 0;
                        memcpy(output_buffer + output_len, evt->data, evt->data_len);
                    } else {
                        // We are appending to an existing buffer
                        char *new_output_buffer = realloc(output_buffer, output_len + evt->data_len + 1);
                        if (new_output_buffer == NULL) {
                            ESP_LOGE(TAG, "Failed to reallocate memory for output buffer");
                            free(output_buffer); // Free original buffer
                            output_buffer = NULL;
                            output_len = 0;
                            return ESP_FAIL;
                        }
                        output_buffer = new_output_buffer;
                        memcpy(output_buffer + output_len, evt->data, evt->data_len);
                    }
                }
                output_len += evt->data_len;
                output_buffer[output_len] = '\0'; // Null-terminate
            } else {
                 ESP_LOGD(TAG, "Chunked response, data not processed in this simple handler.");
                 // For chunked responses, you would typically process each chunk as it arrives
                 // or accumulate them carefully, managing memory dynamically.
            }
            break;
        case HTTP_EVENT_ON_FINISH:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_FINISH");
            if (output_buffer != NULL) {
                // Response is accumulated in output_buffer. Process it here.
                ESP_LOGI(TAG, "Received data: %s", output_buffer);
                // Example: Parse JSON if expected
                // cJSON *root = cJSON_Parse(output_buffer);
                // if (root) { ... cJSON_Delete(root); }
                free(output_buffer);
                output_buffer = NULL;
                output_len = 0;
            }
            break;
        case HTTP_EVENT_DISCONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_DISCONNECTED");
            // Release resources allocated in the event handler, if any were not freed on FINISH
            // This is also a good place to signal task completion if waiting on an event group
            if (output_buffer != NULL) {
                free(output_buffer);
                output_buffer = NULL;
                output_len = 0;
            }
            break;
        case HTTP_EVENT_REDIRECT:
            ESP_LOGD(TAG, "HTTP_EVENT_REDIRECT");
            esp_http_client_set_redirection(evt->client); // Follow redirection
            break;
    }
    return ESP_OK;
}

Tip: The output_buffer in the event handler needs careful memory management, especially for large responses or chunked encoding. For production code, consider using a more robust buffering strategy, potentially passing a pre-allocated buffer via user_data or using a stream-based JSON parser if memory is very constrained. The example above shows a basic accumulation strategy.

HTTPS Considerations:

For HTTPS, server certificate verification is crucial. ESP-IDF’s esp_tls_crypto_bundle_attach() function (or setting cert_pem or crt_bundle_attach in the config) allows you to use a bundle of common CA certificates for this.

// In your configuration:
esp_http_client_config_t config = {
    .url = "https://api.example.com/data",
    .event_handler = _http_event_handler,
    .crt_bundle_attach = esp_crt_bundle_attach, // For ESP-IDF embedded bundle
    // or .cert_pem = "-----BEGIN CERTIFICATE-----\n...", // For a specific PEM certificate
    .transport_type = HTTP_TRANSPORT_OVER_SSL, // Important for HTTPS
};

Refer to Chapter 94 and 95 for more details on TLS/SSL and certificate management.

Practical Examples

Before running these examples, ensure you have:

1. An ESP32 development board.
2. ESP-IDF v5.x installed and configured with VS Code.
3. A working Wi-Fi connection setup in your project. The examples below will assume a wifi_init_sta() function (from previous chapters, Volume 2) is called to connect to Wi-Fi. You’ll need to add the Wi-Fi connection logic to your app_main.
4. The esp_http_client and esp_tls components enabled in your sdkconfig (usually enabled by default if you include the headers). You might need to enable CONFIG_ESP_TLS_ENABLE_GLOBAL_CA_STORE or CONFIG_ESP_TLS_CERTIFICATE_BUNDLE for the esp_crt_bundle_attach to work.

Project Structure (Simplified):

my_rest_client_project/
├── main/
│   ├── CMakeLists.txt
│   ├── main.c
│   └── Kconfig.projbuild (if any custom config)
├── CMakeLists.txt
└── sdkconfig

main/CMakeLists.txt:

idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS ".")

Common main.c includes and Wi-Fi setup (adapt as needed):

#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"

#include "lwip/err.h"
#include "lwip/sys.h"

// For HTTP Client
#include "esp_http_client.h"
#include "esp_crt_bundle.h" // For HTTPS CA bundle

// For JSON parsing (if needed)
#include "cJSON.h"

static const char *TAG = "REST_CLIENT_EXAMPLE";

// (Include your Wi-Fi connection logic here - e.g., wifi_init_sta())
// This often involves an event group to signal connection success.
// For brevity, this setup is omitted but is CRUCIAL.
// Assume wifi_init_sta() connects and waits for IP.

// Example Wi-Fi connection (simplified, refer to Volume 2)
#define WIFI_SSID      "YOUR_WIFI_SSID"
#define WIFI_PASS      "YOUR_WIFI_PASSWORD"
#define WIFI_MAXIMUM_RETRY 5

static EventGroupHandle_t s_wifi_event_group;
#define WIFI_CONNECTED_BIT BIT0
#define WIFI_FAIL_BIT      BIT1
static int s_retry_num = 0;

static void event_handler(void* arg, esp_event_base_t event_base,
                                int32_t event_id, void* event_data) {
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        if (s_retry_num < WIFI_MAXIMUM_RETRY) {
            esp_wifi_connect();
            s_retry_num++;
            ESP_LOGI(TAG, "retry to connect to the AP");
        } else {
            xEventGroupSetBits(s_wifi_event_group, WIFI_FAIL_BIT);
        }
        ESP_LOGI(TAG,"connect to the AP fail");
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
        ESP_LOGI(TAG, "got ip:" IPSTR, IP2STR(&event->ip_info.ip));
        s_retry_num = 0;
        xEventGroupSetBits(s_wifi_event_group, WIFI_CONNECTED_BIT);
    }
}

void wifi_init_sta(void) {
    s_wifi_event_group = xEventGroupCreate();

    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta();

    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));

    esp_event_handler_instance_t instance_any_id;
    esp_event_handler_instance_t instance_got_ip;
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT,
                                                        ESP_EVENT_ANY_ID,
                                                        &event_handler,
                                                        NULL,
                                                        &instance_any_id));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT,
                                                        IP_EVENT_STA_GOT_IP,
                                                        &event_handler,
                                                        NULL,
                                                        &instance_got_ip));

    wifi_config_t wifi_config = {
        .sta = {
            .ssid = WIFI_SSID,
            .password = WIFI_PASS,
            .threshold.authmode = WIFI_AUTH_WPA2_PSK, // Adjust as per your network
        },
    };
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA) );
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config) );
    ESP_ERROR_CHECK(esp_wifi_start() );

    ESP_LOGI(TAG, "wifi_init_sta finished.");

    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group,
            WIFI_CONNECTED_BIT | WIFI_FAIL_BIT,
            pdFALSE,
            pdFALSE,
            portMAX_DELAY);

    if (bits & WIFI_CONNECTED_BIT) {
        ESP_LOGI(TAG, "connected to ap SSID:%s", WIFI_SSID);
    } else if (bits & WIFI_FAIL_BIT) {
        ESP_LOGI(TAG, "Failed to connect to SSID:%s", WIFI_SSID);
    } else {
        ESP_LOGE(TAG, "UNEXPECTED EVENT");
    }
}
// End of Wi-Fi setup example

Example 1: Simple HTTP GET Request

This example fetches data from http://httpbin.org/get, a service that echoes information about the incoming request.

main.c (relevant part for HTTP GET):

// (Include common headers and Wi-Fi setup from above)

// Event handler for HTTP GET
esp_err_t _http_get_event_handler(esp_http_client_event_t *evt) {
    static char *output_buffer; 
    static int output_len;
    switch(evt->event_id) {
        case HTTP_EVENT_ERROR:
            ESP_LOGD(TAG, "HTTP_EVENT_ERROR");
            break;
        case HTTP_EVENT_ON_CONNECTED:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_CONNECTED");
            break;
        case HTTP_EVENT_HEADER_SENT:
            ESP_LOGD(TAG, "HTTP_EVENT_HEADER_SENT");
            break;
        case HTTP_EVENT_ON_HEADER:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_HEADER, key=%s, value=%s", evt->header_key, evt->header_value);
            break;
        case HTTP_EVENT_ON_DATA:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
            // If it's the first data chunk, allocate buffer
            if (output_buffer == NULL) {
                output_buffer = (char *) malloc(evt->data_len + 1);
                if (output_buffer == NULL) {
                    ESP_LOGE(TAG, "Failed to allocate memory for output buffer");
                    return ESP_FAIL;
                }
                output_len = 0;
                memcpy(output_buffer + output_len, evt->data, evt->data_len);
            } else { // Append to existing buffer
                char *new_buffer = realloc(output_buffer, output_len + evt->data_len + 1);
                if (new_buffer == NULL) {
                    ESP_LOGE(TAG, "Failed to reallocate memory for output buffer");
                    free(output_buffer);
                    output_buffer = NULL;
                    output_len = 0;
                    return ESP_FAIL;
                }
                output_buffer = new_buffer;
                memcpy(output_buffer + output_len, evt->data, evt->data_len);
            }
            output_len += evt->data_len;
            output_buffer[output_len] = '\0'; // Null-terminate
            break;
        case HTTP_EVENT_ON_FINISH:
            ESP_LOGD(TAG, "HTTP_EVENT_ON_FINISH");
            if (output_buffer != NULL) {
                ESP_LOGI(TAG, "Received HTTP GET Response:\n%s", output_buffer);
                // You can parse JSON here if the response is JSON
                // cJSON *root = cJSON_Parse(output_buffer);
                // if (root) {
                //    ESP_LOGI(TAG, "Successfully parsed JSON.");
                //    // ... process JSON data ...
                //    cJSON_Delete(root);
                // } else {
                //    ESP_LOGE(TAG, "Failed to parse JSON.");
                // }
                free(output_buffer);
                output_buffer = NULL;
                output_len = 0;
            }
            break;
        case HTTP_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "HTTP_EVENT_DISCONNECTED");
            // In case of an error or premature disconnect, free the buffer
            if (output_buffer != NULL) {
                free(output_buffer);
                output_buffer = NULL;
                output_len = 0;
            }
            break;
        case HTTP_EVENT_REDIRECT:
             ESP_LOGD(TAG, "HTTP_EVENT_REDIRECT");
            esp_http_client_set_redirection(evt->client);
            break;
    }
    return ESP_OK;
}

void http_get_task(void *pvParameters) {
    esp_http_client_config_t config = {
        .url = "http://httpbin.org/get",
        .event_handler = _http_get_event_handler,
        // .user_data = local_response_buffer, // Optional: Pass a buffer to the event handler
        .timeout_ms = 10000, // 10 seconds timeout
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);
    
    // Add a custom header (optional)
    esp_http_client_set_header(client, "X-Custom-Header", "ESP32-Client");

    esp_err_t err = esp_http_client_perform(client);

    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTP GET Status = %d, content_length = %lld",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
    } else {
        ESP_LOGE(TAG, "HTTP GET request failed: %s", esp_err_to_name(err));
    }

    esp_http_client_cleanup(client);
    vTaskDelete(NULL);
}

void app_main(void) {
    // Initialize NVS
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    ESP_LOGI(TAG, "ESP_WIFI_MODE_STA");
    wifi_init_sta(); // Ensure this connects to your Wi-Fi

    // Wait for Wi-Fi connection before starting HTTP task
    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdTRUE, portMAX_DELAY);
    if (bits & WIFI_CONNECTED_BIT) {
        ESP_LOGI(TAG, "WiFi Connected. Starting HTTP GET task.");
        xTaskCreate(&http_get_task, "http_get_task", 8192, NULL, 5, NULL);
    } else {
        ESP_LOGE(TAG, "WiFi connection failed. Cannot start HTTP task.");
    }
}

Build Instructions:

1. Save the code in main/main.c.
2. Configure your Wi-Fi SSID and Password in main.c.
3. Open a terminal in VS Code.
4. Build the project: idf.py build

Run/Flash/Observe Steps:

1. Flash the firmware: idf.py -p /dev/ttyUSB0 flash (replace /dev/ttyUSB0 with your ESP32’s serial port).
2. Monitor the output: idf.py -p /dev/ttyUSB0 monitor.
3. You should see logs indicating Wi-Fi connection, then the HTTP GET request being made, and finally the response from httpbin.org printed in the console. The response will be a JSON object detailing the headers and arguments of your request.

Example 2: HTTPS GET Request

This example fetches data from https://jsonplaceholder.typicode.com/todos/1, a public API for testing, using HTTPS.

main.c (relevant part for HTTPS GET, assumes Wi-Fi and NVS init from Example 1):

// (Include common headers and Wi-Fi setup from above)

// Event handler can be the same as _http_get_event_handler or a new one
// For this example, let's reuse _http_get_event_handler

void https_get_task(void *pvParameters) {
    esp_http_client_config_t config = {
        .url = "https://jsonplaceholder.typicode.com/todos/1",
        .event_handler = _http_get_event_handler, // Reuse the same handler
        .crt_bundle_attach = esp_crt_bundle_attach, // Attach ESP-IDF's CA cert bundle
        .timeout_ms = 15000, // Slightly longer timeout for HTTPS
        .transport_type = HTTP_TRANSPORT_OVER_SSL, // Specify SSL transport
        // .user_data = local_response_buffer, // Optional
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);

    esp_err_t err = esp_http_client_perform(client);

    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTPS GET Status = %d, content_length = %lld",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
    } else {
        ESP_LOGE(TAG, "HTTPS GET request failed: %s", esp_err_to_name(err));
        // You might see ESP_ERR_ESP_TLS_CERT_FLAGS_INVALID if cert bundle is not set up
    }

    esp_http_client_cleanup(client);
    vTaskDelete(NULL);
}

void app_main(void) {
    // (NVS and Wi-Fi init as in Example 1)
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    ESP_LOGI(TAG, "ESP_WIFI_MODE_STA");
    wifi_init_sta();

    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdTRUE, portMAX_DELAY);
    if (bits & WIFI_CONNECTED_BIT) {
        ESP_LOGI(TAG, "WiFi Connected. Starting HTTPS GET task.");
        // xTaskCreate(&http_get_task, "http_get_task", 8192, NULL, 5, NULL); // From Example 1
        xTaskCreate(&https_get_task, "https_get_task", 8192, NULL, 5, NULL); // Run this example
    } else {
        ESP_LOGE(TAG, "WiFi connection failed. Cannot start HTTP task.");
    }
}

Build, Flash, Observe:

1. Ensure CONFIG_ESP_TLS_ENABLE_GLOBAL_CA_STORE=y or CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y and CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y is set in your sdkconfig (via idf.py menuconfig -> Component config -> mbedTLS -> Certificate Bundle). This makes esp_crt_bundle_attach work.
2. Follow the same build, flash, and monitor steps as Example 1.
3. You should see the ESP32 connect via HTTPS and retrieve a JSON object representing a “todo” item.

Warning: HTTPS operations are more resource-intensive (CPU and memory) than HTTP due to encryption and certificate handling.

Example 3: HTTP POST Request with JSON

This example sends a JSON payload to http://httpbin.org/post.

main.c (relevant part for HTTP POST):

// (Include common headers and Wi-Fi setup from above)

// Event handler can be the same as _http_get_event_handler or a new one
// The response from httpbin.org/post will echo our POST data.

void http_post_task(void *pvParameters) {
    const char *post_data = "{\"sensor\":\"temperature\", \"value\":23.5, \"unit\":\"C\"}";

    esp_http_client_config_t config = {
        .url = "http://httpbin.org/post",
        .event_handler = _http_get_event_handler, // Reuse handler to see echoed response
        .method = HTTP_METHOD_POST,
        .timeout_ms = 10000,
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);

    // Set Content-Type header to application/json
    esp_http_client_set_header(client, "Content-Type", "application/json");
    // Set the POST data
    esp_http_client_set_post_field(client, post_data, strlen(post_data));

    esp_err_t err = esp_http_client_perform(client);

    if (err == ESP_OK) {
        ESP_LOGI(TAG, "HTTP POST Status = %d, content_length = %lld",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
        // The response will be logged by the _http_get_event_handler
    } else {
        ESP_LOGE(TAG, "HTTP POST request failed: %s", esp_err_to_name(err));
    }

    esp_http_client_cleanup(client);
    vTaskDelete(NULL);
}

void app_main(void) {
    // (NVS and Wi-Fi init as in Example 1)
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    ESP_LOGI(TAG, "ESP_WIFI_MODE_STA");
    wifi_init_sta();

    EventBits_t bits = xEventGroupWaitBits(s_wifi_event_group, WIFI_CONNECTED_BIT, pdFALSE, pdTRUE, portMAX_DELAY);
    if (bits & WIFI_CONNECTED_BIT) {
        ESP_LOGI(TAG, "WiFi Connected. Starting HTTP POST task.");
        // xTaskCreate(&http_get_task, "http_get_task", 8192, NULL, 5, NULL);
        // xTaskCreate(&https_get_task, "https_get_task", 8192, NULL, 5, NULL);
        xTaskCreate(&http_post_task, "http_post_task", 8192, NULL, 5, NULL); // Run this example
    } else {
        ESP_LOGE(TAG, "WiFi connection failed. Cannot start HTTP task.");
    }
}

Build, Flash, Observe:

1. Follow the same build, flash, and monitor steps.
2. The output will show the request being made, and the response from httpbin.org/post will include a json field containing the data you sent, along with other details like headers.

Variant Notes

While the esp_http_client API is largely consistent across ESP32 variants, there are some practical considerations:

• ESP32: The original ESP32 has dual cores and sufficient RAM for most HTTP/HTTPS client applications. Hardware acceleration for cryptographic operations benefits HTTPS.
• ESP32-S2: Single-core, but with dedicated crypto hardware. Generally good for HTTPS, but memory management is more critical than on the dual-core ESP32.
• ESP32-S3: Dual-core, similar to ESP32 but with AI acceleration and more PSRAM options. Excellent for complex applications involving HTTPS and JSON parsing. Has hardware crypto acceleration.
• ESP32-C3: Single RISC-V core. More memory-constrained than ESP32/S2/S3. HTTPS is feasible but requires careful optimization of buffer sizes and potentially using stream parsers for large JSON responses. It has hardware crypto support.
• ESP32-C6: Single RISC-V core with Wi-Fi 6 and 802.15.4 (Thread/Zigbee) capabilities. Performance for Wi-Fi tasks and crypto should be good. Memory constraints are similar to ESP32-C3.
• ESP32-H2: Primarily designed for 802.15.4 (Thread/Zigbee) and Bluetooth LE. While it does have Wi-Fi capabilities, it’s often used in conjunction with a gateway for cloud connectivity. If used directly for Wi-Fi REST calls, memory and processing power are more limited compared to ESP32/S3. It has hardware crypto support.

General Considerations for All Variants:

• Memory (RAM): HTTPS connections, especially with large certificate chains and big receive/transmit buffers, consume significant RAM. The output_buffer in the event handler, if accumulating large responses, can easily lead to out-of-memory errors on constrained devices. Consider:
  • Processing data in chunks (HTTP_EVENT_ON_DATA) instead of buffering the entire response.
  • Using JSON stream parsers (e.g., cJSON_Utils or third-party libraries) for large payloads.
  • Reducing CONFIG_ESP_HTTP_CLIENT_ENABLE_HTTPS_TRANSPORT_BUFFER_SIZE if memory is tight, but this might impact performance.
• Flash: Storing CA certificate bundles consumes flash space. The full bundle can be large. If you only connect to a few specific servers, consider embedding only their necessary CA certificates.
• Processing Power: TLS handshake and data encryption/decryption for HTTPS require CPU cycles. Hardware cryptographic accelerators on most ESP32 variants significantly help, but performance can still be a factor, especially on single-core variants during intensive operations.
• Task Stack Size: Tasks performing HTTP/HTTPS operations, especially HTTPS, may require larger stack sizes (e.g., 6144 or 8192 bytes) due to the underlying libraries (mbedTLS, lwIP). Monitor stack high water mark during development.

Common Mistakes & Troubleshooting Tips

Exercises

1. Fetch Public IP Address:
   • Modify the HTTPS GET example to make a request to https://api.ipify.org?format=json.
   • Parse the JSON response (e.g., {"ip":"YOUR_IP_ADDRESS"}) using cJSON.
   • Log your public IP address to the console.
2. Control an LED via a Mock API (GET):
   • Set up a simple mock API endpoint using a service like Beeceptor, Mocky.io, or even a simple Python Flask server on your local network.
   • This endpoint should return a simple JSON like {"led_status": "on"} or {"led_status": "off"}.
   • Write an ESP32 application that periodically polls this endpoint using HTTP GET.
   • Parse the response and control an onboard LED (or a GPIO-connected LED) based on the led_status.
3. Post Sensor Data to a Test API (POST):
   • If your ESP32 has a sensor (e.g., internal temperature sensor, or an external one like DHT11/BMP280), read a value from it.
   • Construct a JSON payload (e.g., {"sensor_id": "esp32_temp", "value": 25.5}).
   • POST this JSON data to https://httpbin.org/post (using HTTPS).
   • Log the server’s response, which should echo your sent data.
4. Implement PUT and DELETE Requests:
   • Use a public test API that supports PUT and DELETE methods (e.g., https://jsonplaceholder.typicode.com/posts/1).
   • PUT: Send a PUT request to https://jsonplaceholder.typicode.com/posts/1 with a JSON body to “update” the post. The API will simulate the update and return the “updated” data.{ "id": 1, "title": "My Updated Title", "body": "This is my updated post body.", "userId": 1 }
   • DELETE: Send a DELETE request to https://jsonplaceholder.typicode.com/posts/1. The API will simulate the deletion and typically return an empty body with a 200 OK status.
   • Log the status codes and responses for both operations.

Summary

• REST (Representational State Transfer) is an architectural style for web services, emphasizing statelessness, client-server separation, and a uniform interface.
• RESTful APIs use standard HTTP methods (GET, POST, PUT, DELETE, etc.) to operate on resources identified by URIs.
• JSON is the most common data format for exchanging resource representations in REST APIs.
• The ESP-IDF esp_http_client component provides a comprehensive API for making HTTP and HTTPS requests.
• Configuration is done via esp_http_client_config_t, specifying URL, method, event handler, and security parameters (for HTTPS).
• The event handler is crucial for processing asynchronous events like data reception and connection status.
• For HTTPS, server certificate verification is vital. Use esp_crt_bundle_attach or provide a specific cert_pem.
• POST/PUT requests require setting the Content-Type header (often application/json) and providing the payload using esp_http_client_set_post_field.
• ESP32 variant differences mainly relate to memory, processing power, and hardware crypto capabilities, impacting HTTPS performance and buffer management.
• Common issues include Wi-Fi connectivity problems, certificate errors, malformed requests, unhandled HTTP error codes, and buffer management for responses.

Further Reading

• ESP-IDF HTTP Client Documentation:
  • ESP-IDF HTTP Client API Reference
• ESP-IDF Certificate Bundle Documentation:
  • ESP-IDF Certificate Bundle
• cJSON Library (for JSON parsing in ESP-IDF):
  • cJSON is included in ESP-IDF. You can find its source and examples within the ESP-IDF components directory.
  • cJSON GitHub Repository (DaveGamble/cJSON) for general documentation.
• Understanding REST APIs:
  • What is REST? – REST API Tutorial (restapitutorial.com)
  • Understanding RESTful APIs (Red Hat)
• HTTP Status Codes:
  • MDN Web Docs: HTTP response status codes
• Public Test APIs:
  • httpbin.org – For testing HTTP requests.
  • JSONPlaceholder – Fake Online REST API for Testing and Prototyping.
  • Public APIs GitHub Repository – A collective list of free APIs.