Leave a Comment / By /

Chapter 100: Custom Network Protocol Development

Chapter Objectives

After completing this chapter, you will be able to:

  • Understand the rationale and trade-offs for developing custom network protocols.
  • Identify key design considerations for creating an application-specific protocol.
  • Choose appropriate transport layers (TCP or UDP) based on protocol requirements.
  • Design message framing and data serialization techniques for custom protocols.
  • Implement basic custom protocols on the ESP32 using sockets.
  • Develop both client and server roles for custom protocols on the ESP32.
  • Consider error handling, security, and extensibility in protocol design.
  • Apply these concepts to create efficient and tailored network communication for ESP32 applications.

Introduction

Throughout this volume, we’ve explored a multitude of standard networking protocols like HTTP, MQTT, WebSockets, and SNTP. These protocols are powerful, well-defined, and suitable for a wide array of applications. However, there are scenarios where the overhead of standard protocols might be undesirable, or their features might not perfectly align with the specific needs of a highly optimized or unique embedded application. In such cases, developing a custom network protocol can offer significant advantages in terms of efficiency, simplicity, or tailored functionality.

This chapter will guide you through the process of designing and implementing your own custom network protocols on the ESP32. We’ll cover the fundamental principles, from choosing the right transport layer (TCP or UDP) to defining message structures, serializing data, and handling communication logic. While creating a custom protocol requires careful thought and planning, it can unlock performance and resource utilization benefits crucial for constrained embedded systems. This chapter will empower you to build lean, efficient, and perfectly tailored communication solutions for your ESP32 projects.

Theory

Why Custom Protocols?

While standard protocols offer interoperability and rich feature sets, custom protocols can be advantageous in specific situations:

graph TD
    A["Start: Need Network Communication?"] -->|"Yes"| B{"Standard Protocol Adequate? <br>(HTTP, MQTT, WebSocket, etc.)"};
    A -->|"No"| Z["No Network Protocol Needed"];

    B -->|"Yes"| C["Use Standard Protocol"];
    B -->|"No"| D{"Specific Reasons for Custom?"};

    D -->|"No"| C;
    D -->|"Yes"| E{"Efficiency Critical? <br> (Low Bandwidth/Power, Min Overhead)"};
    E -->|"Yes"| F["Consider Custom Protocol"];
    E -->|"No"| G{"Simplicity/Tailored Functionality Needed?"};
    
    G -->|"Yes"| F;
    G -->|"No"| H{"Unique Communication Pattern?"};

    H -->|"Yes"| F;
    H -->|"No"| I{"Proprietary/Closed System?"};

    I -->|"Yes"| F;
    I -->|"No"| J{"Extreme Resource Constraints?"};

    J -->|"Yes"| F;
    J -->|"No"| K["Re-evaluate Standard Protocols with Customizations/Profiles"];
    K --> C;

    F --> L{"Trade-offs Acceptable? <br> (Interoperability, Dev Effort, Security Risks)"};
    L -->|"Yes"| M["Develop Custom Protocol"];
    L -->|"No"| K;
    
    C --> X["End: Solution Found"];
    M --> X;
    Z --> X;

    classDef primary fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef success fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;
    classDef decision fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef process fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef check fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;

    class A,M primary;
    class B,D,E,G,H,I,J,L decision;
    class F,K process;
    class C,X,Z success;
  1. Efficiency:
    • Reduced Overhead: Standard protocols often include headers and features not needed by every application. A custom protocol can strip these down to the bare essentials, minimizing packet size and processing requirements. This is particularly important for battery-powered devices or networks with limited bandwidth.
    • Optimized Data Representation: Custom binary formats can be more compact than text-based formats like JSON or XML used in some standard protocols.
  2. Simplicity:
    • Tailored Functionality: You implement only the features you need, leading to simpler client and server logic.
    • Easier Debugging (for simple protocols): A well-defined, minimal custom protocol can sometimes be easier to debug than complex interactions within a standard protocol stack.
  3. Specific Requirements:
    • Unique Communication Patterns: Your application might have communication patterns not well-suited to existing request-response or publish-subscribe models.
    • Proprietary Systems: For closed systems where interoperability with external standard clients/servers is not a goal.
    • Extreme Resource Constraints: On very low-power or memory-constrained devices, even a lightweight standard protocol might be too heavy.
Aspect Advantages of Custom Protocols Disadvantages/Trade-offs
Efficiency
  • Reduced Overhead: Minimal headers and features tailored to specific needs, saving bandwidth and processing.
  • Optimized Data Representation: Custom binary formats can be more compact than text-based ones (e.g., JSON, XML).
 Can be less efficient if poorly designed (e.g., inefficient serialization, chatty interactions).
Simplicity
  • Tailored Functionality: Implement only necessary features, leading to simpler client/server logic.
  • Potentially Easier Debugging: For very simple, well-defined protocols.
 Complexity can grow quickly if requirements expand; debugging can be hard without standard tools.
Specific Requirements
  • Unique Communication Patterns: Supports patterns not well-suited to standard models (e.g., custom streaming).
  • Proprietary Systems: Useful for closed ecosystems where external interoperability isn’t a goal.
  • Extreme Resource Constraints: Can be vital for devices with very limited memory, power, or processing.
 Limited to the specific application; not suitable for general-purpose communication.
Interoperability Not a primary goal; designed for a specific system. Lack of Standard Interoperability: Cannot be understood by standard tools, browsers, or other off-the-shelf clients/servers.
Development Effort Can be quicker for very simple needs.
  • Full Responsibility: You define, implement, and debug all aspects (framing, serialization, error handling, etc.).
  • Reinventing the Wheel: Risk of re-implementing features already robustly provided by standard protocols (reliability, flow control, security).
Security Can be designed with specific security needs in mind from the ground up (though this is hard).
  • Challenging to Secure: Designing secure custom protocols is difficult and error-prone.
  • Lacks Vetted Mechanisms: Standard protocols often have well-tested security (e.g., TLS for TCP-based protocols).
Maintenance & Evolution Can be straightforward if the protocol is simple and well-documented. Can become difficult to maintain and extend if not designed with versioning and extensibility in mind.

Trade-offs:

However, developing custom protocols comes with trade-offs:

  • Lack of Interoperability: Your custom protocol won’t be understood by standard tools or clients.
  • Development Effort: You are responsible for defining, implementing, and debugging all aspects of the protocol.
  • Reinventing the Wheel: You might end up re-implementing features already robustly provided by standard protocols (e.g., reliability, security).
  • Security Risks: Designing secure custom protocols is challenging. Standard protocols often have well-vetted security mechanisms (like TLS).

Therefore, the decision to create a custom protocol should be made after carefully evaluating whether existing standard protocols can meet the requirements.

Design Considerations

Designing a custom network protocol involves several key decisions:

  • Transport Layer Choice (TCP vs. UDP):
    • UDP (User Datagram Protocol):
      • Connectionless, message-oriented.
      • Low overhead, fast.
      • Unreliable: Packets can be lost, duplicated, or arrive out of order. No built-in congestion control.
      • Suitable for: Real-time data where occasional loss is acceptable (e.g., streaming sensor data for non-critical display), simple request-response where the application can handle retries, service discovery via broadcast/multicast.
    • TCP (Transmission Control Protocol):
      • Connection-oriented, stream-based.
      • Reliable: Guarantees ordered delivery of data without loss or duplication (through acknowledgments and retransmissions).
      • Flow and congestion control.
      • Higher overhead than UDP due to connection setup and state management.
      • Suitable for: Applications requiring reliable data transfer (e.g., command and control, firmware updates, critical data logging).
Feature UDP (User Datagram Protocol) TCP (Transmission Control Protocol)
Connection Type Connectionless Connection-oriented
Data Delivery Message-oriented (datagrams) Stream-oriented
Reliability Unreliable: Packets can be lost, duplicated, or arrive out of order. No built-in retransmission. Reliable: Guarantees ordered delivery without loss or duplication via acknowledgments and retransmissions.
Overhead Low (smaller header, no connection setup/teardown state) Higher (larger header, connection setup handshake, state management)
Speed Generally faster due to lower overhead and no connection management. Can be slower due to reliability mechanisms and connection management.
Flow Control No built-in flow control. Application must handle or tolerate. Built-in flow control to prevent sender from overwhelming receiver.
Congestion Control No built-in congestion control. Can contribute to network congestion if not managed by application. Built-in congestion control to adapt to network conditions.
Message Boundaries Preserves message boundaries (one send() typically maps to one recv() for the datagram). Does not preserve message boundaries (data is a continuous stream). Application needs message framing.
Use Cases for Custom Protocols
  • Real-time data (e.g., streaming non-critical sensor readings).
  • Simple request-response where application handles retries.
  • Service discovery (broadcast/multicast).
  • Applications where speed and low overhead are paramount and occasional loss is acceptable.
  • Applications requiring reliable data transfer (e.g., commands, firmware updates, critical logs).
  • Situations where data integrity and order are crucial.
  • When complexity of implementing reliability at application layer is to be avoided.
ESP32 Implementation Notes Simpler socket setup. Need to handle potential packet loss or out-of-order arrival in application logic if reliability is needed. More complex socket setup (listen, accept for servers). Requires message framing logic. LwIP handles reliability.
  • Message Framing (especially for TCP):
    • TCP provides a reliable byte stream, not distinct messages. The receiver needs a way to determine where one message ends and the next begins.
    • Common techniques:
      • Delimiter-based: A special character or sequence of characters (e.g., newline \n, CRLF \r\n) marks the end of a message. Simple but requires scanning the data and escaping delimiters if they appear in the payload.
      • Length-prefixing: Each message is preceded by a fixed-size field indicating the length of the upcoming message payload. The receiver reads the length, then reads that many bytes for the payload. More robust than delimiters.
      • Fixed-size messages: All messages have a predefined, constant size. Simplest but least flexible.
    • UDP, being message-oriented, inherently frames data at the packet level. A single send() call on a UDP socket typically corresponds to a single recv() on the receiver, yielding one datagram. However, you still need to define the structure within that datagram.
graph TD
    subgraph TCP Stream Input
        direction LR
        S["Continuous Byte Stream from TCP Socket"]
    end

    S --> P1["Framing Logic Needed"];

    subgraph FramingTechniques["Framing Techniques (Applied by Receiver)"]
        direction TB

        subgraph DelimiterBased["Delimiter-Based"]
            direction LR
            DB1["Read Chunks"] --> DB2{"Delimiter Found? <br> (e.g., \<i>\\n\</i>)"};
            DB2 -- "No" --> DB1;
            DB2 -- "Yes" --> DB3["Extract Message"];
        end

        subgraph LengthPrefixed["Length-Prefixing"]
            direction LR
            LP1["Read Fixed-Size Length Field (L)"] --> LP2["Read L Bytes for Payload"];
            LP2 --> LP3["Extract Message"];
        end

        subgraph FixedSize["Fixed-Size Messages"]
            direction LR
            FS1["Read Fixed N Bytes"] --> FS2["Extract Message"];
        end
    end
    
    P1 --> DelimiterBased;
    P1 --> LengthPrefixed;
    P1 --> FixedSize;

    DB3 --> EM["Processed Message"];
    LP3 --> EM;
    FS2 --> EM;

    classDef primary fill:#EDE9FE,stroke:#5B21B6,stroke-width:2px,color:#5B21B6;
    classDef success fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#065F46;
    classDef decision fill:#FEF3C7,stroke:#D97706,stroke-width:1px,color:#92400E;
    classDef process fill:#DBEAFE,stroke:#2563EB,stroke-width:1px,color:#1E40AF;
    classDef check fill:#FEE2E2,stroke:#DC2626,stroke-width:1px,color:#991B1B;

    class S,P1,EM primary;
    class DB2 decision;
    class DB1,DB3,LP1,LP2,LP3,FS1,FS2 process;
    class FramingTechniques,DelimiterBased,LengthPrefixed,FixedSize process;
    
    style DelimiterBased fill:#f9f9f9,stroke:#333,stroke-width:1px
    style LengthPrefixed fill:#f9f9f9,stroke:#333,stroke-width:1px
    style FixedSize fill:#f9f9f9,stroke:#333,stroke-width:1px
    style FramingTechniques fill:#efefef,stroke:#333,stroke-width:2px
    style S fill:#FFF3E0,stroke:#FFB74D,stroke-width:2px,color:#E65100
    style EM fill:#E8F5E9,stroke:#66BB6A,stroke-width:2px,color:#1B5E20
  • Data Serialization:
    • How data is converted into a byte stream for transmission and parsed back at the receiver.
    • Text-based:
      • Examples: JSON, XML, CSV, custom key-value pairs.
      • Pros: Human-readable, easy to debug, widely supported by parsing libraries.
      • Cons: Verbose (high overhead), slower to parse compared to binary.
    • Binary (Custom):
      • Define a fixed structure (like a C struct) or a sequence of well-defined fields (e.g., first byte is command, next two bytes are length, then payload).
      • Pros: Compact, efficient to parse, minimal overhead.
      • Cons: Not human-readable, requires careful handling of byte order (endianness) if communicating between different architectures, less flexible if the structure changes.
    • Binary (Standardized):
      • Examples: Protocol Buffers (Protobuf), MessagePack, CBOR.
      • Pros: Offer a balance of efficiency, type safety, and schema evolution capabilities. Often more compact than JSON but with more structure than raw custom binary.
      • Cons: May require specific libraries and a schema definition step. For this chapter, we’ll focus on simpler custom binary/text.
Serialization Type Description & Examples Pros Cons ESP32 Considerations
Text-based (Custom) Data represented as human-readable strings.
Examples:
– Key-value pairs: temp=25.3;humid=45.1
– CSV: 25.3,45.1,1012.5
– Custom commands: LED_ON
  • Human-readable, easy to debug manually.
  • Simple to implement for basic needs.
  • Verbose (high overhead).
  • Slower to parse than binary.
  • Requires string manipulation functions (e.g., sprintf, sscanf, strtok).
  • Error-prone parsing if format is not strict.
 Good for simple command/response. Parsing can consume CPU and memory. Standard C string functions are available.
Text-based (Standardized) Using standard text formats.
Examples:
– JSON: {"temp": 25.3, "humid": 45.1}
– XML: <data><temp>25.3</temp>...</data>
  • Widely supported with many parsing libraries.
  • Structured and often self-describing (JSON).
  • Very verbose, especially XML.
  • Parsing libraries can be large (flash/RAM impact).
  • Slower parsing compared to binary.
 ESP-IDF has JSON parsing libraries (e.g., cJSON, esp-json). Consider resource impact for constrained devices. XML is generally too heavy.
Binary (Custom) Data represented directly in binary form, often mapping to C structs or a defined sequence of typed fields.
Example: [cmd_id (1B)] [len (2B)] [payload (var)]
  • Compact, minimal overhead.
  • Efficient to parse/serialize (direct memory copy or field extraction).
  • Fast.
  • Not human-readable without tools.
  • Requires careful handling of byte order (endianness) between different systems.
  • Less flexible if structure changes (versioning needed).
  • Prone to errors if packing/unpacking is incorrect.
 Excellent for performance-critical and low-bandwidth applications. Use __attribute__((packed)) for structs. Be mindful of endianness if communicating with non-ESP32 devices.
Binary (Standardized) Using standard binary serialization formats.
Examples:
– Protocol Buffers (Protobuf)
– MessagePack
– CBOR (Concise Binary Object Representation)
  • Efficient (often more compact than JSON, less than custom binary).
  • Type safety and schema evolution support.
  • Cross-language support.
  • May require specific libraries/tools.
  • Schema definition step usually needed.
  • Can add some complexity compared to raw custom binary.
 Libraries for these might be available for ESP-IDF or can be ported. Good balance for structured, efficient data if custom binary is too rigid or text is too slow/large. Check library resource usage.
  • Command Structure / Message Types:
    • Define different types of messages your protocol will support (e.g., GET_SENSOR_DATA, SET_RELAY_STATE, SENSOR_DATA_RESPONSE, ACK, ERROR).
    • Assign unique identifiers (opcodes) to each message type, often as the first byte(s) of a message.
    • Specify the parameters or payload associated with each message type.
  • Error Handling and Reporting:
    • How will the protocol report errors (e.g., invalid command, failed operation, bad parameters)?
    • Define specific error message types or error codes.
    • Consider if acknowledgments (ACK/NACK) are needed for certain operations.
  • Security Considerations:
    • Custom protocols are often unencrypted by default. If confidentiality or integrity is required:
      • Consider layering your custom protocol over TLS (if using TCP). This requires managing certificates and the TLS handshake.
      • Implement application-level encryption/authentication (more complex and error-prone).
    • Think about authentication: How does the server know the client is legitimate, and vice-versa?
    • Be wary of denial-of-service vulnerabilities (e.g., a client sending huge messages).
Security Goal Consideration / Threat Potential Mitigation Strategies for Custom Protocols ESP32 Notes
Confidentiality
(Prevent Eavesdropping)		 Data transmitted in plaintext can be intercepted and read by unauthorized parties.
  • Layer over TLS (for TCP): Use TLS to encrypt the entire communication channel. This is the most robust approach.
  • Application-Level Encryption: Encrypt the payload of your custom messages using symmetric (e.g., AES) or asymmetric cryptography. Requires key management.
 ESP32 has hardware acceleration for AES. mbedTLS library can be used for TLS or standalone crypto operations. Managing keys securely on the device is critical.
Integrity
(Prevent Data Tampering)		 Data could be modified in transit without detection.
  • TLS: Provides integrity through MACs (Message Authentication Codes).
  • Application-Level MACs/Signatures: Include a MAC (e.g., HMAC-SHA256) or digital signature with each message.
  • Checksums (Basic): Simple checksums (e.g., CRC) can detect accidental corruption but not malicious tampering.
 Hardware SHA acceleration available. mbedTLS provides MAC functions.
Authentication
(Verify Identity)
  • Client Authentication: How does the server know the client is legitimate?
  • Server Authentication: How does the client know it’s talking to the real server?
  • TLS with Certificates: Mutual authentication using X.509 certificates.
  • Pre-Shared Keys (PSK): Both parties have a secret key used for authentication/encryption.
  • API Keys/Tokens: Client includes a secret token in messages.
  • Challenge-Response Mechanisms.
 TLS client/server certificate handling is supported. Secure storage of PSKs or tokens (e.g., in NVS with encryption) is important.
Availability
(Prevent Denial of Service – DoS)		 Malicious actors might try to overwhelm the device or exploit protocol weaknesses to make it unresponsive.
  • Input Validation: Strictly validate all incoming data (lengths, types, ranges). Reject malformed or oversized messages early.
  • Rate Limiting: Limit the number of requests a client can make in a given time.
  • Resource Management: Carefully manage memory and other resources to prevent exhaustion.
  • Connection Limits: For servers, limit concurrent connections.
 Implement robust parsing. Use FreeRTOS features to manage tasks and resources. Be cautious with dynamic memory allocation in network handlers.
Replay Attacks An attacker re-sends valid, previously captured messages to cause unintended actions.
  • Timestamps: Include a timestamp in messages and reject old messages. Requires synchronized time.
  • Nonces/Sequence Numbers: Include a unique, non-repeating number in each message. Server tracks expected/received nonces.
 SNTP can be used for time synchronization. Managing nonces requires state on the server.
Lack of Scrutiny Custom protocols don’t benefit from the widespread review and testing that standard protocols undergo.
  • Thorough Design & Review: Carefully design the protocol with security in mind. Conduct internal security reviews.
  • Keep it Simple: Complexity is the enemy of security.
  • Consider standard mechanisms (like TLS) where possible.
 Focus on well-understood cryptographic primitives if implementing application-level security.
  • Scalability and Extensibility:
    • How will the protocol evolve if new features or message types are needed?
    • Consider versioning for your protocol from the start.
    • Design message formats that can accommodate future additions without breaking existing implementations (e.g., by using Type-Length-Value (TLV) encoding for parameters or leaving reserved fields).

Implementing on ESP32

  • Sockets API: You’ll use the LwIP sockets API (Chapter 85-92) for sending and receiving data over TCP or UDP.
  • Task Management (FreeRTOS):
    • For a server handling multiple clients, each client connection might be managed in a separate FreeRTOS task, or a single task might use select() to handle multiple sockets.
    • Client logic might run in its own task or as part of a larger application task.
  • Data Buffering: Manage send and receive buffers carefully.
  • Parsing and Serialization Logic: Implement functions to construct outgoing messages and parse incoming messages according to your protocol definition.

Practical Examples

These examples assume you have Wi-Fi connectivity established on your ESP32 as covered in previous chapters.

Example 1: Simple UDP-based Sensor Data Protocol

Protocol Definition:

  • Purpose: ESP32 client sends simulated sensor data to a server.
  • Transport: UDP.
  • Message Format (Binary):
    • Device ID (1 byte): Identifier for the ESP32 device.
    • Sensor Type (1 byte): e.g., 0x01 for Temperature, 0x02 for Humidity.
    • Sensor Value (4 bytes): Floating-point value.
    • Timestamp (4 bytes): Seconds since epoch (optional, can be added by server too).Total message size: 1 + 1 + 4 + 4 = 10 bytes.
Field Size (Bytes) Data Type Description Example Value
Device ID 1 uint8_t Identifier for the ESP32 device. 0xA1
Sensor Type 1 uint8_t Type of sensor reading. 0x01 (Temp), 0x02 (Humid)
Sensor Value 4 float The actual sensor reading. 25.75
Timestamp 4 uint32_t Seconds since epoch (Unix timestamp). 1678886400
Total Size 10 N/A Total size of one packet. N/A

1. ESP32 Client (main.c):

C
#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"
#include "esp_netif.h"
#include "lwip/err.h"
#include "lwip/sockets.h"
#include "lwip/sys.h"
#include <lwip/netdb.h>
#include <time.h> // For time_t

#define WIFI_SSID      "YOUR_WIFI_SSID"
#define WIFI_PASS      "YOUR_WIFI_PASSWORD"
#define SERVER_IP      "SERVER_IP_ADDRESS" // Replace with your UDP server's IP
#define SERVER_PORT    12345

static const char *TAG = "UDP_CUSTOM_PROTO";
static SemaphoreHandle_t wifi_connected_sem;

// Our custom packet structure
typedef struct {
    uint8_t device_id;
    uint8_t sensor_type;
    float sensor_value;
    uint32_t timestamp; // Seconds since epoch
} __attribute__((packed)) sensor_packet_t; // packed to ensure no padding

static void udp_client_task(void *pvParameters) {
    char host_ip[] = SERVER_IP;
    int addr_family = AF_INET;
    int ip_protocol = IPPROTO_IP;
    struct sockaddr_in dest_addr;

    dest_addr.sin_addr.s_addr = inet_addr(host_ip);
    dest_addr.sin_family = AF_INET;
    dest_addr.sin_port = htons(SERVER_PORT);

    int sock = socket(addr_family, SOCK_DGRAM, ip_protocol);
    if (sock < 0) {
        ESP_LOGE(TAG, "Unable to create socket: errno %d (%s)", errno, strerror(errno));
        vTaskDelete(NULL);
        return;
    }
    ESP_LOGI(TAG, "Socket created, sending data to %s:%d", host_ip, SERVER_PORT);

    sensor_packet_t packet;
    packet.device_id = 0xA1; // Example device ID

    while (1) {
        // Simulate temperature reading
        packet.sensor_type = 0x01; // Temperature
        packet.sensor_value = 20.0f + (float)(esp_random() % 100) / 10.0f; // Random temp 20.0 - 29.9
        packet.timestamp = (uint32_t)time(NULL); // Get current time as timestamp

        ESP_LOGI(TAG, "Sending Temp: %.2f, TS: %u", packet.sensor_value, packet.timestamp);
        int err = sendto(sock, &packet, sizeof(sensor_packet_t), 0, (struct sockaddr *)&dest_addr, sizeof(dest_addr));
        if (err < 0) {
            ESP_LOGE(TAG, "Error occurred during sending: errno %d (%s)", errno, strerror(errno));
        }
        vTaskDelay(pdMS_TO_TICKS(5000)); // Send every 5 seconds

        // Simulate humidity reading
        packet.sensor_type = 0x02; // Humidity
        packet.sensor_value = 40.0f + (float)(esp_random() % 300) / 10.0f; // Random humidity 40.0 - 69.9
        packet.timestamp = (uint32_t)time(NULL);

        ESP_LOGI(TAG, "Sending Humid: %.2f, TS: %u", packet.sensor_value, packet.timestamp);
        err = sendto(sock, &packet, sizeof(sensor_packet_t), 0, (struct sockaddr *)&dest_addr, sizeof(dest_addr));
        if (err < 0) {
            ESP_LOGE(TAG, "Error occurred during sending: errno %d (%s)", errno, strerror(errno));
        }
        vTaskDelay(pdMS_TO_TICKS(5000));
    }

    ESP_LOGE(TAG, "Shutting down socket and exiting task...");
    shutdown(sock, 0);
    close(sock);
    vTaskDelete(NULL);
}

// Wi-Fi Event Handler & Init (reuse from previous chapters, e.g., Chapter 99)
static void wifi_event_handler(void* arg, esp_event_base_t event_base, int32_t event_id, void* event_data) {
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        xSemaphoreGive(wifi_connected_sem);
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        esp_wifi_connect();
    }
}

void wifi_init_sta(void) {
    wifi_connected_sem = xSemaphoreCreateBinary();
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_sta();
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &wifi_event_handler, NULL, NULL));
    ESP_ERROR_CHECK(esp_event_handler_instance_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &wifi_event_handler, NULL, NULL));
    wifi_config_t wifi_config = { .sta = { .ssid = WIFI_SSID, .password = WIFI_PASS }};
    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA) );
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config) );
    ESP_ERROR_CHECK(esp_wifi_start() );
}

void app_main(void) {
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);

    wifi_init_sta();
    if(xSemaphoreTake(wifi_connected_sem, portMAX_DELAY) == pdTRUE) {
        ESP_LOGI(TAG, "Wi-Fi Connected. Starting UDP client task.");
        // For timestamp to be meaningful, synchronize time first (see Chapter 98)
        // For this example, we'll use un-synchronized time(NULL)
        xTaskCreate(udp_client_task, "udp_client", 4096, NULL, 5, NULL);
    }
}

CMakeLists.txt (main component):

Plaintext
idf_component_register(SRCS "main.c"
                    INCLUDE_DIRS "."
                    REQUIRES esp_wifi esp_event esp_log nvs_flash esp_netif lwip)

2. Simple Python UDP Server (for testing):

Save as udp_server.py on your computer:

Python
import socket
import struct
import datetime

SERVER_IP = "0.0.0.0"  # Listen on all available interfaces
SERVER_PORT = 12345
BUFFER_SIZE = 1024

# Define the same structure format string for unpacking
# B = unsigned char (1 byte), f = float (4 bytes), I = unsigned int (4 bytes)
# Use '<' for little-endian, as ESP32 is little-endian
PACKET_FORMAT = "<BBfI" # DeviceID, SensorType, Value, Timestamp

def main():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind((SERVER_IP, SERVER_PORT))
    print(f"UDP server listening on {SERVER_IP}:{SERVER_PORT}")

    try:
        while True:
            data, addr = sock.recvfrom(BUFFER_SIZE)
            print(f"\nReceived {len(data)} bytes from {addr}")

            if len(data) == struct.calcsize(PACKET_FORMAT):
                try:
                    device_id, sensor_type, sensor_value, timestamp = struct.unpack(PACKET_FORMAT, data)
                    sensor_type_str = "Temperature" if sensor_type == 0x01 else "Humidity" if sensor_type == 0x02 else "Unknown"
                    
                    # Convert Unix timestamp to human-readable time
                    dt_object = datetime.datetime.fromtimestamp(timestamp)
                    
                    print(f"  Device ID  : 0x{device_id:02X}")
                    print(f"  Sensor Type: {sensor_type_str} (0x{sensor_type:02X})")
                    print(f"  Value      : {sensor_value:.2f}")
                    print(f"  Timestamp  : {timestamp} ({dt_object.strftime('%Y-%m-%d %H:%M:%S')})")
                except struct.error as e:
                    print(f"  Error unpacking data: {e}")
                    print(f"  Raw data (hex): {data.hex()}")
            else:
                print(f"  Received packet of unexpected size: {len(data)} bytes")
                print(f"  Raw data (hex): {data.hex()}")

    except KeyboardInterrupt:
        print("\nServer shutting down.")
    finally:
        sock.close()

if __name__ == "__main__":
    main()


sequenceDiagram
    actor Client as ESP32 Client
    participant Server as UDP Server (Python)

    Client->>Client: "Prepare Sensor Packet <br> (DeviceID, Type, Value, Timestamp)"
   
    Client->>Server: "Send UDP Packet (Sensor Data)"
    note right of Client: "Example: DeviceID=0xA1, <br>Type=0x01 (Temp), <br>Value=25.5, TS=..."
   

    Server->>Server: "Receive UDP Packet"
    Server->>Server: "Unpack/Parse Sensor Data <br> (struct.unpack)"
    
    Server->>Server: "Log/Process Data"
    note left of Server: "Server prints decoded data <br>to console."

3. Build, Flash, and Observe:

  1. Replace Wi-Fi credentials in main.c.
  2. Replace SERVER_IP_ADDRESS in main.c with the IP address of the computer where you’ll run udp_server.py.
  3. Run python udp_server.py on your computer.
  4. Build and flash the ESP32 project.
  5. Observe the ESP32 logs and the Python server output. The server should print the decoded sensor data.

Tip: The __attribute__((packed)) in the C struct definition is important to prevent the C compiler from adding padding bytes, ensuring the struct’s memory layout matches the intended binary protocol. Always verify endianness if communicating between different architectures. ESP32 is little-endian.

Example 2: TCP-based Command/Response Protocol (ESP32 as Server)

Protocol Definition:

  • Purpose: Client sends commands to ESP32 server, server responds.
  • Transport: TCP.
  • Message Framing: Newline (\n) terminated text messages.
  • Commands from Client:
    • GET_STATUS\n
    • LED_ON\n
    • LED_OFF\n
  • Responses from Server (ESP32):
    • STATUS:OK,TEMP:25.3,LED:ON\n
    • ACK:LED_ON\n
    • ACK:LED_OFF\n
    • ERR:UNKNOWN_CMD\n
    • ERR:CMD_FAILED\n

1. ESP32 Server (main.c):

C
#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event.h"
#include "esp_log.h"
#include "nvs_flash.h"
#include "esp_netif.h"
#include "lwip/err.h"
#include "lwip/sockets.h"
#include "lwip/sys.h"
#include <lwip/netdb.h>

#define WIFI_SSID      "YOUR_WIFI_SSID_AP" // Or connect ESP32 to existing network
#define WIFI_PASS      "YOUR_WIFI_PASSWORD_AP"
#define SERVER_PORT    8888
#define MAX_CLIENTS    1 // For simplicity, handle one client at a time in this example
#define RCV_BUF_SIZE   64

static const char *TAG = "TCP_CUSTOM_PROTO_SERVER";
static bool led_state = false; // false = OFF, true = ON

// Simulate getting temperature
float get_temperature_mock() {
    return 20.0f + (float)(esp_random() % 150) / 10.0f; // 20.0 - 34.9
}

static void handle_tcp_client(const int sock) {
    char rx_buffer[RCV_BUF_SIZE];
    char tx_buffer[RCV_BUF_SIZE];
    int len;

    ESP_LOGI(TAG, "TCP client %d connected.", sock);

    do {
        len = recv(sock, rx_buffer, sizeof(rx_buffer) - 1, 0);
        if (len < 0) {
            ESP_LOGE(TAG, "recv failed: errno %d (%s)", errno, strerror(errno));
            break;
        } else if (len == 0) {
            ESP_LOGI(TAG, "Connection closed by client %d", sock);
            break;
        } else {
            rx_buffer[len] = 0; // Null-terminate whatever we received
            // Remove trailing newline if present for easier string comparison
            if (rx_buffer[len - 1] == '\n') {
                rx_buffer[len - 1] = 0;
                if (len > 1 && rx_buffer[len - 2] == '\r') { // Handle CRLF
                    rx_buffer[len - 2] = 0;
                }
            }
            ESP_LOGI(TAG, "Received %d bytes from client %d: '%s'", len, sock, rx_buffer);

            // Process command
            if (strcmp(rx_buffer, "GET_STATUS") == 0) {
                float temp = get_temperature_mock();
                snprintf(tx_buffer, sizeof(tx_buffer), "STATUS:OK,TEMP:%.1f,LED:%s\n", temp, led_state ? "ON" : "OFF");
            } else if (strcmp(rx_buffer, "LED_ON") == 0) {
                led_state = true;
                // TODO: Add actual GPIO control for an LED here
                ESP_LOGI(TAG, "Turning LED ON");
                snprintf(tx_buffer, sizeof(tx_buffer), "ACK:LED_ON\n");
            } else if (strcmp(rx_buffer, "LED_OFF") == 0) {
                led_state = false;
                // TODO: Add actual GPIO control for an LED here
                ESP_LOGI(TAG, "Turning LED OFF");
                snprintf(tx_buffer, sizeof(tx_buffer), "ACK:LED_OFF\n");
            } else {
                snprintf(tx_buffer, sizeof(tx_buffer), "ERR:UNKNOWN_CMD\n");
            }

            int to_write = strlen(tx_buffer);
            int written = send(sock, tx_buffer, to_write, 0);
            if (written < 0) {
                ESP_LOGE(TAG, "Error occurred during sending: errno %d (%s)", errno, strerror(errno));
                break; // Exit loop on send error
            } else if (written < to_write) {
                ESP_LOGW(TAG, "Sent partial data: %d of %d bytes", written, to_write);
            }
             ESP_LOGI(TAG, "Sent: %s", tx_buffer);
        }
    } while (len > 0);

    ESP_LOGI(TAG, "Shutting down client socket %d", sock);
    shutdown(sock, 0);
    close(sock);
}

static void tcp_server_task(void *pvParameters) {
    char addr_str[128];
    int addr_family = AF_INET;
    int ip_protocol = IPPROTO_IP;
    struct sockaddr_in dest_addr;

    dest_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    dest_addr.sin_family = AF_INET;
    dest_addr.sin_port = htons(SERVER_PORT);

    int listen_sock = socket(addr_family, SOCK_STREAM, ip_protocol);
    if (listen_sock < 0) {
        ESP_LOGE(TAG, "Unable to create socket: errno %d (%s)", errno, strerror(errno));
        vTaskDelete(NULL);
        return;
    }
    ESP_LOGI(TAG, "Listen socket created");

    int err = bind(listen_sock, (struct sockaddr *)&dest_addr, sizeof(dest_addr));
    if (err != 0) {
        ESP_LOGE(TAG, "Socket unable to bind: errno %d (%s)", errno, strerror(errno));
        goto CLEAN_UP;
    }
    ESP_LOGI(TAG, "Socket bound, port %d", SERVER_PORT);

    err = listen(listen_sock, 1); // Listen with a backlog of 1
    if (err != 0) {
        ESP_LOGE(TAG, "Error occurred during listen: errno %d (%s)", errno, strerror(errno));
        goto CLEAN_UP;
    }
    ESP_LOGI(TAG, "Socket listening...");

    while (1) {
        struct sockaddr_in source_addr; // Large enough for both IPv4 or IPv6
        socklen_t addr_len = sizeof(source_addr);
        int sock = accept(listen_sock, (struct sockaddr *)&source_addr, &addr_len);
        if (sock < 0) {
            ESP_LOGE(TAG, "Unable to accept connection: errno %d (%s)", errno, strerror(errno));
            // Optionally add a small delay here if accept fails continuously
            vTaskDelay(pdMS_TO_TICKS(100)); 
            continue; // Continue listening
        }

        // Convert client IP address to string
        if (source_addr.sin_family == PF_INET) {
            inet_ntoa_r(((struct sockaddr_in *)&source_addr)->sin_addr.s_addr, addr_str, sizeof(addr_str) - 1);
        }
        ESP_LOGI(TAG, "Socket accepted connection from: %s, socket: %d", addr_str, sock);

        handle_tcp_client(sock); // Handle one client at a time
        // For multiple concurrent clients, you would typically spawn a new task here
        // or use a non-blocking approach with select().
    }

CLEAN_UP:
    close(listen_sock);
    vTaskDelete(NULL);
}

// Wi-Fi Event Handler & Init (reuse, ensure ESP32 can get an IP or is in AP mode)
// For server, AP mode is often convenient for direct client connection
static void wifi_event_handler_ap(void* arg, esp_event_base_t event_base, int32_t event_id, void* event_data) {
    if (event_id == WIFI_EVENT_AP_STACONNECTED) {
        wifi_event_ap_staconnected_t* event = (wifi_event_ap_staconnected_t*) event_data;
        ESP_LOGI(TAG, "Station "MACSTR" joined, AID=%d", MAC2STR(event->mac), event->aid);
    } else if (event_id == WIFI_EVENT_AP_STADISCONNECTED) {
        wifi_event_ap_stadisconnected_t* event = (wifi_event_ap_stadisconnected_t*) event_data;
        ESP_LOGI(TAG, "Station "MACSTR" left, AID=%d", MAC2STR(event->mac), event->aid);
    }
}

void wifi_init_softap(void) {
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    esp_netif_create_default_wifi_ap(); // Create AP interface

    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));

    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &wifi_event_handler_ap, NULL, NULL));

    wifi_config_t wifi_config = {
        .ap = {
            .ssid = WIFI_SSID,
            .ssid_len = strlen(WIFI_SSID),
            .password = WIFI_PASS,
            .max_connection = 4,
            .authmode = WIFI_AUTH_WPA_WPA2_PSK
        },
    };
    if (strlen(WIFI_PASS) == 0) {
        wifi_config.ap.authmode = WIFI_AUTH_OPEN;
    }

    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_AP));
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_AP, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start());

    ESP_LOGI(TAG, "wifi_init_softap finished. SSID:%s password:%s", WIFI_SSID, WIFI_PASS);
    esp_netif_ip_info_t ip_info;
    esp_netif_get_ip_info(esp_netif_get_handle_from_ifkey("WIFI_AP_DEF"), &ip_info);
    ESP_LOGI(TAG, "AP IP Address: " IPSTR, IP2STR(&ip_info.ip));
}


void app_main(void) {
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
      ESP_ERROR_CHECK(nvs_flash_erase());
      ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK(ret);
    
    // Use SoftAP mode for the server for easier direct connection
    wifi_init_softap(); 
    // Or use wifi_init_sta() if connecting to an existing network
    // Ensure you log the ESP32's IP address if in STA mode.

    xTaskCreate(tcp_server_task, "tcp_server", 4096, NULL, 5, NULL);
}

CMakeLists.txt (main component): Same as UDP example.

2. Simple Python TCP Client (for testing):

Save as tcp_client.py:

Python
import socket
import time

# ESP32's IP address (if in STA mode) or 192.168.4.1 (if ESP32 is AP)
SERVER_IP = "192.168.4.1" 
SERVER_PORT = 8888
BUFFER_SIZE = 1024

def send_command(sock, command):
    print(f"Sending: {command.strip()}")
    sock.sendall(command.encode('utf-8'))
    response = sock.recv(BUFFER_SIZE).decode('utf-8')
    print(f"Received: {response.strip()}")
    return response

def main():
    try:
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.connect((SERVER_IP, SERVER_PORT))
            print(f"Connected to ESP32 server at {SERVER_IP}:{SERVER_PORT}")

            send_command(s, "GET_STATUS\n")
            time.sleep(1)
            send_command(s, "LED_ON\n")
            time.sleep(1)
            send_command(s, "GET_STATUS\n")
            time.sleep(1)
            send_command(s, "LED_OFF\n")
            time.sleep(1)
            send_command(s, "GET_STATUS\n")
            time.sleep(1)
            send_command(s, "INVALID_CMD\n") # Test error handling
            time.sleep(1)

    except ConnectionRefusedError:
        print(f"Connection refused. Is the ESP32 server running at {SERVER_IP}:{SERVER_PORT}?")
    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    main()

3. Build, Flash, and Observe:

  1. Configure Wi-Fi in main.c. If using AP mode, set WIFI_SSID_AP and WIFI_PASS_AP.
  2. Update SERVER_IP in tcp_client.py to the ESP32’s IP address (default 192.168.4.1 if ESP32 is AP, or its assigned IP if in STA mode).
  3. Build and flash the ESP32 project.
  4. Run python tcp_client.py on your computer (after connecting to the ESP32’s AP if it’s in AP mode).
  5. Observe the client and server logs.

Variant Notes

  • ESP32, ESP32-S2, ESP32-S3: Well-suited for custom protocols, both client and server roles. Dual-core variants (ESP32, S3) can better handle multiple TCP clients if server logic becomes complex. Hardware crypto can be leveraged if adding TLS to a custom TCP protocol.
  • ESP32-C3, ESP32-C6: Can run custom protocols. Resource constraints (single core, RAM) might favor simpler, UDP-based, or binary protocols, especially for server applications or when WSS-like security is added. Efficiency of the custom protocol is key.
  • ESP32-H2:
    • If using IP over 6LoWPAN (via a Thread border router), custom IP-based protocols (UDP/TCP) are feasible but subject to the characteristics of 6LoWPAN (e.g., smaller MTU, potentially higher latency). Protocol design should be mindful of packet sizes.
    • More commonly for ESP32-H2, custom protocols might be designed directly over its native 802.15.4 MAC layer or Bluetooth LE GATT custom services/characteristics if IP is not used. This involves different APIs and design principles (not covered by LwIP sockets) and would focus on very low-power, short-range communication. The principles of message definition and data serialization still apply.

General Considerations for all variants:

  • Memory Usage: Complex parsing logic (e.g., for text-based protocols) or large buffers can consume significant RAM. Binary protocols are generally more memory-efficient.
  • Processing Power: CPU load depends on parsing complexity and the rate of message handling.
  • Flash Usage: Libraries for serialization (e.g., JSON parsers, if used) add to firmware size.

Common Mistakes & Troubleshooting Tips

Mistake / Issue Symptom(s) Troubleshooting / Solution
Endianness Mismatches (Binary Protocols) Multi-byte numerical data (e.g., int, float, uint16_t) is corrupted or nonsensical when read by the receiving system. Values appear swapped or completely wrong. Fix:
  • Establish a Network Byte Order (typically Big Endian) and convert using htonl(), htons(), ntohl(), ntohs() on both client and server before sending and after receiving.
  • Alternatively, ensure both ends use the same endianness (ESP32 is little-endian) and document this. This is less portable if other systems join.
  • For C structs, __attribute__((packed)) helps with alignment but does not solve endianness between different architectures.
  • Use serialization formats (e.g., Protobuf, MessagePack) that handle endianness internally.
TCP Message Framing Issues
  • Receiver gets multiple messages concatenated as one.
  • Receiver gets partial messages or splits one message into multiple reads.
  • Parsing errors due to incorrect message boundaries.
 Fix: Implement a clear framing mechanism:
  • Length-prefixing: Prepend each message with its length (e.g., 2 or 4 bytes). Receiver reads length, then reads that many bytes for the payload.
  • Delimiter-based: End each message with a unique delimiter (e.g., \n, \r\n). Receiver buffers data until delimiter is found. Ensure delimiter cannot appear in payload or escape it.
  • Fixed-size messages: If all messages are the same known size. Simplest but least flexible.
Ensure the receiver logic correctly identifies message boundaries before attempting to parse.
Buffer Overflows Crashes, unexpected behavior, security vulnerabilities. Occurs when incoming data exceeds the allocated buffer size during recv() or string operations. Fix:
  • Always check received data length against buffer capacity.
  • If using length-prefixing, read the length first. If it’s too large for your buffer, reject the message or handle it appropriately (e.g., read in chunks if designed for it, but this is complex).
  • For delimiter-based, ensure your read loop doesn’t write past the end of the buffer while searching for the delimiter.
  • Use “safe” string functions if applicable (e.g., strncpy, snprintf) and always null-terminate strings correctly within buffer bounds.
Blocking Socket Operations Stalling Tasks A task performing blocking calls like accept(), recv(), connect() without timeouts becomes unresponsive if no data arrives or no connection is made. Other functions in the same task don’t run. Fix:
  • Use non-blocking sockets with select() or poll() to check socket readiness before calling blocking functions.
  • Set socket timeouts (e.g., SO_RCVTIMEO, SO_SNDTIMEO) so blocking calls eventually return.
  • For servers, consider handling each client connection in a separate FreeRTOS task to prevent one client from blocking others.
  • For clients, ensure connect() has a timeout or is handled in a way that doesn’t freeze critical operations.
Not Handling Partial Sends/Receives (TCP)
  • send() might not transmit all requested bytes in one call, especially for large data or congested networks.
  • recv() might not receive the entire message in one call, even if the sender sent it all at once.
This leads to incomplete messages or data corruption. 		Fix:
  • For send(): Loop until all bytes are sent, updating the pointer and remaining byte count. Check the return value of send().
  • For recv(): Loop until the expected number of bytes (determined by your framing mechanism) is received or an error/timeout occurs. Buffer incoming data.
Ignoring Network Errors / Incorrect Error Checking Socket functions (socket, bind, listen, connect, send, recv, etc.) return error codes (often -1) and set errno. Ignoring these leads to silent failures and difficult debugging. Fix:
  • Always check the return value of every socket function call.
  • If an error is indicated (e.g., -1), immediately check the value of errno.
  • Log the error using strerror(errno) to get a human-readable message. ESP_LOGE(TAG, “Socket error: %s”, strerror(errno)).
  • Handle errors gracefully (e.g., close socket, retry with backoff, notify user/system).
UDP Packet Loss/Reordering Not Handled Application assumes UDP packets will always arrive, or arrive in order. This leads to missing data or incorrect state if the application relies on sequence. Fix (if reliability is needed over UDP):
  • Implement application-level acknowledgments (ACKs).
  • Include sequence numbers in packets to detect loss or reorder.
  • Implement retransmission logic for unacknowledged packets.
  • Buffer packets at the receiver to reorder if necessary.
  • Alternatively, design the protocol to be tolerant of occasional loss if appropriate for the application (e.g., streaming non-critical sensor data).

Exercises

  1. UDP Protocol with ACK:
    • Modify the UDP sensor data protocol (Example 1).
    • The server (Python script) should send back a simple acknowledgment packet (e.g., [DeviceID_byte][ACK_Opcode_byte]) to the ESP32 after successfully receiving and parsing a sensor packet.
    • The ESP32 client should attempt to receive this ACK within a short timeout after sending data. Log if ACK is received or not. This introduces basic reliability.
  2. TCP Protocol Extensibility:
    • Extend the TCP command/response protocol (Example 2).
    • Add a new command SET_CONFIG:PARAM_NAME=VALUE\n (e.g., SET_CONFIG:INTERVAL=5000\n).
    • The ESP32 server should parse this, (conceptually) store the configuration, and respond with ACK:SET_CONFIG:PARAM_NAME\n.
    • Add a GET_CONFIG:PARAM_NAME\n command for the client to retrieve a stored configuration value.
  3. Length-Prefixed TCP Messaging:
    • Design and implement a simple TCP client/server where messages are framed using a 2-byte length prefix (network byte order) followed by a UTF-8 string payload.
    • The ESP32 can act as either client or server. The other end can be a Python script.
    • Client sends “Hello ESP32”. Server receives, logs, and sends back “Hello Client”.
  4. Simple Service Discovery (UDP Broadcast):
    • Design a protocol where an ESP32 client sends a UDP broadcast packet (e.g., containing “ESP32_DISCOVER_REQUEST”) on a specific port.
    • One or more ESP32 servers on the same network listen for this broadcast.
    • Upon receiving the request, each server responds directly to the client’s IP address (obtained from the received broadcast packet) with a UDP packet containing its device information (e.g., “ESP32_SERVER_RESPONSE,DeviceName,IPAddress”).
    • The client collects responses for a few seconds.

Summary

  • Developing custom network protocols can offer efficiency and tailored functionality when standard protocols are not a perfect fit, but involves trade-offs like reduced interoperability and increased development effort.
  • Key design considerations include choosing TCP vs. UDP, defining message framing (especially for TCP), selecting data serialization methods (text vs. binary), structuring commands/message types, and planning for error handling and security.
  • UDP is suitable for low-overhead, non-critical real-time data; TCP provides reliability for critical data transfer.
  • Binary protocols are generally more efficient but require careful handling of endianness and are less human-readable than text-based formats like JSON or delimited strings.
  • Implementation on ESP32 uses the LwIP sockets API and FreeRTOS for task management, especially for servers handling multiple clients.
  • Always check return codes from socket operations and handle errno to diagnose issues.
  • Resource constraints on different ESP32 variants may influence protocol design choices, favoring simplicity and efficiency on more limited devices.
  • Careful planning for extensibility and versioning can make custom protocols more maintainable in the long run.

Further Reading

This chapter marks the end of Volume 4, “Advanced Networking and Internet Protocols.” You have now journeyed from fundamental networking concepts to implementing various standard protocols and even designing your own. This knowledge forms a critical foundation for building sophisticated, connected embedded systems with the ESP32. The subsequent volumes will build upon this by exploring IoT-specific protocols, cloud connectivity, and interactions with a wider range of peripherals.

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top