Chapter 75: IPC: System V Message Queues (msgget, msgsnd, msgrcv, msgctl)

Chapter Objectives

Upon completing this chapter, you will be able to:

• Understand the architecture and principles of System V message queues as a form of Inter-Process Communication (IPC).
• Implement robust IPC mechanisms by creating, controlling, and using message queues with the msgget(), msgctl(), msgsnd(), and msgrcv() system calls.
• Design and build multi-process embedded applications on a Raspberry Pi 5 that exchange structured data reliably.
• Configure message queue permissions and manage their lifecycle within the Linux kernel.
• Debug common issues related to message queue identifiers, permissions, and blocking behavior.
• Apply message queues to solve practical problems, such as communication between a data acquisition process and a data logging process.

Introduction

In the world of embedded Linux, systems are rarely monolithic. They are often composed of multiple, independent processes working in concert to achieve a common goal. A sensor monitoring process might need to alert a logging process when a critical threshold is reached. A user interface process may need to send commands to a motor control process. This coordination requires a robust and reliable method for processes to communicate with each other, known as Inter-Process Communication (IPC). While simpler mechanisms like signals or pipes exist, they often fall short when applications demand more structured, persistent, and prioritized communication.

This is where System V message queues provide a powerful solution. As one of the three original System V IPC mechanisms (along with semaphores and shared memory), message queues offer a time-tested and kernel-persistent method for passing formatted data between processes. Unlike pipes, which transmit an unstructured stream of bytes, message queues handle discrete packets of data, or messages. Each message can have a specific type, allowing a receiving process to selectively fetch messages based on priority or category. Furthermore, message queues are not tied to the lifecycle of the creating process; they persist in the kernel until explicitly removed, allowing for asynchronous communication between processes that may not even run concurrently.

This chapter will guide you through the theory and practice of using System V message queues on your Raspberry Pi 5. We will explore the fundamental system calls that form the API for this mechanism, delving into how the kernel manages queues and messages. You will learn to create applications that can exchange complex data structures, manage queue permissions, and control the lifecycle of these powerful IPC resources. By the end of this chapter, you will have the skills to implement sophisticated multi-process architectures, a cornerstone of modern embedded Linux systems.

Technical Background

System V IPC, first introduced in UNIX System V Release 2, represents a foundational set of tools for creating complex multi-process applications. These mechanisms were designed to provide more structure and persistence than the pipes and signals that preceded them. Message queues, in particular, were conceived as a sort of electronic “mailbox” system managed by the operating system kernel, allowing any process with the correct permissions to send and receive messages.

The Message Queue Architecture

At its core, a System V message queue is a kernel-managed linked list of messages. It is not a file, a device, or a memory segment in the user’s address space. Instead, it is a distinct kernel object identified by a unique integer, the message queue identifier (often abbreviated as msqid). This separation from the file system is a key characteristic of System V IPC. To access a queue, a process does not open a file; it uses a special system call, msgget(), to obtain the msqid.

To ensure that unrelated processes can find and access the same queue, the msgget() call relies on a key, which is of the key_t data type (typically an integer). This key serves as a well-known name for the queue. Processes that wish to communicate agree on a common key. A special key value, IPC_PRIVATE, can be used to guarantee the creation of a new, unique queue, though this is less common in embedded systems where processes are designed to collaborate. Often, the ftok() function is used to generate a key from a file path and an integer, providing a convenient way to derive a consistent key across different program executions.

Each message queue in the kernel is associated with a msqid_ds data structure, which holds metadata about the queue. This structure is defined in <sys/msg.h> and contains important information, including the permissions for accessing the queue, the process IDs of the last sender and receiver, timestamps for various operations, and the current number of messages and bytes on the queue. The msgctl() system call allows a process to query this structure (to get status information) or to modify it (to change permissions or remove the queue).

Permissions and Ownership

Security is a fundamental aspect of the System V IPC model. Every message queue has an owner (user ID and group ID) and a set of permissions, much like a file. These permissions determine who can read from (receive) and write to (send) the queue. The permissions are specified as an octal number when the queue is created with msgget(). For example, a permission mode of 0666 grants read and write access to the owner, the group, and all other users. In an embedded context, where processes often run under the same user (e.g., root), permissions can be more relaxed, but it is a best practice to apply the principle of least privilege, granting only the access that is strictly necessary.

The ownership of a new queue is set to the effective user ID and group ID of the creating process. The msgctl() system call, using the IPC_SET command, allows the owner of the queue (or a privileged process) to change the ownership and permissions after creation.

The Message Structure

The data exchanged via message queues is not an arbitrary stream of bytes but is encapsulated in a specific structure. Every message must begin with a positive long integer field, referred to as the message type. The kernel uses this type to support advanced message retrieval. The rest of the message is the payload, a block of data of a specified size.

A common C declaration for a message buffer looks like this:

struct msg_buffer {
    long msg_type;
    char msg_text[128]; // The payload
};

The msg_type field is crucial. When a process sends a message using msgsnd(), it sets this value. When another process receives a message using msgrcv(), it can request a message of a specific type, the first message of any type, or the first message with a type less than or equal to a given value. This feature is incredibly powerful for implementing priority messaging. For instance, a process could use type 1 for high-priority alerts and type 2 for routine data, and the receiver could always check for type 1 messages before processing any type 2 messages.

The Core System Calls

The functionality of System V message queues is exposed through four primary system calls:

1. msgget(key_t key, int msgflg): This is the gateway to using a message queue. It takes a key and a set of flags (msgflg). If a queue with the given key already exists, msgget() returns its identifier. If it does not exist and the IPC_CREAT flag is specified, a new queue is created. If IPC_CREAT is used with IPC_EXCL, the call will fail if the queue already exists, preventing accidental reuse of an old queue. The flags also include the permission bits for the new queue. On success, it returns a valid msqid; on failure, it returns -1 and sets errno.
2. msgsnd(int msqid, const void *msgp, size_t msgsz, int msgflg): This call sends a message. It takes the msqid of the target queue, a pointer msgp to the user-defined message structure (which must start with a long msg_type), the size of the message payload msgsz (excluding the msg_type field), and flags. The most common flag is IPC_NOWAIT. If the queue is full (either because it contains the system-defined maximum number of messages or the maximum total number of bytes), a call to msgsnd() will normally block until space becomes available. If IPC_NOWAIT is set, the call will fail immediately with errno set to EAGAIN instead of blocking.
3. msgrcv(int msqid, void *msgp, size_t msgsz, long msgtyp, int msgflg): This call retrieves a message from a queue. It takes the msqid, a pointer msgp to a buffer to store the received message, the maximum size msgsz of the payload buffer, a msgtyp parameter to specify which message to retrieve, and flags. The msgtyp parameter is what makes message queues so flexible:
   • If msgtyp is 0, the first message on the queue is retrieved.
   • If msgtyp is a positive integer, the first message with a type equal to msgtyp is retrieved.
   • If msgtyp is a negative integer, the first message with a type less than or equal to the absolute value of msgtyp is retrieved.
   Like msgsnd(), msgrcv() will block by default if no matching message is on the queue. The IPC_NOWAIT flag causes it to return immediately with errno set to ENOMSG in this case.
4. msgctl(int msqid, int cmd, struct msqid_ds *buf): This is the control function for message queues. It performs various administrative tasks on the queue specified by msqid, depending on the cmd argument:
   • IPC_STAT: Copies the msqid_ds structure for the queue into the buffer pointed to by buf. This is used to query the queue’s status.
   • IPC_SET: Modifies the permissions and ownership of the queue based on the values in the buf structure provided by the user. Only the owner or a privileged process can do this.
   • IPC_RMID: Removes the message queue from the kernel. This is a critical cleanup step. Once removed, the queue and any messages on it are gone forever. All processes currently blocked in msgsnd() or msgrcv() on this queue will wake up and return with an error (EIDRM).

The persistence of message queues is both a feature and a responsibility. An embedded system that creates message queues must have a clean shutdown procedure that uses msgctl() with IPC_RMID to remove them. Otherwise, these queues will remain in the kernel, consuming resources, until the next system reboot. The ipcs and ipcrm command-line utilities are invaluable for developers to inspect and manually clean up IPC objects during development and debugging.

sequenceDiagram
    actor Server
    actor Client
    participant Kernel

    Server->>+Kernel: msgget(KEY, IPC_CREAT | 0666)
    Note right of Server: Create a new message queue
    Kernel-->>-Server: Returns msqid
    Server->>+Kernel: msgrcv(msqid, &buf, SIZE, 1, 0)
    Note right of Server: Blocks, waiting for message of type 1

    Client->>+Kernel: msgget(KEY, 0666)
    Note left of Client: Get ID of existing queue
    Kernel-->>-Client: Returns msqid

    Client->>+Kernel: msgsnd(msqid, &msg, SIZE, 0)
    Note left of Client: Send message with type 1
    Kernel-->>-Client: Returns 0 (success)

    Kernel-->>-Server: Unblocks with message data
    Note right of Server: Process message from client

    Server->>+Kernel: msgctl(msqid, IPC_RMID, NULL)
    Note right of Server: Destroy the message queue
    Kernel-->>-Server: Returns 0 (success)

Practical Examples

Theory provides the foundation, but true understanding comes from hands-on implementation. In this section, we will build a simple client-server system on the Raspberry Pi 5 using System V message queues. The “server” process will create a message queue and wait for messages. The “client” process will send a message containing a string to the server, which will then print the received message. This example demonstrates the complete lifecycle: creation, sending, receiving, and destruction.

System Setup

Ensure you have a standard Raspberry Pi OS installation on your Raspberry Pi 5. You will need the gcc compiler, which is included by default. All work will be done from the command line via SSH or a directly connected terminal.

File Structure

We will create two C files in a directory, for example, ~/msgq_example/:

msgq_example/
├── client.c
├── server.c
└── common.h

The common.h file will store the key and message structure definition, ensuring both client and server use the same values.

common.h

#ifndef COMMON_H
#define COMMON_H

#include <sys/types.h>

// Define a key that both the client and server can use to identify the message queue.
// 0x1234 is an arbitrary but unique integer for this example.
#define MSG_QUEUE_KEY 1234

// Define the maximum size of the message text.
#define MAX_MSG_SIZE 256

// Define the structure for our messages.
// It must start with a `long` type for the message type.
struct message_buffer {
    long message_type;
    char message_text[MAX_MSG_SIZE];
};

#endif // COMMON_H

Server Implementation (server.c)

The server is responsible for creating the message queue, waiting for a message, processing it, and finally, cleaning up the queue.

// server.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ipc.h>
#include <sys/msg.h>
#include <errno.h>
#include "common.h"

int main() {
    int msqid;
    struct message_buffer rcv_buffer;

    printf("Server: Starting up...\n");

    // 1. Create the message queue.
    // We use MSG_QUEUE_KEY as the unique identifier.
    // IPC_CREAT creates the queue if it doesn't exist.
    // 0666 gives read/write permissions to all users.
    msqid = msgget(MSG_QUEUE_KEY, IPC_CREAT | 0666);
    if (msqid == -1) {
        perror("msgget failed");
        exit(EXIT_FAILURE);
    }
    printf("Server: Message queue created with ID: %d\n", msqid);

    // 2. Wait to receive a message.
    // We are waiting for a message of type 1.
    // The call will block until a message of type 1 arrives.
    printf("Server: Waiting for a message...\n");
    if (msgrcv(msqid, &rcv_buffer, MAX_MSG_SIZE, 1, 0) == -1) {
        perror("msgrcv failed");
        exit(EXIT_FAILURE);
    }

    // 3. Process the received message.
    printf("Server: Message received.\n");
    printf("  - Type: %ld\n", rcv_buffer.message_type);
    printf("  - Text: \"%s\"\n", rcv_buffer.message_text);

    // 4. Destroy the message queue.
    // This is a critical cleanup step.
    printf("Server: Removing message queue...\n");
    if (msgctl(msqid, IPC_RMID, NULL) == -1) {
        perror("msgctl(IPC_RMID) failed");
        exit(EXIT_FAILURE);
    }

    printf("Server: Shutdown complete.\n");
    return 0;
}

Explanation of server.c:

1. msgget(): The server creates the message queue. By using IPC_CREAT, it will either create a new queue or get the ID of an existing one with the same key. The 0666 permissions make it widely accessible, which is convenient for this example.
2. msgrcv(): The server blocks here, waiting for a message. We specify 1 as the message_type, so it will only accept messages sent with that type. The 0 flag indicates a blocking receive.
3. Processing: Once a message is received, its contents are printed to the console.
4. msgctl(IPC_RMID): This is the crucial cleanup step. The server, being the owner of the resource, removes it from the kernel. If this were omitted, the queue would persist after the program terminates.

Client Implementation (client.c)

The client’s job is simpler: get the identifier for the existing queue and send a single message to it.

// client.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ipc.h>
#include <sys/msg.h>
#include <errno.h>
#include "common.h"

int main() {
    int msqid;
    struct message_buffer snd_buffer;

    printf("Client: Starting up...\n");

    // 1. Get the message queue ID.
    // Note: No IPC_CREAT flag. The client assumes the server has already created it.
    msqid = msgget(MSG_QUEUE_KEY, 0666);
    if (msqid == -1) {
        perror("msgget failed");
        exit(EXIT_FAILURE);
    }
    printf("Client: Connected to message queue with ID: %d\n", msqid);

    // 2. Prepare the message to send.
    snd_buffer.message_type = 1; // Must be a positive integer
    strcpy(snd_buffer.message_text, "Hello from the client process!");

    // 3. Send the message.
    // The size is the size of the text, not the whole struct.
    printf("Client: Sending message...\n");
    if (msgsnd(msqid, &snd_buffer, strlen(snd_buffer.message_text) + 1, 0) == -1) {
        perror("msgsnd failed");
        exit(EXIT_FAILURE);
    }

    printf("Client: Message sent successfully.\n");
    return 0;
}

Explanation of client.c:

1. msgget(): The client calls msgget() with the same key but without IPC_CREAT. This ensures it only gets the ID of an existing queue. If the server isn’t running and the queue doesn’t exist, this call will fail.
2. Message Preparation: A message buffer is populated. The message_type is set to 1 to match what the server is expecting. A string is copied into the payload.
3. msgsnd(): The message is sent. The size argument is crucial: it’s the size of the payload (message_text), not the entire struct message_buffer. We add + 1 to include the null terminator of the string. The 0 flag indicates a blocking send.

Build and Execution Steps

1. Compile the Code: Open a terminal on your Raspberry Pi 5 and navigate to the ~/msgq_example/ directory. Compile both programs.

gcc -o server server.c -Wall
gcc -o client client.c -Wall


The -Wall flag enables all compiler warnings, which is a good practice.

2. Run the Server: You will need two separate terminal windows to see the interaction. In the first terminal, start the server.

./server


Expected Server Output (Terminal 1):

Server: Starting up...
Server: Message queue created with ID: 0
Server: Waiting for a message...


The server is now running and blocked in the msgrcv() call. The queue ID might be different from 0 on your system.

3. Run the Client: In the second terminal, run the client.

./client


Expected Client Output (Terminal 2):

Client: Starting up...
Client: Connected to message queue with ID: 0
Client: Sending message...
Client: Message sent successfully.


The client sends its message and exits immediately.

5. Observe the Server’s Final Output: As soon as the client sends the message, the server’s msgrcv() call will unblock. Look back at the first terminal.

Expected Server Output (Terminal 1, after client runs):

Server: Message received.
  - Type: 1
  - Text: "Hello from the client process!"
Server: Removing message queue...
Server: Shutdown complete.


The server receives the message, prints its contents, removes the queue from the kernel, and then exits. The IPC resource has been cleanly managed.

Common Mistakes & Troubleshooting

System V message queues are powerful but can be unforgiving. Many common issues stem from misunderstanding their lifecycle, permissions, or the blocking nature of the API calls.

Exercises

These exercises will help you solidify your understanding of message queues by building more complex and practical applications.

1. Client-Server with Reply:
   • Objective: Modify the example so the client sends a message and then waits for a reply from the server.
   • Guidance:
     1. The client’s message should include its process ID (PID).
     2. The server, after receiving the message, should print it and then send a confirmation message back to the client.
     3. The server should use the client’s PID as the message_type for the reply message.
     4. The client, after sending its initial message, should call msgrcv(), waiting for a message whose type matches its own PID.
   • Verification: The client should print the confirmation message it receives from the server before exiting.
2. Prioritized Message Handling:
   • Objective: Create a system where messages can be sent with different priorities.
   • Guidance:
     1. Create one server process and two client processes (“high-priority client” and “low-priority client”).
     2. The low-priority client should send 5 messages with message_type = 10.
     3. The high-priority client should send 1 message with message_type = 1.
     4. Run the low-priority client first to fill the queue, then run the high-priority client.
     5. Modify the server to first try receiving a message of type 1 using msgrcv() with the IPC_NOWAIT flag. If it gets one, it processes it. If not (ENOMSG), it then makes a blocking call to msgrcv() to get any message (type 0).
   • Verification: The server should process the single high-priority message before any of the low-priority messages, even though the low-priority messages were sent first.
3. LED Control System:
   • Objective: Use a message queue to control a physical component on the Raspberry Pi 5.
   • Hardware: Connect an LED and a 220Ω resistor to GPIO 17 and a Ground pin.
   • Guidance:
     1. Write a “controller” process that creates a message queue and uses a library like libgpiod to control GPIO 17.
     2. The controller should wait for messages. A message with type 1 and text “ON” should turn the LED on. A message with type 1 and text “OFF” should turn it off. A message with type 2 should cause the controller to clean up (turn off LED, remove queue) and exit.
     3. Write a separate “command” client program that takes a command-line argument (“on”, “off”, or “exit”) and sends the appropriate message to the controller.
   • Verification: Running ./command on should light the LED. Running ./command off should turn it off. Running ./command exit should terminate the controller process cleanly.
4. Queue Status Monitor:
   • Objective: Write a utility that inspects the status of a message queue.
   • Guidance:
     1. Write a program that takes a message queue key as a command-line argument.
     2. The program should use msgget() to get the queue ID.
     3. It should then call msgctl() with the IPC_STAT command to retrieve the msqid_ds structure.
     4. Print out useful information from the structure, such as the current number of messages on the queue (msg_qnum), the PID of the last sender (msg_lspid), and the PID of the last receiver (msg_lrpid).
   • Verification: Run your server from the main example. While it’s waiting for a message, run your status monitor utility. It should report 0 messages. Then run the client and run the monitor again before the server has time to shut down (add a sleep(5) in the server before msgctl(IPC_RMID)). The monitor should now report the client’s PID as the last sender.

Summary

• System V Message Queues are a kernel-persistent IPC mechanism for exchanging structured messages between processes.
• Access to queues is managed via a key_t key and a message queue identifier (msqid) returned by msgget().
• The msgget() system call is used to either create a new queue or get the ID of an existing one. It sets the crucial ownership and permission flags.
• Messages are sent with msgsnd() and received with msgrcv(). These calls are blocking by default but can be made non-blocking with the IPC_NOWAIT flag.
• Every message must have a positive long type, which enables powerful prioritized message retrieval. A receiver can fetch messages by specific type, any type, or the first type up to a certain value.
• The msgctl() system call is used for administrative control, most importantly for querying status with IPC_STAT and for removing the queue from the kernel with IPC_RMID.
• Proper cleanup is essential. Leaked message queues consume kernel memory until the system reboots or they are manually removed with the ipcrm command.

Further Reading

1. Linux man-pages: The official and most authoritative source.
   • man 2 msgget
   • man 2 msgsnd
   • man 2 msgrcv
   • man 2 msgctl
   • man 7 svipc
2. “The Linux Programming Interface” by Michael Kerrisk: Chapter 46 provides an exhaustive and excellent explanation of System V message queues.
3. “Advanced Programming in the UNIX Environment” by W. Richard Stevens and Stephen A. Rago: A classic text that covers System V IPC in great detail.
4. Yocto Project Mega-Manual: While not directly about System V, understanding how build systems like Yocto manage system startup (systemd) is crucial for robustly deploying multi-process applications.
5. Raspberry Pi Documentation: Official hardware documentation for GPIO and other peripherals relevant to building embedded projects.
6. “Understanding the Linux Kernel” by Daniel P. Bovet & Marco Cesati: For those who want a deeper understanding of how the kernel implements IPC mechanisms.
7. LWN.net: An excellent source for articles on the internals of the Linux kernel, often including historical context and discussions on the evolution of IPC.