Programming with Python | Chapter 23: Working with APIs and the requests Library

Chapter Objectives

Understand the concept of an API (Application Programming Interface).
Learn about Web APIs, particularly REST APIs that use HTTP methods.
Understand common HTTP methods like GET and POST.
Install and use the third-party requests library to interact with web APIs.
Make GET requests to fetch data from an API endpoint.
Make basic POST requests to send data to an API endpoint.
Inspect the Response object (status_code, headers, content).
Work with API responses, particularly those in JSON format using response.json().
Pass URL parameters in GET requests.
Send data in the request body for POST requests.
Handle potential HTTP errors using response.raise_for_status() and exception handling.

Introduction

Modern applications rarely exist in isolation. They often need to communicate with other software components, services, or data sources. An Application Programming Interface (API) defines a set of rules and protocols that allow different software applications to communicate with each other. Web APIs are particularly common, allowing programs to interact with web services over the internet, typically using the HTTP protocol. These APIs enable access to vast amounts of data (like weather information, stock prices, social media posts) or allow control over external services (like sending emails or processing payments). This chapter introduces the fundamentals of web APIs and demonstrates how to use the popular third-party Python library, requests, to easily send HTTP requests and handle the responses, often involving the JSON data format we learned about previously.

Theory & Explanation

What is an API?

An API (Application Programming Interface) is essentially a contract or specification provided by a software component that defines how other components can interact with it. It outlines the available functions or operations, the data formats required for input, and the data formats provided as output. APIs hide the internal complexity of a system, exposing only the necessary parts for interaction.

Analogy: Think of a restaurant menu. The menu (API) tells you what dishes (operations/data) you can order, what information you need to provide (your order), and what you’ll get back (the food). You don’t need to know the exact kitchen procedures (internal implementation) to use the menu.

Web APIs (HTTP/REST)

Web APIs allow interaction with services over the web using the HTTP protocol (the same protocol your web browser uses). Many modern web APIs follow the principles of REST (Representational State Transfer). RESTful APIs typically involve:

Resources: Identifiable entities (e.g., a user, a product, a weather report) accessed via URLs (Uniform Resource Locators), also called endpoints. Example endpoint: https://api.example.com/v1/users/123
HTTP Methods (Verbs): Standard HTTP methods define the action to be performed on the resource:
- GET: Retrieve data about a resource. (Safe, repeatable).
- POST: Create a new resource or submit data for processing.
- PUT: Update an existing resource completely.
- PATCH: Partially update an existing resource.
- DELETE: Remove a resource.
Data Format: Data is often exchanged in JSON format, although XML or other formats can also be used.
Statelessness: Each request from the client to the server must contain all the information needed to understand and process the request. The server doesn’t store client context between requests (though authentication tokens are often used).

The `requests` Library

While Python has built-in modules for handling HTTP (urllib), the third-party requests library is overwhelmingly preferred due to its simplicity and ease of use.

Installation:

You need to install it first using pip:

Python

pip install requests

Making a `GET` Request:

The GET method is used to request data from a specified resource (URL/endpoint).

Python

import requests

api_url = "https://jsonplaceholder.typicode.com/todos/1" # A free fake API endpoint

try:
    # Make the GET request
    response = requests.get(api_url)

    # Raise an exception for bad status codes (4xx or 5xx)
    response.raise_for_status()

    # If successful (status code 2xx)
    print(f"GET Request Successful! Status Code: {response.status_code}")

    # Process the response (often JSON)
    data = response.json() # Parses JSON response into a Python dict/list
    print("Response Data:")
    print(data)
    print(f"\nTodo Title: {data.get('title')}")

except requests.exceptions.RequestException as e:
    print(f"An error occurred during the request: {e}")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

graph TD
    A[Start] --> B("Define API URL");
    B --> C{"Call <i>requests.get(url)</i>"};
    C -- Response Received --> D{Check <i>response.status_code</i> or <i>response.ok</i>};
    D -- Success (e.g., 200) --> E{"Call <i>response.json()</i>"};
    E -- Parsed Data --> F["Process Python <b>dict/list</b>"];
    D -- Error (e.g., 404, 500) --> G["Handle HTTP Error"];
    E -- JSONDecodeError --> H["Handle JSON Parsing Error"];
    F --> Z[End];
    G --> Z;
    H --> Z;

    style E fill:#ccf,stroke:#333,stroke-width:1px
    style F fill:#cfc,stroke:#333,stroke-width:1px
    style G fill:#fcc,stroke:#c33,stroke-width:1px
    style H fill:#fcc,stroke:#c33,stroke-width:1px

The Response Object

When you make a request using requests, it returns a Response object containing the server’s response. Key attributes/methods include:

response.status_code: The HTTP status code (integer). Common codes:
- 200 OK: Request succeeded.
- 201 Created: Resource successfully created (often after POST/PUT).
- 204 No Content: Request succeeded, but no data to return.
- 400 Bad Request: Server couldn’t understand the request (e.g., invalid syntax).
- 401 Unauthorized: Authentication required.
- 403 Forbidden: Authenticated, but lack permission for the resource.
- 404 Not Found: The requested resource doesn’t exist.
- 500 Internal Server Error: A generic error occurred on the server.
response.ok: Boolean, True if status_code is less than 400, False otherwise.
response.raise_for_status(): Raises an requests.exceptions.HTTPError exception if the status code indicates an error (4xx or 5xx). Does nothing if the status code is successful (2xx). It’s good practice to call this after getting a response.
response.text: The response body as a string (decoded using guessed encoding).
response.content: The response body as raw bytes.
response.json(): Parses the response body (if it’s valid JSON) into a Python dictionary or list. Raises requests.exceptions.JSONDecodeError if parsing fails.
response.headers: A dictionary-like object containing the HTTP response headers (e.g., Content-Type, Date).

graph TD
    RFS1[Start] --> RFS2{"Make Request (GET/POST etc.)"};
    RFS2 -- Response Received --> RFS3{Start `try` block};
    RFS3 --> RFS4{"Call <i>response.raise_for_status()</i>"};
    RFS4 -- No Exception (Status OK) --> RFS5["Process successful response (e.g., <i>response.json()</i>)"];
    RFS4 -- HTTPError Exception (Status 4xx/5xx) --> RFS6{`except requests.exceptions.HTTPError as e`};
    RFS6 --> RFS7["Handle HTTP error (e.g., log <i>e</i>)"];
    RFS5 --> RFS8{End `try` block};
    RFS7 --> RFS8;
    RFS8 --> RFSZ[End];

    style RFS4 fill:#ccf,stroke:#333,stroke-width:1px
    style RFS5 fill:#cfc,stroke:#333,stroke-width:1px
    style RFS6 fill:#fcc,stroke:#c33,stroke-width:1px

Working with JSON APIs

Most modern web APIs return data in JSON format. The response.json() method makes working with this data easy.

Python

import requests

api_url = "https://api.github.com/users/python" # GitHub API endpoint

try:
    response = requests.get(api_url)
    response.raise_for_status() # Check for HTTP errors

    user_data = response.json() # Parse JSON response

    print(f"User Name: {user_data.get('name')}")
    print(f"Public Repos: {user_data.get('public_repos')}")
    print(f"Location: {user_data.get('location')}")
    print(f"Followers: {user_data.get('followers')}")

except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
except requests.exceptions.JSONDecodeError:
    print("Failed to decode JSON response.")
except KeyError as e:
    print(f"Missing expected key in response data: {e}")

Passing URL Parameters (`params`)

GET requests often accept parameters appended to the URL (e.g., ?key1=value1&key2=value2) for filtering, searching, or pagination. requests allows you to pass these as a dictionary to the params argument.

Python

import requests

search_url = "https://jsonplaceholder.typicode.com/posts"
# Define parameters as a dictionary
query_params = {'userId': 1, '_limit': 5} # Get posts by userId 1, limit to 5 results

try:
    response = requests.get(search_url, params=query_params)
    response.raise_for_status()
    # requests builds the final URL: https://jsonplaceholder.typicode.com/posts?userId=1&_limit=5
    print(f"Request URL: {response.url}")

    posts = response.json()
    print(f"\nFound {len(posts)} posts for userId 1 (limit 5):")
    for post in posts:
        print(f"- ID: {post['id']}, Title: {post['title'][:30]}...") # Print first 30 chars

except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

graph TD
    GP1[Start] --> GP2["Define URL & <b>params</b> dict"];
    GP2 --> GP3{"Call <i>requests.get(url, params=params)</i>"};
    GP3 -- requests builds final URL --> GP4("Sends request to e.g., <i>url?key=value</i>");
    GP4 -- Response Received --> GP5{"Check status & process response (as in basic GET)"};
    GP5 --> GPZ[End];

    style GP3 fill:#ccf,stroke:#333,stroke-width:1px

Making a `POST` Request

The POST method is typically used to send data to an API endpoint to create a new resource or submit data for processing. Data is usually sent in the request body, often as JSON.

graph TD
    P1[Start] --> P2["Define API URL & Python <b>payload</b> (dict)"];
    P2 --> P3{"Call <i>requests.post(url, json=payload)</i>"};
    P3 -- Response Received --> P4{Check <i>response.status_code</i>};
    P4 -- Success (e.g., 201 Created) --> P5{"Optional: Call <i>response.json()</i> for created resource details"};
    P5 -- Parsed Data --> P6["Process Response Data"];
    P4 -- Error (e.g., 400, 500) --> P7["Handle HTTP Error"];
    P5 -- JSONDecodeError --> P8["Handle JSON Parsing Error"];
    P6 --> PZ[End];
    P7 --> PZ;
    P8 --> PZ;

    style P3 fill:#ccf,stroke:#333,stroke-width:1px
    style P5 fill:#ccf,stroke:#333,stroke-width:1px
    style P6 fill:#cfc,stroke:#333,stroke-width:1px
    style P7 fill:#fcc,stroke:#c33,stroke-width:1px
    style P8 fill:#fcc,stroke:#c33,stroke-width:1px

Python

import requests
import json # Often needed to format the data being sent

post_url = "https://jsonplaceholder.typicode.com/posts"

# Data to send (as a Python dictionary)
new_post_data = {
    'title': 'My New Post',
    'body': 'This is the content of my new post.',
    'userId': 10 # Example user ID
}

# Headers often needed for POST, especially to indicate JSON content
headers = {'Content-Type': 'application/json; charset=utf-8'}

try:
    # Send POST request with data converted to JSON in the body
    # Use 'json=' parameter for automatic JSON encoding and correct headers
    response = requests.post(post_url, json=new_post_data)
    # Alternative (manual encoding):
    # response = requests.post(post_url, data=json.dumps(new_post_data), headers=headers)

    response.raise_for_status() # Check for 4xx/5xx errors

    print(f"POST Request Successful! Status Code: {response.status_code}")
    # APIs often return the created resource (with its new ID) in the response
    created_post = response.json()
    print("Response Data (Created Post):")
    print(created_post)

except requests.exceptions.RequestException as e:
    print(f"POST request failed: {e}")

Note: When using json=python_dict, requests automatically serializes the dictionary to a JSON string and sets the Content-Type header to application/json. This is generally preferred over manually using data=json.dumps(...) and setting headers.

Code Examples

Example 1: Fetching Data from a Public API (Random User)

Python

# fetch_random_user.py
import requests

API_ENDPOINT = "https://randomuser.me/api/"

print("Fetching random user data...")
try:
    response = requests.get(API_ENDPOINT)
    response.raise_for_status() # Check for HTTP errors

    data = response.json()

    # API returns results in a list, get the first one
    if data['results']:
        user = data['results'][0]
        name = user['name']
        location = user['location']
        email = user['email']

        print("\n--- Random User ---")
        print(f"Name: {name['title']}. {name['first']} {name['last']}")
        print(f"Gender: {user['gender']}")
        print(f"Email: {email}")
        print(f"Location: {location['city']}, {location['country']}")
        print(f"Coordinates: ({location['coordinates']['latitude']}, {location['coordinates']['longitude']})")
        print(f"Photo (thumbnail): {user['picture']['thumbnail']}")
        print("-------------------")
    else:
        print("No user data found in the response.")

except requests.exceptions.RequestException as e:
    print(f"Could not fetch data: {e}")
except (KeyError, IndexError, TypeError, json.JSONDecodeError) as e:
    # Handle potential issues with the structure of the returned JSON
    print(f"Error processing response data: {e}")
    # print("Response text was:", response.text) # Uncomment for debugging

Explanation:

Makes a GET request to the randomuser.me API.
Checks for HTTP errors using raise_for_status().
Parses the JSON response using response.json().
Extracts specific fields (name, location, email, etc.) from the nested dictionary structure returned by the API.
Includes basic error handling for request issues and potential problems with the JSON structure (e.g., missing keys).

Example 2: Posting Data to a Mock API (ReqRes.in)

Python

# post_user_data.py
import requests

API_ENDPOINT = "https://reqres.in/api/users" # Mock API for testing POST

user_to_create = {
    "name": "Neo",
    "job": "The One"
}

print(f"Attempting to create user: {user_to_create}")

try:
    # Use json parameter to send dictionary as JSON body
    response = requests.post(API_ENDPOINT, json=user_to_create)

    # Check status code explicitly (ReqRes often returns 201 Created)
    if response.status_code == 201:
        print(f"\nUser created successfully! Status: {response.status_code}")
        created_data = response.json()
        print("Server Response:")
        print(created_data)
        # Response typically includes the created data plus an 'id' and 'createdAt' timestamp
        print(f"\nNew User ID: {created_data.get('id')}")
    else:
        # Handle other potential success/failure codes if needed
        print(f"\nRequest completed, but status code was: {response.status_code}")
        print(f"Response body: {response.text}")
        # Optionally raise for status for non-201 codes if desired
        # response.raise_for_status()

except requests.exceptions.RequestException as e:
    print(f"\nAn error occurred during the POST request: {e}")

Explanation:

Defines the data for a new user in a Python dictionary.
Makes a POST request to the reqres.in mock API endpoint, sending the dictionary using the json parameter. requests handles JSON serialization and headers.
Checks specifically for the 201 Created status code, which is common for successful resource creation via POST.
Parses and prints the JSON response from the server, which usually includes the created resource’s details and a server-assigned ID.

Common Mistakes or Pitfalls

Forgetting to Install requests: requests is not built-in; pip install requests is required.
Ignoring HTTP Status Codes: Not checking response.status_code or calling response.raise_for_status() can lead to processing invalid or error responses as if they were successful.
JSON Decoding Errors: Calling response.json() on a response that is not valid JSON (e.g., HTML error page, plain text) will raise a JSONDecodeError. Check response.headers['Content-Type'] or handle the exception.
Incorrect HTTP Method: Using GET when POST is required by the API, or vice-versa. API documentation specifies the correct method for each endpoint.
Missing/Incorrect Headers: Some APIs require specific HTTP headers (e.g., Content-Type for POST, Accept for desired response format, Authorization for authentication tokens). Forgetting or misspelling them can cause errors (often 400 Bad Request or 401/403).
Incorrect Data Formatting: Sending data in the wrong format (e.g., not JSON-encoding POST data when the API expects application/json). Using the json= parameter in requests.post helps avoid this.
Rate Limiting: Making too many requests to an API in a short period can trigger rate limits, resulting in error status codes (often 429 Too Many Requests). Check API documentation for limits and implement delays (time.sleep()) if necessary.
Handling API Structure Changes: Relying too rigidly on the exact structure of an API response. If the API provider changes the structure (e.g., renames a key), your code might break. Check for key existence (.get() or in) or use more robust parsing.

Chapter Summary

Type	Function/Attribute	Purpose	Key Syntax/Example
Request	`requests.get(url, params=None, ...)`	Sends an HTTP GET request to retrieve data from a URL.	`response = requests.get('api/users', params={'id': 1})`
Request	`requests.post(url, data=None, json=None, ...)`	Sends an HTTP POST request, often to submit data or create a resource. Use `json=` for automatic dict-to-JSON conversion.	`payload = {'name': 'Neo'}` `response = requests.post('api/users', json=payload)`
Response	`response.status_code`	Integer representing the HTTP status code returned by the server (e.g., 200, 404, 500).	`if response.status_code == 200: print("OK")`
Response	`response.ok`	Boolean indicating if the request was successful (status code < 400).	`if response.ok: print("Success!")`
Response	`response.raise_for_status()`	Raises an `HTTPError` exception if the status code indicates an error (4xx or 5xx). Does nothing for success codes (2xx).	`try:` `response.raise_for_status()` `except requests.exceptions.HTTPError as err:` `print(f"HTTP Error: {err}")`
Response	`response.text`	The response body decoded as a string.	`html_content = response.text`
Response	`response.content`	The response body as raw bytes (useful for non-text content like images).	`image_bytes = response.content`
Response	`response.json()`	Parses the response body (if valid JSON) into a Python dictionary or list. Raises `JSONDecodeError` on failure.	`try:` `data = response.json()` `except requests.exceptions.JSONDecodeError:` `print("Not valid JSON")`
Response	`response.headers`	A case-insensitive dictionary-like object containing the HTTP response headers.	`content_type = response.headers.get('Content-Type')`
Response	`response.url`	The final URL the request was made to (after potential redirects and parameter encoding).	`print(f"Requested URL: {response.url}")`

APIs allow software components to communicate. Web APIs use HTTP, often following REST principles (URLs/endpoints, HTTP verbs, JSON data).
The requests library simplifies making HTTP requests in Python (pip install requests).
Use requests.get(url, params=...) to retrieve data.
Use requests.post(url, json=...) to send data (usually creating resources).
The Response object contains the server’s reply: status_code, ok, headers, text, content, json().
Always check the status code (e.g., using response.raise_for_status()) before processing the response.
Use response.json() to parse JSON responses into Python dictionaries/lists.
Handle potential requests.exceptions.RequestException errors and JSONDecodeError.

Exercises & Mini Projects

Exercises

GET Request: Use requests to make a GET request to https://jsonplaceholder.typicode.com/users/1. Check the status code. If successful, parse the JSON response and print the user’s name and email address.
GET with Params: Make a GET request to https://jsonplaceholder.typicode.com/comments. Use the params argument to filter comments for postId=5. Print the number of comments received and the email address of the first comment in the list.
Check Headers: Make a GET request to https://api.github.com. Print the Content-Type header from the response headers.
Error Handling: Make a GET request to an invalid URL (e.g., https://httpbin.org/status/404 which returns a 404). Use response.raise_for_status() within a try...except requests.exceptions.HTTPError block to catch and print the HTTP error.
JSON Handling: Make a GET request to https://jsonplaceholder.typicode.com/posts/101 (this ID likely doesn’t exist). Check if the response status code is 404. Try calling response.json() and wrap it in a try...except requests.exceptions.JSONDecodeError block (or check content type first) as a 404 response might not contain valid JSON.

Mini Project: Public API Data Fetcher

Goal: Write a script that fetches data from a simple, free public API and displays formatted information. Choose one of the following APIs:

Open Notify (ISS Location): http://api.open-notify.org/iss-now.json (Returns current latitude/longitude of the International Space Station)
Public APIs List: https://api.publicapis.org/random (Returns information about a random free public API)
Cat Facts: https://catfact.ninja/fact (Returns a random cat fact)

Steps:

Import requests.
Define API URL: Choose one API from the list above and store its URL in a variable.
Make GET Request:
- Use a try...except requests.exceptions.RequestException block.
- Inside the try block, make a GET request to the API URL.
- Call response.raise_for_status() to check for HTTP errors.
Process JSON Response:
- If the request is successful, parse the JSON response using response.json().
- Use another try...except (KeyError, json.JSONDecodeError) block around the JSON processing for safety.
- Extract the relevant information based on the chosen API’s response structure (you may need to visit the URL in your browser first to see what the JSON looks like).
  - ISS Location: Extract latitude, longitude, timestamp.
  - Public API: Extract API, Description, Link, Category.
  - Cat Fact: Extract fact, length.
Display Data: Print the extracted information in a user-friendly formatted way.
Error Handling: Ensure your except blocks print informative messages if the request fails or the JSON parsing/processing encounters an issue.

Additional Sources:

How to Use an API in Python – Dataquest

Programming with Python | Chapter 23: Working with APIs and the requests Library

Chapter Objectives

Introduction

Theory & Explanation

What is an API?

Web APIs (HTTP/REST)

The requests Library

Installation:

Making a GET Request:

The Response Object

Working with JSON APIs

Passing URL Parameters (params)

Making a POST Request

Code Examples

Example 1: Fetching Data from a Public API (Random User)

Example 2: Posting Data to a Mock API (ReqRes.in)

Common Mistakes or Pitfalls

Chapter Summary

Exercises & Mini Projects

Exercises

Mini Project: Public API Data Fetcher

Related Posts

Leave a Comment Cancel Reply

The `requests` Library

Making a `GET` Request:

Passing URL Parameters (`params`)

Making a `POST` Request