Python requests library: How to set headers

Introduction

Making HTTP requests in Python is a core functionality for many applications. The requests library simplifies this task, providing a way to send both simple and complex requests with ease. In this tutorial, we will focus on how to set headers in your requests, allowing you to interact with web services that require specific contextual information.

Basic Usage of Headers

Headers are a key part of HTTP requests and responses. They define operating parameters of the HTTP transaction, such as content type or authentication tokens. In the requests library, setting headers is straightforward:

import requests

url = 'https://api.example.com/data'
headers = {
    'User-Agent': 'my-app/0.0.1',
    'Accept': 'application/json',
    'Authorization': 'Bearer YOUR_ACCESS_TOKEN',
}

response = requests.get(url, headers=headers)
print(response.json())

Custom Headers for POST Requests

When making POST requests, you often need to send additional data in the headers, like ‘Content-Type’. Here’s an example of setting custom headers for a POST request:

import requests

url = 'https://api.example.com/submit'
data = {'key': 'value'}
headers = {
    'Content-Type': 'application/json',
}

response = requests.post(url, json=data, headers=headers)
print(response.text)

Dynamic Headers

Sometimes headers need to be dynamically constructed. One way to do this is by defining a function that generates the necessary headers:

import requests

def create_headers(token):
    return {
        'Authorization': f'Bearer {token}',
        'Accept': 'application/json'
    }

url = 'https://api.example.com/data'
token = 'YOUR_ACCESS_TOKEN'
headers = create_headers(token)

response = requests.get(url, headers=headers)
print(response.json())

Advanced Usage: Session Objects and Header Persistence

In advanced scenarios, you might want to persist certain headers across multiple requests. The requests.Session object allows you to do this elegantly:

import requests

session = requests.Session()
session.headers.update({
    'User-Agent': 'my-app/0.1',
    'Authorization': 'Bearer YOUR_ACCESS_TOKEN'
})

response1 = session.get('https://api.example.com/data1')
response2 = session.get('https://api.example.com/data2')

print(response1.json())
print(response2.json())

Conditional Headers and Error Handling

It is important to handle situations where headers may or may not be needed based on the response code or the presence of certain data. The following example demonstrates conditional headers and basic error handling:

import requests
from requests.exceptions import HTTPError

url = 'https://api.example.com/conditional'
def get_with_conditional_header(url, token=None):
    headers = {}
    if token:
        headers['Authorization'] = f'Bearer {token}'
    response = requests.get(url, headers=headers)
    try:
        response.raise_for_status()
        return response.json()
    except HTTPError as http_err:
        print(f'HTTP error occurred: {http_err}')
    except Exception as err:
        print(f'Other error occurred: {err}')

json_response = get_with_conditional_header(url, token='YOUR_ACCESS_TOKEN')
print(json_response)

Working with Redirects

By default, requests follows redirects, but there are times when you may need to inspect or modify headers before the next redirected request. Here’s how to handle redirects manually:

import requests

url = 'https://api.example.com/redirect'
response = requests.get(url, allow_redirects=False)
if response.status_code == 301:
    new_url = response.headers['Location']
    new_response = requests.get(new_url)
    print(new_response.content)

Handling Cookies

Cookies are another form of header, and requests offers a simple interface for handling them:

import requests

url = 'https://api.example.com/needs_cookies'
session = requests.Session()
session.cookies.set('session_id', '123456789')
response = session.get(url)

print(response.text)

Conclusion

Throughout this tutorial, you’ve learned how to set headers in the Python requests library, from the basics to more advanced techniques. Knowing how to properly set headers is essential for making HTTP requests because they provide critical information to servers, and in turn, help control the response you receive. Practice with different types of headers and configurations to master your HTTP requests in Python.