Sling Academy
Home/Python/Python requests library: How to set headers

Python requests library: How to set headers

Last updated: January 02, 2024

Introduction

Making HTTP requests in Python is a core functionality for many applications. The requests library simplifies this task, providing a way to send both simple and complex requests with ease. In this tutorial, we will focus on how to set headers in your requests, allowing you to interact with web services that require specific contextual information.

Basic Usage of Headers

Headers are a key part of HTTP requests and responses. They define operating parameters of the HTTP transaction, such as content type or authentication tokens. In the requests library, setting headers is straightforward:

import requests

url = 'https://api.example.com/data'
headers = {
    'User-Agent': 'my-app/0.0.1',
    'Accept': 'application/json',
    'Authorization': 'Bearer YOUR_ACCESS_TOKEN',
}

response = requests.get(url, headers=headers)
print(response.json())

Custom Headers for POST Requests

When making POST requests, you often need to send additional data in the headers, like ‘Content-Type’. Here’s an example of setting custom headers for a POST request:

import requests

url = 'https://api.example.com/submit'
data = {'key': 'value'}
headers = {
    'Content-Type': 'application/json',
}

response = requests.post(url, json=data, headers=headers)
print(response.text)

Dynamic Headers

Sometimes headers need to be dynamically constructed. One way to do this is by defining a function that generates the necessary headers:

import requests

def create_headers(token):
    return {
        'Authorization': f'Bearer {token}',
        'Accept': 'application/json'
    }

url = 'https://api.example.com/data'
token = 'YOUR_ACCESS_TOKEN'
headers = create_headers(token)

response = requests.get(url, headers=headers)
print(response.json())

Advanced Usage: Session Objects and Header Persistence

In advanced scenarios, you might want to persist certain headers across multiple requests. The requests.Session object allows you to do this elegantly:

import requests

session = requests.Session()
session.headers.update({
    'User-Agent': 'my-app/0.1',
    'Authorization': 'Bearer YOUR_ACCESS_TOKEN'
})

response1 = session.get('https://api.example.com/data1')
response2 = session.get('https://api.example.com/data2')

print(response1.json())
print(response2.json())

Conditional Headers and Error Handling

It is important to handle situations where headers may or may not be needed based on the response code or the presence of certain data. The following example demonstrates conditional headers and basic error handling:

import requests
from requests.exceptions import HTTPError

url = 'https://api.example.com/conditional'
def get_with_conditional_header(url, token=None):
    headers = {}
    if token:
        headers['Authorization'] = f'Bearer {token}'
    response = requests.get(url, headers=headers)
    try:
        response.raise_for_status()
        return response.json()
    except HTTPError as http_err:
        print(f'HTTP error occurred: {http_err}')
    except Exception as err:
        print(f'Other error occurred: {err}')

json_response = get_with_conditional_header(url, token='YOUR_ACCESS_TOKEN')
print(json_response)

Working with Redirects

By default, requests follows redirects, but there are times when you may need to inspect or modify headers before the next redirected request. Here’s how to handle redirects manually:

import requests

url = 'https://api.example.com/redirect'
response = requests.get(url, allow_redirects=False)
if response.status_code == 301:
    new_url = response.headers['Location']
    new_response = requests.get(new_url)
    print(new_response.content)

Handling Cookies

Cookies are another form of header, and requests offers a simple interface for handling them:

import requests

url = 'https://api.example.com/needs_cookies'
session = requests.Session()
session.cookies.set('session_id', '123456789')
response = session.get(url)

print(response.text)

Conclusion

Throughout this tutorial, you’ve learned how to set headers in the Python requests library, from the basics to more advanced techniques. Knowing how to properly set headers is essential for making HTTP requests because they provide critical information to servers, and in turn, help control the response you receive. Practice with different types of headers and configurations to master your HTTP requests in Python.

Next Article: Python requests library: How to set params

Previous Article: Python: 3 ways to Install Packages Offline (without Internet)

Series: Python: Network & JSON tutorials

Python

You May Also Like

  • Python Warning: Secure coding is not enabled for restorable state
  • Python TypeError: write() argument must be str, not bytes
  • 4 ways to install Python modules on Windows without admin rights
  • Python TypeError: object of type ‘NoneType’ has no len()
  • Python: How to access command-line arguments (3 approaches)
  • Understanding ‘Never’ type in Python 3.11+ (5 examples)
  • Python: 3 Ways to Retrieve City/Country from IP Address
  • Using Type Aliases in Python: A Practical Guide (with Examples)
  • Python: Defining distinct types using NewType class
  • Using Optional Type in Python (explained with examples)
  • Python: How to Override Methods in Classes
  • Python: Define Generic Types for Lists of Nested Dictionaries
  • Python: Defining type for a list that can contain both numbers and strings
  • Using TypeGuard in Python (Python 3.10+)
  • Python: Using ‘NoReturn’ type with functions
  • Type Casting in Python: The Ultimate Guide (with Examples)
  • Python: Using type hints with class methods and properties
  • Python: Typing a function with default parameters
  • Python: Typing a function that can return multiple types