Python Requests Module: How to Set Timeouts

Introduction

In web development, it’s crucial to ensure requests respond within an acceptable timeframe. The Python Requests module provides easy-to-use functions to handle HTTP requests, including the ability to set timeouts to prevent hanging requests.

Getting Started with Requests

To start with Python’s Requests module, ensure you have it installed:

pip install requests

Here’s a simple example to illustrate how to make a GET request:

import requests

response = requests.get('https://api.example.com/data')
print(response.text)

Understanding Timeouts in Requests

By default, the Requests module doesn’t apply a timeout to your requests. This means your code could end up waiting indefinitely for a response. The solution is to define a timeout period – the maximum amount of time you’ll wait for a response.

response = requests.get('https://api.example.com/data', timeout=5)

# Raises an error if the request takes longer than 5 seconds

Setting Timeouts: The Basics

Setting a timeout is straightforward. You pass the timeout parameter with your request, specifying the number of seconds you’ll wait:

import requests

try:
    response = requests.get('https://api.example.com/data', timeout=10)
    print(response.text)
except requests.exceptions.Timeout:
    print('The request timed out')

This code will wait up to 10 seconds for a response before raising a Timeout exception.

Advanced Timeout Settings

The timeout parameter can also be a tuple (connect_timeout, read_timeout), allowing you to set separate timeouts for the connection and the response:

try:
    response = requests.get('https://api.example.com/data', timeout=(3.05, 27))
    print(response.text)
except requests.exceptions.ReadTimeout:
    print('The server did not send any data in the allotted amount of time.')
except requests.exceptions.ConnectTimeout:
    print('The request timed out while trying to connect to the remote server')

This specifies a max of 3.05 seconds to establish a connection and 27 seconds to wait for a response.

Error Handling and Timeouts

Beyond setting timeouts, it’s important to handle the exceptions these timeouts might raise:

import requests

try:
    response = requests.get('https://api.example.com/data', timeout=5)
except requests.exceptions.Timeout as e:
    print(f'Timeout occurred: {str(e)}')
except requests.exceptions.RequestException as e:
    # Handle any request exceptions
    print(f'An error occurred: {str(e)}')

Real World Scenarios

In a live production environment, you might be dealing with multiple endpoints, which can have different response behaviors. You might want to set different timeouts based on the endpoint:

ENDPOINT_TIMEOUTS = {
    '/fast': (1, 5),
    '/slow': (10, 30)
}

endpoint = '/slow'
try:
    response = requests.get(f'https://api.example.com{endpoint}',
                            timeout=ENDPOINT_TIMEOUTS.get(endpoint, (3, 9)))
    print(response.text)
except requests.exceptions.Timeout:
    print(f'{endpoint} request timed out')

Best Practices

While setting timeouts, it’s good practice to base the values on measurements of how your requests typically perform, rather than guessing. Additionally, think about how your application should handle exceptions and potentially retry requests. Combining timeouts with retry strategies can lead to more robust request handling:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

try:
    response = session.get('https://api.example.com/data', timeout=2.5)
    print(response.text)
except requests.exceptions.RequestException as e:
    print(f'Exception occurred: {str(e)}')

Conclusion

The Python Requests module is a versatile tool that, when paired with appropriate timeout settings, allows you to maintain control over your application’s flow and prevent unexpected hanging requests. Remember to handle timeouts thoughtfully to ensure your application can deal with network uncertainties effectively.