Python Requests module: Auto retry for failed HTTP requests

Introduction

Building resilience into HTTP requests is critical for robust applications. This tutorial dives into how you can implement automatic retries for failed HTTP requests using the Python Requests module, ensuring your application remains stable even when faced with unreliable networks or flaky services.

Understanding HTTP Retries

Before jumping into the code, let’s understand when and why HTTP requests may need to be retried. Network flakiness, server overloads, and transient issues can cause requests to fail, but these problems are often temporary. By implementing a retry mechanism, we give our program a chance to overcome intermittent failures without manual intervention.

Simple Retry with Requests

To begin, let’s look at a basic example of making an HTTP request using the Requests module and handling failures:

import requests

try:
    response = requests.get('https://example.com')
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    # Handle the HTTP error here, maybe retry once.
    print(f'HTTP error occurred: {e}')
except requests.exceptions.RequestException as e:
    # Handle any other errors here.
    print(f'Other error occurred: {e}')

Using urllib3 Retry

Now let’s take our retry logic a step further. Requests uses urllib3 under the hood, which provides a more robust way to handle retries with its Retry class:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

session = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
session.mount('https://', HTTPAdapter(max_retries=retries))

try:
    response = session.get('https://example.com')
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    print(f'HTTP Error: {e}')
except requests.exceptions.ConnectionError as e:
    print(f'Connection Failed: {e}')
except requests.exceptions.Timeout as e:
    print(f'Timeout Occurred: {e}')
except requests.exceptions.RequestException as e:
    print(f'An Error Occurred: {e}')

This code automatically retries any GET request to ‘https://example.com’ up to five times if it encounters one of the specified status codes. It uses an exponential backoff strategy to avoid overwhelming the server.

Advanced Retry Strategies

For more complex scenarios, you may want to define your own retry strategy. This is where you can extend the Retry class and implement specific methods to meet your requirements:

class CustomRetry(Retry):
    def __init__(self, total=5, connect=None, read=None, redirect=None, status=None):
        super().__init__(total, connect, read, redirect, status)
    
    def increment(self, *args, **kwargs):
        if self.total < 1:
            raise MaxRetryError(self, 'https://example.com')
        self.total -= 1

        # Implement custom backoff logic if needed
        
        return self

session = requests.Session()
adapter = HTTPAdapter(max_retries=CustomRetry(total=3))
session.mount('https://', adapter)

# Your request logic here

In this example, we have created a CustomRetry class that overrides the increment method to provide custom backoff logic and retry conditions.

Handling Idempotent Requests

When implementing retries, it’s important to consider the idempotence of the requests. GET, HEAD, OPTIONS, DELETE are typically safe to retry, as they do not change the server’s state. For other methods like POST or PATCH which can modify server state, additional caution is needed to ensure retrying the request won’t cause undesired side effects. The code examples above can be modified to target specific HTTP methods by adjusting the `method_whitelist` parameter of the Retry class.

Using External Libraries

If you’re looking for an even more feature-rich retry approach, there are several external libraries like `requests_retry`, `backoff`, and `tenacity` which integrate with requests to offer advanced retry mechanisms with minimal code changes.

import requests
from tenacity import retry, wait_fixed

@retry(wait=wait_fixed(2))
def make_request(url):
    response = requests.get(url)
    response.raise_for_status()
    return response

try:
    response = make_request('https://example.com')
except requests.exceptions.RequestException as e:
    print(f'Failed after retrying: {e}')

The `tenacity` library gives us a simple decorator to add retry logic with fixed or exponential backoff strategies, among other features.

Conclusion

In conclusion, automatically handling retries for HTTP requests in Python is an essential feature for creating resilient applications. We explored incremental approaches from using the Requests module with custom exception handling to more sophisticated strategies involving urllib3’s Retry class and external libraries. Implementing retries requires careful consideration of HTTP methodology and error handling but will ultimately lead to more robust solutions capable of weathering the inevitable network issues faced in production.