Overview
Managing the rate of requests is crucial when dealing with web services to avoid overwhelming servers and violating API rate limits. The aiohttp library in Python enables asynchronous requests, and this tutorial demonstrates how to rate-limit them.
Before diving into rate-limiting with aiohttp, ensure you have Python 3.7 or later installed with the aiohttp package. You can install aiohttp using pip install aiohttp.
Basic Rate-Limiting
Let’s begin with a basic example that sends HTTP requests and rate-limits them to a specified number per second.
import aiohttp
import asyncio
import time
async def fetch(url, session):
async with session.get(url) as response:
return await response.text()
async def rate_limited_requester(urls, rate):
async with aiohttp.ClientSession() as session:
tasks = []
for url in urls:
task = asyncio.create_task(fetch(url, session))
tasks.append(task)
await asyncio.sleep(1 / rate)
responses = await asyncio.gather(*tasks)
return responses
urls = ['http://httpbin.org/get' for _ in range(10)]
rate = 2 # Two requests per second
loop = asyncio.get_event_loop()
loop.run_until_complete(rate_limited_requester(urls, rate))
Using a Semaphore
Another approach is to use a semaphore to limit the number of simultaneous requests. This can be combined with the rate limiting.
import aiohttp
import asyncio
import time
async def fetch(url, session, semaphore):
async with semaphore:
async with session.get(url) as response:
return await response.text()
async def rate_limited_requester(urls, rate):
async with aiohttp.ClientSession() as session:
semaphore = asyncio.Semaphore(rate)
tasks = []
for url in urls:
task = asyncio.create_task(fetch(url, session, semaphore))
tasks.append(task)
responses = await asyncio.gather(*tasks)
return responses
urls = ['http://httpbin.org/get' for _ in range(10)]
rate = 2 # Two requests per second
loop = asyncio.get_event_loop()
loop.run_until_complete(rate_limited_requester(urls, rate))
You can read a dedicated guide to Semaphore here: Python asyncio.Semaphore class explained (with examples).
Advanced Techniques
For more precise rate-limiting, we can use third-party libraries such as aiolimiter, which provides an asynchronous leaky bucket algorithm implementation.
from aiolimiter import AsyncLimiter
import aiohttp
import asyncio
limiter = AsyncLimiter(max_rate=2, time_period=1)
async def fetch(url, session, limiter):
async with limiter:
async with session.get(url) as response:
return await response.text()
async def main(urls):
async with aiohttp.ClientSession() as session:
tasks = [fetch(url, session, limiter) for url in urls]
return await asyncio.gather(*tasks)
urls = ['http://httpbin.org/get' for _ in range(20)]
loop = asyncio.get_event_loop()
loop.run_until_complete(main(urls))
Incorporating a retry mechanism or a backoff strategy when rate limits are reached can be an essential feature for a robust system. Here’s how to implement a simple exponential backoff in case of a 429 status (Too Many Requests) response.
import random
async def fetch_with_backoff(url, session, limiter, max_retries=5):
for retry in range(max_retries):
await limiter
async with session.get(url) as response:
if response.status == 429:
backoff_time = 2 ** retry + random.uniform(0, 1)
await asyncio.sleep(backoff_time)
else:
return await response.text()
return 'Max retries reached'
# Rest of the code remains similar
Conclusion
This tutorial has walked you through several ways to rate-limit requests using the aiohttp library in Python. By applying these patterns, you can avoid API rate limits, make your application more robust, and protect the services you are interacting with from being overwhelmed.
