Python Requests module: How to use proxy

Introduction

Proxies are intermediary servers between your machine and the internet. Using proxies with the Python Requests module allows you to request data from web services while masking your original IP address or, to access web services that might be blocked in your region. This guide walks you through using proxies with Python’s famous requests module.

Setting Up Your Environment

To begin, you need to have Python installed on your system. Python’s requests module will also need to be installed, which you can easily do using pip:

pip install requests

Basic Proxy Usage

For basic use, you can pass your proxy details directly to the requests.get() call:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

response = requests.get('http://example.org', proxies=proxies)
print(response.text)

This sends your request to http://example.org via the proxy server located at IP 10.10.1.10 using port 3128 for HTTP and port 1080 for HTTPS requests.

Using Environment Variables

You can also set environment variables for proxies. This is particularly useful if multiple scripts or applications on your system require access through the same proxy. Here’s how you can set it on Unix-based systems:

export HTTP_PROXY='http://10.10.1.10:3128'
export HTTPS_PROXY='http://10.10.1.10:1080'

In Windows, you would set the environment variable like this:

set HTTP_PROXY=http://10.10.1.10:3128
set HTTPS_PROXY=http://10.10.1.10:1080

Once these variables are set, requests will automatically use these proxies without specifying them in the code.

Proxy Authentication

If your proxy server requires authentication, you can include your username and password in the proxy URL:

proxies = {
  'http': 'http://username:[email protected]:3128',
  'https': 'http://username:[email protected]:1080',
}

response = requests.get('http://example.org', proxies=proxies)
print(response.text)

Be aware that including credentials in your code should be avoided in a production environment due to security concerns. It’s better to use environment variables or an external configuration file for such sensitive data.

Advanced Use: Custom Proxy Middleware

For more complex proxy use cases, you might want to consider creating a session with custom middleware. The requests library allows extending functionality through adapters:

import requests
from requests.auth import HTTPProxyAuth
from requests.adapters import HTTPAdapter

class ProxyAdapter(HTTPAdapter):
    def __init__(self, proxy_url, proxy_auth, *args, **kwargs):
        super(ProxyAdapter, self).__init__(*args, **kwargs)
        self.proxy_url = proxy_url
        self.proxy_auth = proxy_auth

    def send(self, request, *args, **kwargs):
        request.headers['Proxy-Authorization'] = self.proxy_auth
        return super(ProxyAdapter, self).send(request, *args, **kwargs)

auth = HTTPProxyAuth('username', 'password')
adapter = ProxyAdapter('http://10.10.1.10:3128', auth)
session = requests.Session()
session.mount('http://', adapter)

response = session.get('http://example.org')
print(response.text)

This pattern is especially useful when you require fine-grained control over proxy behavior across all your HTTP requests.

Handling Proxies and Redirects

When dealing with redirects, the requests module by default will strip auth from the request. To handle this, you need to subclass the Session object and override how redirects are handled:

import requests
from requests.auth import HTTPProxyAuth

auth = HTTPProxyAuth('username', 'password')

class NoStripSession(requests.Session):
    def rebuild_auth(self, prepared_request, response):
        # Retain auth upon redirect
        pass

session = NoStripSession()
response = session.get('http://example.org', auth=auth, proxies=proxies)
print(response.text)

This customization gives you the flexibility to maintain proxy authentication even after a redirect has occurred.

Using HTTP(S) Proxies with Other HTTP Methods

Beyond GET requests, proxies in the Requests module can be used with POST, PUT, DELETE, HEAD and OPTIONS methods. Here’s an example using the POST method:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

data = {'key': 'value'}
response = requests.post('http://httpbin.org/post', data=data, proxies=proxies)
print(response.text)

The use of proxies is mostly similar across these methods, varying mostly in the type of data sent and the specific method used.

Conclusion

In this guide, we covered how to use proxies with the Python Requests module with increasing complexity. By incorporating proxies into your requests, you can handle more diverse network conditions and requirements. Always remember to secure proxy details, especially if authentication is involved, and consider your security needs when writing and deploying code that interacts with proxies.