Python ‘requests’ module: Handle CSV response

Introduction

Working with CSV data is common in Python, especially when interacting with web APIs. The ‘requests’ module provides an easy-to-use method for fetching resources from the internet, including CSV files.

Getting Started with Requests

Firstly, ensure the ‘requests’ module is installed:

pip install requests

To fetch a CSV file using ‘requests’, you can make a simple GET request as follows:

import requests

url = 'http://example.com/data.csv'
response = requests.get(url)

Assuming the URL returns a CSV file, the response content will be a string that represents the CSV file.

Reading CSV Response into Memory

You can read this CSV data using Python’s csv library:

import csv
from io import StringIO

data = StringIO(response.text)
csv_reader = csv.reader(data, delimiter=',')
for row in csv_reader:
    print(row)

Handling CSV Data with Pandas

Pandas is a powerful tool for data manipulation. To work with CSV data using Pandas:

import pandas as pd

data = pd.read_csv(StringIO(response.text))
print(data.head())

Saving CSV to a File

If you want to save the CSV content to a file, you can do the following:

with open('downloaded.csv', 'w') as file:
    file.write(response.text)

Advanced Usage: Streaming Large CSV Files

For large CSV files, you may want to stream the response to avoid loading it all into memory:

with requests.get(url, stream=True) as r:
    with open('large_file.csv', 'w') as f:
        for chunk in r.iter_content(chunk_size=8192):
            if chunk:
                f.write(chunk.decode('utf-8'))

Handling Exceptions

It’s important to handle potential exceptions that may occur during the network request:

try:
    response = requests.get(url, timeout=5)
    response.raise_for_status()  # Raises an HTTPError if the HTTP request returned an unsuccessful status code
    # handle CSV data here
except requests.exceptions.HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')
except requests.exceptions.ConnectionError as conn_err:
    print(f'Connection error occurred: {conn_err}')
except requests.exceptions.Timeout as timeout_err:
    print(f'Timeout error occurred: {timeout_err}')
except Exception as err:
    print(f'An error occurred: {err}')

Conclusion

Handling CSV responses with Python’s ‘requests’ module is straightforward. By combining it with either the native ‘csv’ module or with ‘pandas’, Python programmers can efficiently process CSV data obtained from the web for analysis, storage, or other purposes.