Python ‘requests’ module: Handle XML response

Introduction

Working with network resources in Python often involves sending HTTP requests and handling the responses. The requests module simplifies this task, including when responses come in XML format. This tutorial demonstrates parsing XML responses using Python ‘requests’ and other related libraries.

Getting Started with ‘requests’

To begin, install the ‘requests’ module if it’s not already installed:

pip install requests

Send a basic GET request and print the response content:

import requests

url = 'http://example.com/data.xml'
response = requests.get(url)

print(response.content)

Now let’s assume the content returned is in XML format. We can start working on parsing it.

Parsing XML Responses

To parse XML, the ‘xml.etree.ElementTree’ module from the Python standard library is commonly used:

import xml.etree.ElementTree as ET

data = response.content
root = ET.fromstring(data)

print(root.tag)

You should see the root element’s tag output to the console.

Navigating XML Elements

You can iterate through the elements:

for child in root:
    print(child.tag, child.attrib)

for elem in root.iter('desired_element'):
    print(elem.text)

Replace ‘desired_element’ with the specific tag you are searching for.

Handling Namespaces

XML namespaces can complicate the parsing process. To deal with namespaces, register them upfront or use the namespace argument when calling methods like find():

ET.register_namespace('ns', 'namespace_url')

# or

root.find('ns:tag', namespaces={'ns': 'namespace_url'})

Advanced Parsing with lxml

For more advanced needs or when dealing with very large documents, consider ‘lxml’:

from lxml import etree

# parsing
parser = etree.XMLParser(recover=True)
root = etree.fromstring(data, parser=parser)

# xpath expressions
root.xpath('//namespace:tag', namespaces={'namespace': 'namespace_url'})

Remember to install lxml with pip install lxml before you try the above examples.

Converting XML to Other Formats

Convert XML to a dictionary or JSON with the ‘xmltodict’ package:

import xmltodict

json_data = xmltodict.parse(data)
print(json_data)

Install it with pip install xmltodict.

Handling HTTP Errors

Always check for HTTP errors when you make requests:

response = requests.get(url)
if response.status_code != 200:
    # Handle error
    print('Error:', response.status_code)
#elif specific error handling

Look for specific status codes and handle accordingly.

Exception Handling

Exception handling ensures that your application can handle issues such as connection errors:

try:
    response = requests.get(url)
    response.raise_for_status()
except requests.exceptions.HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')
except Exception as err:
    print(f'Other error occurred: {err}')

HTTP Auth and Headers

Pass authentication credentials and custom headers with your requests:

auth = ('username', 'password')
headers = {'header-name': 'header-value'}
response = requests.get(url, auth=auth, headers=headers)

# Parsing response XML follows the same as before

POST Requests and XML Data

Sending XML data via POST often requires setting the correct content type:

headers = {'Content-Type': 'application/xml'}
response = requests.post(url, data=xml_data.encode('utf-8'), headers=headers)

Ensure xml_data contains your XML formatted correctly.

Session Objects

Using Session objects can improve performance by reusing the underlying TCP connection:

with requests.Session() as session:
    response = session.get(url)

# Parsing XML is similar to before

Testing with Mock Responses

Use ‘requests-mock’ or ‘responses’ packages to mock HTTP requests for testing:

# Using requests-mock
import requests_mock
with requests_mock.Mocker() as m:
    m.get(url, text='data')
    # Make the request and handle XML as needed

Conclusion

Handling XML responses with the Python ‘requests’ module is a common task that’s greatly streamlined by the variety of support libraries. Whether your requirement is basic parsing or handling complex scenarios, this tutorial covered the tools to help integrate XML-based services into your Python applications efficiently.