Introduction
Working with network resources in Python often involves sending HTTP requests and handling the responses. The requests module simplifies this task, including when responses come in XML format. This tutorial demonstrates parsing XML responses using Python ‘requests’ and other related libraries.
Getting Started with ‘requests’
To begin, install the ‘requests’ module if it’s not already installed:
pip install requests
Send a basic GET request and print the response content:
import requests
url = 'http://example.com/data.xml'
response = requests.get(url)
print(response.content)
Now let’s assume the content returned is in XML format. We can start working on parsing it.
Parsing XML Responses
To parse XML, the ‘xml.etree.ElementTree’ module from the Python standard library is commonly used:
import xml.etree.ElementTree as ET
data = response.content
root = ET.fromstring(data)
print(root.tag)
You should see the root element’s tag output to the console.
You can iterate through the elements:
for child in root:
print(child.tag, child.attrib)
for elem in root.iter('desired_element'):
print(elem.text)
Replace ‘desired_element’ with the specific tag you are searching for.
Handling Namespaces
XML namespaces can complicate the parsing process. To deal with namespaces, register them upfront or use the namespace argument when calling methods like find():
ET.register_namespace('ns', 'namespace_url')
# or
root.find('ns:tag', namespaces={'ns': 'namespace_url'})
Advanced Parsing with lxml
For more advanced needs or when dealing with very large documents, consider ‘lxml’:
from lxml import etree
# parsing
parser = etree.XMLParser(recover=True)
root = etree.fromstring(data, parser=parser)
# xpath expressions
root.xpath('//namespace:tag', namespaces={'namespace': 'namespace_url'})
Remember to install lxml with pip install lxml before you try the above examples.
Converting XML to Other Formats
Convert XML to a dictionary or JSON with the ‘xmltodict’ package:
import xmltodict
json_data = xmltodict.parse(data)
print(json_data)
Install it with pip install xmltodict.
Handling HTTP Errors
Always check for HTTP errors when you make requests:
response = requests.get(url)
if response.status_code != 200:
# Handle error
print('Error:', response.status_code)
#elif specific error handling
Look for specific status codes and handle accordingly.
Exception Handling
Exception handling ensures that your application can handle issues such as connection errors:
try:
response = requests.get(url)
response.raise_for_status()
except requests.exceptions.HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
HTTP Auth and Headers
Pass authentication credentials and custom headers with your requests:
auth = ('username', 'password')
headers = {'header-name': 'header-value'}
response = requests.get(url, auth=auth, headers=headers)
# Parsing response XML follows the same as before
POST Requests and XML Data
Sending XML data via POST often requires setting the correct content type:
headers = {'Content-Type': 'application/xml'}
response = requests.post(url, data=xml_data.encode('utf-8'), headers=headers)
Ensure xml_data contains your XML formatted correctly.
Session Objects
Using Session objects can improve performance by reusing the underlying TCP connection:
with requests.Session() as session:
response = session.get(url)
# Parsing XML is similar to before
Testing with Mock Responses
Use ‘requests-mock’ or ‘responses’ packages to mock HTTP requests for testing:
# Using requests-mock
import requests_mock
with requests_mock.Mocker() as m:
m.get(url, text='data')
# Make the request and handle XML as needed
Conclusion
Handling XML responses with the Python ‘requests’ module is a common task that’s greatly streamlined by the variety of support libraries. Whether your requirement is basic parsing or handling complex scenarios, this tutorial covered the tools to help integrate XML-based services into your Python applications efficiently.
