Python ‘requests’ module: How to disable log messages

Overview

Dealing with noisy log messages from the Python ‘requests’ module can be distracting. This tutorial outlines how to manage and disable these unwanted logs.

Introduction to Logging

In any software application, logging provides insights into what’s happening internally. The Python ‘requests’ module comes with its built-in logging for various events. However, sometimes these logs can clutter your console output or log files, especially when you are making numerous HTTP requests and only some of them or none are relevant for your logging purposes.

To begin, let’s make sure we understand the basics. When using the ‘requests’ library to make HTTP calls, it uses the python logging library to log messages like connection information and the details of any requests made.

import requests

response = requests.get('https://api.example.com/data')
print(response.content)

The above code performs a simple GET request. Depending on your logging configuration, executing this script might or might not show logs.

Configuring Python Logging

Before we attempt to suppress the logging from the ‘requests’ library, it’s essential to have a basic understanding of how Python’s logging module works.

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('example_logger')
logger.info('This is an info message')

The ‘basicConfig’ method configures the logging to output messages of level INFO and above. Now, let’s proceed to how ‘requests’ uses logging.

Understanding ‘requests’ Logging

The ‘requests’ module logs information using different log levels (DEBUG, INFO, WARNING, ERROR, and CRITICAL). By default, the log level is set to WARNING, which means INFO and DEBUG messages are not shown.

When you wish to see detailed information about the HTTP transactions, you can set the log level of the ‘requests’ library to DEBUG.

import logging

# Configure logging
logging.basicConfig(level=logging.DEBUG)

# Make an HTTP request
response = requests.get('https://api.example.com/data')
# This will print a lot of debug information from 'requests'.

However, this method prints everything, which might not always be desirable.

Disabling Logging for ‘requests’

Setting the Log Level

One of the easiest ways to disable ‘requests’ logging is to increase the log level.

import logging
import requests

# Set log level to ERROR to suppress INFO and DEBUG logs
logging.getLogger('requests').setLevel(logging.ERROR)

response = requests.get('https://api.example.com/data')

By setting the ‘requests’ logger level to ERROR, you eliminate most of the logs produced by successful HTTP requests.

Completely Silencing the Logger

In case you want to completely silence the ‘requests’ logging, you can use logging’s `NullHandler`.

import logging
import requests

logging.getLogger('requests').addHandler(logging.NullHandler())

response = requests.get('https://api.example.com/data')

This means that ‘requests’ will not output any logs at all. Even ERROR or CRITICAL messages will be suppressed, so use this method with caution.

More Granular Control

Filtering Using a Logging Filter

Suppose you want to filter out log messages based on certain conditions. You can define custom filters to do so.

import logging
import requests

class NoParsingFilter(logging.Filter):
    def filter(self, record):
        return not record.getMessage().startswith('Starting new HTTP')

logger = logging.getLogger('requests.packages.urllib3')
logger.addFilter(NoParsingFilter())

response = requests.get('https://api.example.com/data')

This custom logging filter will ignore ‘Starting new HTTP connection’ messages from ‘urllib3’, which is the library ‘requests’ uses to make HTTP connections.

Adjust Logging at Runtime

Sometimes you may need to adjust logging levels dynamically, during runtime. You can use the logging configuration to do this as well.

import logging
import requests

# Function to disable requests logging
def disable_requests_logging():
    logging.getLogger('requests').setLevel(logging.ERROR)

# Function to enable requests logging
def enable_requests_logging():
    logging.getLogger('requests').setLevel(logging.DEBUG)

# Disable 'requests' logging temporarily
disable_requests_logging()
response = requests.get('https://api.example.com/data')
# Enable 'requests' logging back
enable_requests_logging()

You can use these functions to switch the logging on and off as per the requirements of different parts of your code.

Managing Loggers Hierarchically

‘requests’ depends on ‘urllib3’ for its operations, and ‘urllib3’ comes with its own loggers. Disabling ‘requests’ logging does not necessarily disable ‘urllib3’ logs. You’ve to do it explicitly as shown:

logging.getLogger('urllib3').setLevel(logging.ERROR)

There may be multiple loggers in a hierarchy, and understanding this structure is crucial in complex applications.

Logging to Different Destinations

If your application requires logging to a file or any other destination while suppressing console output, this is achievable through logging configurations that handle different handlers.

import logging
from logging.handlers import RotatingFileHandler

# Configure file handler
file_handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1)
file_handler.setLevel(logging.ERROR)

logging.getLogger('').addHandler(file_handler)

# Set 'requests' logger to ERROR, but keep file logging
logging.getLogger('requests').setLevel(logging.ERROR)

By adding a file handler and setting its level, you can decide what gets logged to the file versus what appears in the console.

Conclusion

Effectively managing log messages produced by the ‘requests’ module can streamline the development process and clear up valuable console space. Remember to adjust logging levels cautiously, as suppressing error messages might hinder debugging. With the techniques illustrated in this tutorial, you’re now equipped to control ‘requests’ logging to meet the needs of your application.