Pandas: Checking if a DataFrame contains only numeric data (4 ways)

Introduction

When working with Pandas DataFrames in Python, it’s a common requirement to verify if the DataFrame contains only numeric data. This could be pivotal in data preprocessing, feature selection, or when performing statistical analyses, where numeric data is a prerequisite. In this tutorial, we’ll explore four efficient methods to check if a DataFrame consists exclusively of numeric data. We’ll cover practical examples and some pitfalls to watch out for.

Using dtypes

The dtypes attribute of a DataFrame returns the data type of each column. You can use this feature to determine if all columns are of numeric types (int64, float64, etc.).

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, 3], 'B': [4.5, 5.5, 6.5], 'C': ['text', 7, 8]}
df = pd.DataFrame(data)

# Check if all columns are numeric
df_numeric = df.select_dtypes(include=["number"])

# Verify if the DataFrame is fully numeric
is_numeric_df = df.shape[1] == df_numeric.shape[1]
print(is_numeric_df)

This method entails comparing the number of columns in the original DataFrame with the number of columns returned by select_dtypes(include=["number"]). If the numbers match, then all columns are numeric.

Checking Each Column Individually

Another approach is to iterate through each column and check if its data type is numeric. This can be done using the pd.api.types.is_numeric_dtype function.

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, 3], 'B': [4.5, 5.5, 6.5], 'C': ['text', 7, 8]}
df = pd.DataFrame(data)

# Function to check if all columns are numeric
def all_numeric(df):
    for column in df.columns:
        if not pd.api.types.is_numeric_dtype(df[column]):
            return False
    return True

print(all_numeric(df))

This method gives you more control and allows you to identify which specific column is not numeric, should there be any.

Utilizing the _get_numeric_data() Method

The _get_numeric_data() method can be used as a shortcut to select all numeric columns from a DataFrame. Here’s how:

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, 3], 'B': [4.5, 5.5, 6.5], 'C': ['text', 7, 8]}
df = pd.DataFrame(data)

# Get only the numeric columns
df_numeric = df._get_numeric_data()

# Check if all columns in the original DataFrame are numeric
count_numeric = df_numeric.shape[1]
is_all_numeric = count_numeric == df.shape[1]
print(is_all_numeric)

This approach is somewhat similar to the first method but utilizes a different technique to filter numeric data directly.

Applying the applymap() Function with a Custom Check

For a more fine-grained analysis, you can use the applymap() function, which applies a function to each element of the DataFrame. By combining this with a custom function that tests for numeracy, you can check if all data in the DataFrame are numeric.

import pandas as pd

# Sample DataFrame
data = {'A': [1, 2, 3], 'B': [4.5, 5.5, 6.5], 'C': ['text', 7, 8]}
df = pd.DataFrame(data)

# Define a function to check for numeric
def is_numeric(val):
    try:
        float(val)
    except ValueError:
        return False
    return True

This custom function attempts to convert a value to float, catching a ValueError if the conversion fails, which implies the value is not numeric. You can then apply this function to the DataFrame:

# Apply 'is_numeric' to the DataFrame elements and verify if all are True
all_numeric = df.applymap(is_numeric).all().all()
print(all_numeric)

This thorough method ensures that not just the data type is numeric but also checks the data values are indeed interpretable as numbers.

Conclusion

Checking if a DataFrame is exclusively composed of numeric data can be crucial for various data analysis tasks in Python. By employing one of the four methods highlighted in this tutorial—whether it’s using the dtypes property, iterating each column, utilizing _get_numeric_data(), or the applymap function with a custom numeric validation—you can confidently assess the data type composition of your DataFrame. Each method has its use case, depending on the level of granularity and control you need over the verification process.