How to interpret Kafka logs (with 8 examples)

Introduction

Apache Kafka is a powerful distributed streaming platform for handling real-time data feeds. It’s robust and fault-tolerant, but like any complex system, it can be challenging to understand its internal workings. One of the most valuable resources for diagnosing issues and tracking system behavior in Kafka is its log output.

This tutorial will guide you through interpreting Kafka logs, from basic log reading to analyzing advanced log entries.

Understanding Kafka Logging

Kafka uses SLF4J for logging, which is an abstraction that allows selecting the concrete logging implementation at deployment time. Logs often contain vital information separated into multiple levels such as ERROR, WARN, INFO, DEBUG, and TRACE.

Example 1: Basic INFO log entry

INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

This log entry indicates that a Kafka broker with the ID 1 has successfully started.

Example 2: WARN log entry

WARN [ReplicaFetcherThread-0-1] Fetcher reached max/full bytes before completing the request (kafka.server.ReplicaFetcherThread)

This warning suggests that a replica fetcher thread has retrieved as much data as it’s configured to, indicating possible configuration issues or heavy load.

Common Kafka Log Files

There are several log files that you would typically encounter when working with Kafka:

• server.log – Contains general logs for a Kafka broker instance.
• state-change.log – Holds logs related to the change of state in partitions and replicas.
• controller.log – Captures logs by the controller – the broker responsible for managing the cluster state.

Basic Log Analysis

Let’s start by looking at server.log and identifying some easy-to-recognize patterns:

Example 3: Log showing successful partition reassignment

INFO [Partition reassignment handler for partition test-0] Reassignment deletion of partition test-0 completed successfully (kafka.server.ZkPartitionStateMachine)

This info log entry signifies that the reassignment of partition test-0 was successful.

Delving into Broker Startup Logs

When Kafka starts, the broker logs show a series of startup activities:

Example 4: Kafka broker startup logs

INFO Kafka version: 2.4.0 (org.apache.kafka.common.utils.AppInfoParser) 
INFO Kafka commitId: 77a89fcf8d7fa018 (org.apache.kafka.common.utils.AppInfoParser) 
INFO [KafkaServer id=1] Startup complete. (kafka.server.KafkaServer)

These show the version, commit ID, and the successful startup completion notice for the broker.

Deeper Analysis with DEBUG and TRACE

DEBUG and TRACE levels provide more in-depth insight. These are typically turned off in production due to their verbose nature but can be invaluable for troubleshooting.

Example 5: DEBUG log with rich details

DEBUG [RequestSendThread controllerId=1] Controller 1 epoch 5 sends request (correlationId=79) (kafka.controller.ControllerChannelManager)

These DEBUG logs can reveal interactions at the controller level, along with epochs and correlation IDs for requests.

Analyzing Errors and Exceptions

Exception logs are critical for identifying problems:

Example 6: ERROR log with stack trace

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) 
java.io.IOException: No such file or directory
	net.some.kafka.network.SocketServer
...

This log indicates a KafkaServer startup failure due to a missing file or directory.

Configuring Kafka Log Levels

Kafka allows dynamic configuration of log levels using the Kafka admin client tool.

Example 7: Changing log levels dynamically

$ kafka-configs.sh --zookeeper localhost:2181 --entity-type brokers --entity-name 1 --add-config "log4j.logger.kafka=DEBUG, kafkaAppender"

This command sets the logging level of Kafka on broker 1 to DEBUG.

Advanced Log Interpretation Techniques

For comprehensive analysis, logs can be aggregated using tools like ELK stack, Grafana, and Splunk.

Example 8: Aggregating logs with the ELK stack

o aggregate logs with the ELK stack (Elasticsearch, Logstash, Kibana), specifically for Kafka logs, you would generally use Filebeat to ship the logs to Elasticsearch, and then use Kibana for visualization. Here’s a basic example of how you can set this up:

Step 1: Configure Filebeat

First, you’ll need to configure Filebeat to send Kafka logs to Elasticsearch. Here’s an example of a Filebeat configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/kafka/logs/*.log

output.elasticsearch:
  hosts: ["localhost:9200"]

setup.kibana:
  host: "localhost:5601"

# Optional: Setup Filebeat to use Kibana dashboards
setup.dashboards.enabled: true

Here:

• Filebeat Inputs: This section defines where Filebeat will read the Kafka logs from. Adjust /path/to/kafka/logs/*.log to point to your Kafka log files.
• Output Elasticsearch: Specifies the Elasticsearch instance to which Filebeat will send the logs.
• Setup Kibana: Configures the Kibana instance for log visualization.
• Kibana Dashboards: Enables the use of default Kibana dashboards for Kafka.

Step 2: Start Filebeat

Run Filebeat to start shipping logs:

filebeat -e

Step 3: Visualize in Kibana

1. Access Kibana: Open Kibana in a web browser (typically at http://localhost:5601).
2. Create Index Pattern: Create an index pattern in Kibana for Filebeat (filebeat-* by default).
3. Explore and Visualize: Use Kibana’s Discover feature to explore Kafka logs, create visualizations, and build dashboards.

Notes:

• Make sure Elasticsearch and Kibana services are running and accessible.
• Filebeat configuration may vary based on your Kafka version and logging setup. The provided example is a basic template.
• Kibana offers powerful tools for creating custom visualizations and dashboards. You can explore various options to best represent your Kafka logs.
• If your Kafka logs are in JSON format, additional configurations may be needed in Filebeat for proper parsing.

By pushing Kafka logs into Elasticsearch via Filebeat and visualizing them with Kibana, you can identify patterns and anomalies over time.

Conclusion

Interpreting Kafka logs is crucial for the operation and maintenance of a Kafka cluster. From basic INFO logs to TRACE level logs, understanding these messages empowers a developer or system administrator to ensure that Kafka is running smoothly and to deal promptly with any issues. As log complexity increases, so does the need for robust analysis tools and the knowledge to use them effectively.