ZooKeeper-less Kafka: Using Kraft and Raft Metadata Mode

Introduction

Apache Kafka has been traditionally known to rely on ZooKeeper for distributed coordination and metadata storage. However, with the introduction of KIP-500, Kafka aims to replace ZooKeeper with a self-managed metadata quorum built using its internal Raft implementation, thereby simplifying the Kafka stack. This process of running Kafka without ZooKeeper is known as using Kafka Raft, or Kraft.

In this tutorial, we will explore how to setup and work with ZooKeeper-less Kafka, using the new KRaft mode. This mode leverages Raft protocol for consensus and comes with several benefits including simplified operations, reduced latencies, and better scalability.

Prerequisites

• Apache Kafka 2.8 or higher
• Java Runtime Environment 11 or higher
• Basic understanding of Kafka and distributed systems

Setting Up Kafka in KRaft Mode

First, we need to set up a Kafka cluster using KRaft mode. Here is how you can start a single-node Kafka cluster without depending on ZooKeeper:

# Step 1: Download and extract Kafka
wget https://archive.apache.org/dist/kafka/{kafka_version}/kafka_{scala_version}-{kafka_version}.tgz
tar -xzf kafka_{scala_version}-{kafka_version}.tgz
cd kafka_{scala_version}-{kafka_version}

# Step 2: Create a config file for KRaft mode
# Let's name it 'kraft.properties'
# A sample configuration may contain:
cat > kraft.properties <<- 'EOF'
node.id=1
process.roles=broker,controller
listeners=PLAINTEXT://:9092
controller.quorum.voters=1@:9093
EOF

# Step 3: Initialize the cluster storage
bin/kafka-storage.sh format --config kraft.properties --cluster-id $(bin/kafka-storage.sh random-uuid)

# Step 4: Start the Kafka server
bin/kafka-server-start.sh kraft.properties

This will start a single Kafka broker and controller that does not depend on ZooKeeper.

Broker Configuration for Multi-Node Clusters

For larger clusters, you will designate certain nodes as brokers and others as both brokers and controllers. You must indicate these roles in their respective properties files and assign unique node ids to each.

Additionally, in multi-broker setups, you should list down all voter IDs in the controller.quorum.voters field, separated by commas:

node.id=2
process.roles=broker
listeners=PLAINTEXT://:9092
controller.quorum.voters=1@:9093,2@:9094,3@:9095

It’s essential to start each Kafka node with its respective configuration file. The Kafka nodes with both broker and controller roles will participate in the metadata quorum, thus helping in managing the overall cluster metadata.

Creating and Managing Topics

Now that we have our Kafka cluster running in KRaft mode, you can proceed to create topics. Here is how you can create a topic named example-topic with a single partition and a replica:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic example-topic --partitions 1 --replica 1

Similarly, you can list, describe, and delete topics using the kafka-topics.sh tool:

# Listing topics
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Describing a topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic example-topic

# Deleting a topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic example-topic

Advanced Configuration

To ensure high availability and resilience, you may configure multiple controllers in your Kafka cluster. For example, having three controllers would mean a setup where at least two need to be operational to maintain quorum:

controller.quorum.voters=1@:9093,2@:9094,3@:9095

# In this configuration, each controller's listener port must be unique and match the port specified in the controller.quorum.voters part.

Beyond that, security features such as TLS and SASL can be similarly configured as they would be in a Kafka cluster with ZooKeeper. For instance, to enable SSL/TLS for internal communication, you can specify the following settings in your kraft.properties file:

listeners=SSL://:9092
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=yourkeystorepassword
ssl.key.password=yourkeypassword
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=yourtruststorepassword

Monitoring Kafka in KRaft Mode

Once the Kafka cluster is up and running, monitoring plays a crucial role. Kafka ships with various tools and JMX metrics that you can use to monitor its health and performance. The techniques and the tools needed to monitor Kafka remain much the same in KRaft mode, including the use of Kafka’s own internal tools like kafka-consumer-groups.sh for consumer group management, and external tools such as Prometheus and Grafana for a more graphical representation of metrics.

Step-by-Step Example

Step 1: Set Up JMX Metrics for Kafka

Enable JMX in Kafka’s configuration by setting properties in the server.properties file:

JMX_PORT=9999
KAFKA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Step 2: Configure Prometheus JMX Exporter

Download the JMX Exporter jar file and create a configuration file (kafka.yml) that defines the JMX metrics to expose to Prometheus.

Step 3: Run Kafka with JMX Exporter

Start Kafka with the JMX Exporter by adding it to the Kafka’s classpath and specifying the configuration file:

KAFKA_OPTS="$KAFKA_OPTS -javaagent:/path/to/jmx_prometheus_javaagent.jar=8080:/path/to/kafka.yml"
kafka-server-start.sh config/server.properties

Step 4: Configure Prometheus

Add the Kafka broker with the JMX Exporter port to the Prometheus configuration file (prometheus.yml):

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['kafka-broker-host:8080']

Step 5: Set Up Grafana

Install and start Grafana, then configure it to use Prometheus as the data source.

Step 6: Create Dashboards in Grafana

Create dashboards in Grafana to visualize Kafka metrics.

Step 7: Using Kafka’s Internal Tools

Use Kafka’s internal tools like kafka-consumer-groups.sh for consumer group management:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-consumer-group

Step 8: Monitor and Alert

Set up alerting in Grafana based on the metrics to monitor the Kafka cluster.

Conclusion

Transitioning to using Kafka without ZooKeeper reflects a major architectural simplification and consolidation effort by the Kafka community. Utilizing KRaft mode reduces operational complexity, eliminates the performance overhead caused by ZooKeeper, and opens a path for new capabilities within the Kafka ecosystem. By mastering KRaft mode, developers and administrators can better leverage the strengths of Kafka as a distributed system.