Introduction to Apache Kafka Topics
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation and used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. At the heart of Apache Kafka are topics — the categories or feeds to which records are published.
In this tutorial, we’ll take a deep dive into Kafka topics. We’ll learn how to create and manage topics, from basic to advanced usages, and understand the configuration options available. We will make use of the Kafka Command Line Interface (CLI) for demonstrations.
Prerequisites
Before getting into this tutorial, you should have:
- Apache Kafka installed and running
- Basic understanding of Kafka concepts
- Access to the command line
Creating Kafka Topics
Creating a new topic in Kafka can be done using the Kafka CLI command kafka-topics. Here’s the basic syntax:
bin/kafka-topics.sh --create --bootstrap-server <host>:<port> --replication-factor <number> --partitions <number> --topic <topic_name>
Example: Creating a simple topic named ‘test_topic’ with a single partition and a replication factor of one.
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test_topic
The output will confirm successful creation:
Created topic "test_topic".
Listing Kafka Topics
You can list all the Kafka topics using the kafka-topics command as well:
bin/kafka-topics.sh --list --bootstrap-server <host>:<port>
For example:
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
The output will display a list of all topics:
test_topic
other_topic
...Describing Kafka Topics
To get detailed information about a specific topic, use the kafka-topics command with the --describe flag:
bin/kafka-topics.sh --describe --bootstrap-server <host>:<port> --topic <topic_name>
This command is particularly useful to understand partition distribution and replication factors. For example:
bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test_topic
The output might look like this:
Topic:test_topic TopicId:asdf1234 Partitions:1 ReplicationFactor:1 Configs:
Topic: test_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Altering Topics
To modify existing topics, the Kafka CLI provides the --alter flag. This allows you to change configurations like the number of partitions.
Example: Increasing the number of partitions for ‘test_topic’ to 3.
bin/kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic test_topic --partitions 3
The output will signify that the topic has been altered:
The number of partitions for topic "test_topic" has been increased from 1 to 3.
Deleting Topics
Topics can be deleted in Kafka using the kafka-topics command with the --delete flag:
bin/kafka-topics.sh --delete --bootstrap-server <host>:<port> --topic <topic_name>
Executing the command will delete the topic:
bin/kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic test_topic
And the output:
Topic test_topic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
Conclusion
In this guide, we have explored how to create, list, describe, alter, and delete Kafka topics using the Kafka CLI. Understanding how to effectively manage Kafka topics is crucial for maintaining a healthy and efficient streaming platform.
