Sling Academy
Home/DevOps/How to Add and Manage Brokers in Kafka (with Examples)

How to Add and Manage Brokers in Kafka (with Examples)

Last updated: January 31, 2024

Introduction

Apache Kafka is a distributed stream processing system which is widely used for handling real-time data feeds. As businesses grow and data demands increase, the ability to scale and manage Kafka brokers becomes essential. This tutorial will guide you through the basic to advanced steps required to add and manage brokers within a Kafka cluster.

What are Kafka Brokers?

Before delving into the management of brokers, it’s important to understand what a broker is. In Kafka, a broker is a server that stores data and serves clients. A Kafka cluster consists of one or more brokers to ensure fault tolerance and high availability.

Let’s start with some prerequisites needed before adding or managing Kafka brokers:

  • Java Runtime Environment (JRE) or Java Development Kit (JDK) installed.
  • Apache Kafka downloaded and extracted.
  • A basic understanding of Kafka architecture and concepts.

Setting Up Your First Broker

To set up a Kafka broker, you first need to configure the Kafka server properties file. This file is located in the config directory of your Kafka installation. For the first broker, you can use the default configuration.

# Start the ZooKeeper service
bin/zookeeper-server-start.sh config/zookeeper.properties

# Start the Kafka broker
bin/kafka-server-start.sh config/server.properties

This will start a single Kafka broker along with ZooKeeper, which manages broker coordination.

Adding a New Broker to the Cluster

To add a new broker to an existing Kafka cluster, you need to create a new server.properties file for the new broker and configure the following properties:

broker.id=2
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-2

It is important that the broker.id is unique within the cluster. Additionally, you need to set a separate log.dirs path and ensure that the listeners port does not conflict with existing brokers.

Start the new broker by running:

bin/kafka-server-start.sh config/server-2.properties

Expanding the Cluster

Expanding your Kafka cluster involves adding more brokers. You can repeat the above process for each new broker you wish to add. A guide on how to replicate topics and partitions among the new brokers for scalability and fault tolerance may be required, which is beyond the scope of this tutorial.

Monitoring Kafka Brokers

Once your brokers are up and running, monitoring is crucial. Kafka comes with several inbuilt monitoring capabilities which can be accessed by JMX (Java Management Extensions). You can use the following command to enable JMX when starting a broker.

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" bin/kafka-server-start.sh config/server.properties

There are also various community and commercial tools available for Kafka monitoring, such as Prometheus and Grafana, which provide visual dashboards to monitor your Kafka cluster health.

Removing Brokers from the Cluster

As with any dynamic system, sometimes you may need to remove brokers from the cluster. This could be for maintenance, scaling down, or decommissioning a server.

To remove a broker, you need to ensure that no partitions use the broker as their leader, which requires reassigning partitions through Kafka’s partition reassignment tool.

bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092
 --reassignment-json-file reassign.json --execute

Where reassign.json contains the reassignment plan. Ensure no significant impact on your system’s performance or availability before, during, and after removing a broker.

Conclusion

Managing brokers within a Kafka cluster is a key aspect of maintaining a scalable, reliable messaging system. By following the processes outlined in this tutorial, you should now have a good grasp of how to add new brokers, monitor them efficiently, and gracefully remove them if necessary. As your Kafka ecosystem grows, leveraging these skills will be fundamental to achieving robust data systems design and operation.

Next Article: Handling Partition Replication in Kafka: A Practical Guide

Previous Article: How to Expand a Kafka Cluster (with Examples)

Series: Apache Kafka Tutorials

DevOps

You May Also Like

  • How to reset Ubuntu to factory settings (4 approaches)
  • Making GET requests with cURL: A practical guide (with examples)
  • Git: What is .DS_Store and should you ignore it?
  • NGINX underscores_in_headers: Explained with examples
  • How to use Jenkins CI with private GitHub repositories
  • Terraform: Understanding State and State Files (with Examples)
  • SHA1, SHA256, and SHA512 in Terraform: A Practical Guide
  • CSRF Protection in Jenkins: An In-depth Guide (with examples)
  • Terraform: How to Merge 2 Maps
  • Terraform: How to extract filename/extension from a path
  • JSON encoding/decoding in Terraform: Explained with examples
  • Sorting Lists in Terraform: A Practical Guide
  • Terraform: How to trigger a Lambda function on resource creation
  • How to use Terraform templates
  • Understanding terraform_remote_state data source: Explained with examples
  • Jenkins Authorization: A Practical Guide (with examples)
  • Solving Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap
  • Understanding Artifacts in Jenkins: A Practical Guide (with examples)
  • Using Jenkins with AWS EC2 and S3: A Practical Guide