Sling Academy
Home/DevOps/How to install and configure Apache Kafka on Windows

How to install and configure Apache Kafka on Windows

Last updated: January 29, 2024

Apache Kafka is a distributed streaming platform that is widely used for building real-time data pipelines and streaming apps. It is generally preferred for its high-throughput, built-in partitioning, replication, and inherent fault tolerance. Installing and configuring Kafka on a Windows machine involves several steps that can seem daunting at first, but with the right guide, it can be a straightforward process. This tutorial will help you set up and run Apache Kafka on a Windows environment, moving from the basics to more advanced configurations, along with examples.

Prerequisites

Before diving into the installation process, you must ensure your Windows machine meets the following requirements:

  • Java Runtime Environment (JRE) or Java Development Kit (JDK) version 8 or higher installed.
  • Administrator permissions are likely required to install and run the services.

Downloading and Installing Kafka

Step 1: Install Java

Since Kafka is written in Scala and runs on the Java Virtual Machine (JVM), you need to have Java installed:

java -version

If you don’t have Java installed, download it from Oracle’s website or use an open-source version like OpenJDK.

Step 2: Download Kafka

Go to the official Apache Kafka downloads page and download the latest binary for Windows.

# Example download link (always choose the latest version)
curl "http://www.apache.org/dyn/closer.cgi?path=/kafka/latest/kafka_2.12-2.8.0.tgz" -o kafka.tgz

Step 3: Install Kafka

Extract the downloaded tar file using an archiving tool like 7-Zip:

7z x kafka.tgz -oC:\\path_to_kafka
unzip kafka_2.12-2.8.0.tgz -d C:\\path_to_kafka

Starting Kafka Server

Step 1: Start the Zookeeper Server

Kafka uses Zookeeper to store metadata about the Kafka cluster and consumer client details. Start Zookeeper with the following command:

.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

Step 2: Start the Kafka Server

After Zookeeper has started, open another command prompt instance to start the Kafka server:

.\bin\windows\kafka-server-start.bat .\config\server.properties

Creating Kafka Topics

Step 1: Create a Topic

Kafka stores and transports bytes in the form of messages through a topic. Create a topic using the following:

.\bin\windows\kafka-topics.bat --create --topic myFirstTopic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

This will result in a topic named “myFirstTopic” being created with a single partition and a replication factor of one.

Producing and Consuming Messages

Step 1: Producing Messages

Send a message to the topic you created using the Kafka producer:

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic myFirstTopic
>Hello Kafka

Step 2: Consuming Messages

Open another command prompt to start the Kafka consumer and begin consuming messages from the topic:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic myFirstTopic --from-beginning
Hello Kafka

Advanced Configurations

Now that you got Kafka running, you might want to consider advanced configurations such as setting up a multi-broker cluster for fault-tolerance, optimizing performance by adjusting topic and broker configurations, and securing your Kafka cluster through access control lists (ACLs) and SSL encryption. Details on how to perform these steps fall outside the scope of this beginner guide but can be found in the Kafka documentation.

Conclusion

With this tutorial, you should now have a basic Apache Kafka setup running on your Windows machine. This guide has taken you through installing Java, downloading Kafka, and starting servers, as well as producing and consuming simple messages. The possibilities of Kafka are vast, and so is the potential to scale and secure your Kafka clusters.

Next Article: How to set up Kafka on Mac

Previous Article: An Introduction to Apache Kafka & Event Streaming

Series: Apache Kafka Tutorials

DevOps

You May Also Like

  • How to reset Ubuntu to factory settings (4 approaches)
  • Making GET requests with cURL: A practical guide (with examples)
  • Git: What is .DS_Store and should you ignore it?
  • NGINX underscores_in_headers: Explained with examples
  • How to use Jenkins CI with private GitHub repositories
  • Terraform: Understanding State and State Files (with Examples)
  • SHA1, SHA256, and SHA512 in Terraform: A Practical Guide
  • CSRF Protection in Jenkins: An In-depth Guide (with examples)
  • Terraform: How to Merge 2 Maps
  • Terraform: How to extract filename/extension from a path
  • JSON encoding/decoding in Terraform: Explained with examples
  • Sorting Lists in Terraform: A Practical Guide
  • Terraform: How to trigger a Lambda function on resource creation
  • How to use Terraform templates
  • Understanding terraform_remote_state data source: Explained with examples
  • Jenkins Authorization: A Practical Guide (with examples)
  • Solving Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap
  • Understanding Artifacts in Jenkins: A Practical Guide (with examples)
  • Using Jenkins with AWS EC2 and S3: A Practical Guide