Introduction
Apache Kafka is a distributed streaming platform that is widely used for building real-time messaging systems. It’s known for its high throughput, reliability, and replication capabilities, making it a favorable choice for various applications. Kafka runs on predefined default ports, but you might encounter situations where you wish to run Kafka services on custom ports (e.g. due to port conflicts or specific network requirements). In this tutorial, we will walk you through the process of configuring and running Kafka services on custom ports.
Prerequisites
To get the most out of this guide, you should have the following:
- Apache Kafka installation
- Basic understanding of Kafka concepts such as topics, brokers, producers, and consumers
- Familiarity with command-line tools
Understanding Default Kafka Ports
Before diving into custom configurations, you should identify the default ports that Kafka utilizes:
- Broker: 9092
- ZooKeeper: 2181
Modifying Kafka Configuration for Custom Ports
To change the default ports, we need to modify the Kafka server configuration file, typically located at config/server.properties. Open this file in your favorite text editor to proceed.
# Set the broker listening port to a custom value
listeners=PLAINTEXT://:9093
This example sets the Kafka broker to listen on port 9093 instead of the default 9092. For ZooKeeper, you’ll need to adjust the configuration in config/zookeeper.properties, changing the clientPort setting.
# Set ZooKeeper client port to a custom value
clientPort=2182
Starting Kafka Services on Custom Ports
After updating the configuration files, you need to start the ZooKeeper service and the Kafka server with the new settings. Use the following commands:
# Start ZooKeeper service
bin/zookeeper-server-start.sh config/zookeeper.properties
# Start Kafka broker
bin/kafka-server-start.sh config/server.properties
Once both services are running, you can confirm they are operating on the custom ports by checking the logs or by using tools like lsof or netstat.
Connecting Producers and Consumers to Custom Ports
Producers and consumers must be configured to connect to the Kafka broker’s new port. For a producer, the configuration would look something like this:
# Configure and start a producer that connects to the custom port
bin/kafka-console-producer.sh --broker-list localhost:9093 --topic some-topic
Consumers need a similar change:
# Configure and start a consumer that connects to the custom port
bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic some-topic --from-beginning
Configuring Multiple Kafka Brokers with Custom Ports
For a Kafka cluster with multiple brokers, you need to assign a unique port to each broker. Each broker’s configuration file should have a distinct broker.id and a corresponding listeners configuration.
Here is an example of two brokers on the same machine:
# Broker 1
broker.id=1
listeners=PLAINTEXT://:9093
# Broker 2
broker.id=2
listeners=PLAINTEXT://:9094
Each of these configuration files should be passed to the respective broker on startup.
Advanced Configuration: Setting up Listeners for Multiple Network Interfaces
Sometimes you might need to set up Kafka to listen on different interfaces or IP addresses. This can be done by specifying multiple listeners in the server.properties file:
# Set up Kafka to listen on multiple interfaces
listeners=PLAINTEXT://your.internal.ip:9092,PLAINTEXT://your.external.ip:9093
When doing so, make sure to also configure the advertised.listeners property to help Kafka brokers advertise correct addresses to producers and consumers.
Configuring Kafka Connect with Custom Ports
If you’re using Kafka Connect, changing the port configuration is relatively straightforward as well. Usually, Kafka Connect’s REST API port (default 8083) can be set with:
# Set Kafka Connect to run on a custom port
rest.port=8084
Add this line to your Kafka Connect worker configuration file before starting Kafka Connect.
Troubleshooting
When you set Kafka to run on custom ports, there are several common issues you might run into such as port conflicts, firewall restrictions or misconfigured advertised listeners. Always ensure that the ports are correctly opened in your firewall and valid configurations are applied.
Conclusion
Running Kafka on custom ports is straightforward and can be done by configuring the properties of brokers, ZooKeeper, and other services within your Kafka deployment. Custom port configurations can help address various network and deployment constraints, making Kafka a versatile choice for distributed streaming applications.
