How to create multiple brokers in Apache Kafka

Production setups of Kafka must have multiple brokers for redundancy. That increases fault tolerance. That ensures the system uptime even numerous brokers go offline.
Here are the steps to create multiple Kafka brokers.

1. Create copies of config/server.properties files that match the number of brokers.

I create three brokers, so I need three copies.
They are server_0.properties, server_1.properties, and server_2.properties

2. Assign the broker.id, listeners, and log.dirs unique values in each file.

The listeners must have unique port numbers.

server.properties files

server_0.properties
broker.id=0
listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka_0-logs

server_1.properties
broker.id=1
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kafka_1-logs

server_2.properties
broker.id=2
listeners=PLAINTEXT://localhost:9094
log.dirs=/tmp/kafka_2-logs

3. Start the zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

4. Start the brokers

Here I have to start all three brokers individually.

bin/kafka-server-start.sh config/server_0.properties
bin/kafka-server-start.sh config/server_1.properties
bin/kafka-server-start.sh config/server_2.properties

5. Create a Topic.

Now I need a topic to publish messages. Here is how to create a topic in Kafka.

bin/kafka-topics.sh --create --topic TEST-TOPIC --bootstrap-server localhost:9092,localhost:9093,localhost:9094

Note that I need this topic created across all the brokers. Hence I provide the hosts for all three brokers.

6. List all the topics in a broker.

Now I can verify if the topic I created exists in all the brokers.

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

7. Create a producer to publish messages on the topics.

I create a single producer that publishes messages to all three brokers

bin/kafka-console-producer.sh --topic TEST-TOPIC --broker-list localhost:9092,localhost:9093,localhost:9094

8. Verify if each broker in the cluster receives the messages published to the topic.

The below command creates consumers for each broker I made in the cluster. Repeat it for all the brokers in separate terminal windows. That way, I know that each broker receives the messages published on the topic.

bin/kafka-console-consumer.sh --topic TEST-TOPIC --bootstrap-server localhost:9092

I can listen to all the brokers at once on the fourth window.

bin/kafka-console-consumer.sh --topic TEST-TOPIC --bootstrap-server localhost:9092,localhost:9093,localhost:9094