Download Apache Kafka

To download Apache Kafka, open a web browser and navigate to the official Apache Kafka website.

On the homepage, locate the Download Kafka button in the top-right corner and click on it.

On the download page, you will find different versions of Apache Kafka available for download. The latest stable release at the time of writing is Apache Kafka 3.6.0. However, if you need an earlier version, you can find it on this page as well.

If a newer version is available when you download, you can try using it. However, if it does not work with certain commands from a tutorial or guide, you may need to download the same version used in that guide.

After the Kafka archive file has been downloaded, it needs to be extracted. Follow these steps:

1. Open a terminal or command prompt.
2. Change the directory to the Downloads folder:
   cd ~/Downloads
3. List the files in the folder to verify the Kafka archive:
   ls
4. Extract the Kafka archive using the following command:
   tar -xvzf kafka_2.13-3.6.0.tgz
5. After extraction, verify the new folder:
   ls
6. You should see a new folder named kafka_2.13-3.6.0.
7. To simplify navigation, rename the extracted folder to just kafka:
   mv kafka_2.13-3.6.0 kafka
8. Since files in the Downloads folder may be deleted over time, move Kafka to a preferred location such as the user’s home directory:
   mv kafka ~/kafka

Start Apache Kafka Server

Understanding Kafka’s Configuration Files

Now that we have downloaded Apache Kafka, we can start it and check if it works. There are two ways to start Kafka:

• Using Zookeeper (Deprecated)
• Using KRaft Mode (Recommended)

Since Zookeeper is being deprecated, we will start Apache Kafka using KRaft mode. Inside the Kafka folder, you will find a config directory, which contains a kraft subdirectory. Inside this kraft folder, there are three configuration files:

• controller.properties – Configuration for the controller managing cluster metadata and leader election.
• broker.properties – Configuration for a broker responsible for storing and serving data.
• server.properties – Configuration for a server acting as both a broker and a controller.

Prepare the Storage Directory

Before starting the Kafka server, we need to prepare the storage directories and generate a unique cluster ID.

1. Open a terminal and navigate to the bin folder inside the Kafka directory:
   cd ~/kafka/bin
2. Generate a unique cluster ID:
   ./kafka-storage.sh random-uuid
   Copy the generated UUID.
3. Format the log directories for each server using the cluster ID:
   ./kafka-storage.sh format -t -c ../config/kraft/server.properties
   Replace <CLUSTER_ID> with the generated UUID.
4. This step is necessary for running Kafka in KRaft mode, ensuring storage directories are compatible and do not contain previous Zookeeper-based data.

Start Kafka Server

To start the Kafka server, run the following command:
./kafka-server-start.sh ../config/kraft/server.properties

This will start the Kafka server with default configuration properties.

Start Multiple Kafka Servers in a Cluster

Running multiple Kafka servers (brokers) in a cluster is essential for high availability, fault tolerance, and scalability. This tutorial will guide you through setting up and starting multiple Kafka servers using KRaft mode.

Step 1: Prepare Kafka Configuration Files

Each Kafka server (broker) requires a unique configuration file. Inside the Kafka config/kraft/ directory, duplicate the server.properties file and rename them appropriately:

• server-1.properties
• server-2.properties
• server-3.properties

Update Configuration in Each File

Modify the following properties in each server-*.properties file:

Example: server-1.properties

broker.id=1
controller.quorum.voters=1@localhost:9093,2@localhost:9095,3@localhost:9097
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
log.dirs=/tmp/server-1/kraft-combined-logs

Example: server-2.properties

node.id=2
controller.quorum.voters=1@localhost:9093,2@localhost:9095,3@localhost:9097
listeners=PLAINTEXT://:9094,CONTROLLER://:9095
advertised.listeners=PLAINTEXT://localhost:9094,CONTROLLER://localhost:9095
log.dirs=/tmp/server-2/kraft-combined-logs

Example: server-3.properties

node.id=3
controller.quorum.voters=1@localhost:9093,2@localhost:9095,3@localhost:9097
listeners=PLAINTEXT://:9096,CONTROLLER://:9097
advertised.listeners=PLAINTEXT://localhost:9096,CONTROLLER://localhost:9097
log.dirs=/tmp/server-3/kraft-combined-logs

Step 2: Format Storage Directories

Before starting the servers, initialize the storage directories using the same cluster ID.

Generate a unique cluster ID:

cd ~/kafka/bin 
./kafka-storage.sh random-uuid

Copy the generated UUID.

Format the log directories for each server using the cluster ID:

./kafka-storage.sh format -t <CLUSTER_ID> -c ../config/kraft/server-1.properties
./kafka-storage.sh format -t <CLUSTER_ID> -c ../config/kraft/server-2.properties
./kafka-storage.sh format -t <CLUSTER_ID> -c ../config/kraft/server-3.properties

Replace <CLUSTER_ID> with the generated UUID.

Step 3: Start Multiple Kafka Servers

Now, start each Kafka broker in a separate terminal tab.

1. Open a terminal and navigate to the Kafka bin directory:
   cd ~/kafka/bin
2. Start the first server:
   ./kafka-server-start.sh ../config/kraft/server-1.properties
3. Open a new terminal tab and start the second server:
   ./kafka-server-start.sh ../config/kraft/server-2.properties
4. Open another terminal tab and start the third server:
   ./kafka-server-start.sh ../config/kraft/server-3.properties

Verify each server’s terminal for any startup errors.

How to gracefully stop Apache Kafka Servers

Now that you know how to start multiple Apache Kafka servers, it’s important to understand how to stop them properly. A controlled shutdown ensures that no messages are lost and prevents unnecessary errors or exceptions.

Step 1: Stop Producers and Consumers First

Before shutting down your Kafka servers, it is a good practice to stop all running producers and consumers. This helps to:

• Prevent message loss: If producers continue to send messages to offline servers, there is a chance of losing those messages.
• Avoid system errors: If producers and consumers keep interacting with a server that is shutting down, exceptions may occur.
• Reduce complexity: Exceptions might trigger error-handling or retry logic, leading to unnecessary complications in your application.

By stopping producers and consumers first, you put your system into a safe state before shutting down Kafka servers.

Step 2: Stop Kafka Servers

Quick Shutdown (Not Recommended)

A quick way to stop a Kafka server is by pressing Ctrl + C in the terminal where the server is running. However, this method is not ideal because:

• It does not allow Kafka to complete all graceful shutdown tasks.
• Log segments might not be closed properly.
• It can lead to potential issues in data consistency and network connections.

Graceful Shutdown (Recommended)

The better approach is to use the Kafka CLI script designed for stopping Kafka servers gracefully. This method ensures that:

• Shutdown hooks are executed properly.
• Network connections are properly closed.
• In-memory data is flushed and committed to disk.
• Log segments are closed, and necessary cleanup is performed.

Enabling Controlled Shutdown

By default, Kafka enables controlled shutdown through the controlled.shutdown.enable property. This property is set to true in the Kafka server.properties file. If you need to manually configure it, you can modify this setting in server.properties.

Step 3: Execute the Kafka Shutdown Command

To gracefully stop all Kafka servers, follow these steps:

1. Open a new terminal tab.
2. Navigate to the Kafka installation folder.
3. Execute the Kafka shutdown script from the bin directory.

For Linux/macOS:

bin/kafka-server-stop.sh

For Windows:

bin\windows\kafka-server-stop.bat