Introduction

Apache Kafka provides a Command Line Interface (CLI) to manage and interact with topics within a Kafka cluster. While Java code can be used for complex interactions, the CLI is a quick and convenient way to perform common tasks such as listing topics, checking their status, and modifying configurations.

Kafka Topics CLI Overview

Kafka provides a script to manage topics, available for different operating systems:

• Mac/Unix: kafka-topics.sh
• Windows: kafka-topics.bat

This script allows users to perform various topic-related operations, including:

• Creating a new topic
• Listing existing topics
• Describing a topic (providing detailed information)
• Deleting a topic
• Modifying an existing topic

Prerequisites

Before creating a Kafka topic, ensure that your Kafka server is running. If you are working with a Kafka cluster, you may need to start multiple Kafka servers. Open separate terminal windows or tabs for each server and start them accordingly.

Using Kafka Topics CLI

Creating a New Topic

To create a new Kafka topic, first navigate to the Kafka installation directory and locate the bin folder where CLI scripts are stored.

On Mac/Linux, use:

cd /path/to/kafka
cd bin

On Windows, navigate to the bin\windows directory:

cd C:\path\to\kafka\bin\windows

Once inside the bin folder, you can create a new Kafka topic with the following command:

kafka-topics.sh --create --topic topic-one --bootstrap-server localhost:9092,localhost:9093 --partitions 3 --replication-factor 3

For Windows:

kafka-topics.bat --create --topic topic-one --bootstrap-server localhost:9092,localhost:9093 --partitions 3 --replication-factor 3

Explanation of Parameters:

• --topic topic-one: Specifies the topic name. It should be unique and meaningful.
• --partitions 3: Defines the number of partitions for parallelism and fault tolerance.
• --replication-factor 3: Ensures data availability by storing copies across multiple brokers.
• --bootstrap-server localhost:9092,localhost:9093: Specifies the Kafka brokers to use.

To create another topic, simply change the topic name:

kafka-topics.sh --create --topic topic-two --bootstrap-server localhost:9092,localhost:9093 --partitions 3 --replication-factor 3

Listing All Topics

To list all available Kafka topics:

kafka-topics.sh --list --bootstrap-server localhost:9092

For Windows:

kafka-topics.bat --list --bootstrap-server localhost:9092

Describing a Topic

To get details of a specific topic:

kafka-topics.sh --describe --topic topic-one --bootstrap-server localhost:9092

For Windows:

kafka-topics.bat --describe --topic topic-one --bootstrap-server localhost:9092

Understanding the Output of describe Command:

• Topic Name: The name of the Kafka topic.
• Topic ID: A unique identifier assigned to the topic.
• Partitions: The number of partitions the topic has.
• Replication Factor: The number of copies stored across brokers.
• Leader: The broker responsible for handling read/write operations for a partition.
• Replicas: Copies of a partition stored on other brokers.
• In-Sync Replicas (ISR): Replicas that are up to date with the leader and can take over if needed.
• Segment Size: The maximum size of a log segment before a new one is created.

Deleting a Topic

Deleting a Kafka topic is a permanent action and cannot be undone unless you have a backup. Before proceeding, ensure that the delete.topic.enable property in the Kafka server configuration is set to true (it is true by default).

To delete a topic:.

kafka-topics.sh --delete --topic topic-one --bootstrap-server localhost:9092

For Windows:

kafka-topics.bat --delete --topic topic-one --bootstrap-server localhost:9092

To verify the deletion, list the topics again:

kafka-topics.sh --list --bootstrap-server localhost:9092

If the topic was successfully deleted, it will not appear in the list.