Learnitweb

Introduction to Kafka Consumer CLI

Introduction

In this tutorial, you will learn how to use the Apache Kafka Console Consumer to read messages from a Kafka topic using the Apache Kafka command-line interface (CLI).

Introduction to Kafka Console Consumer

Kafka Console Consumer is a built-in tool provided by Apache Kafka that allows you to read messages from a specified Kafka topic and display them in your terminal. This is a useful way to inspect messages without writing any Java or application code.

Why Use Kafka Console Consumer?

  • It provides a quick and easy way to check messages in a topic.
  • It helps in debugging by displaying messages directly in the terminal.
  • You can choose to read only new messages or retrieve all messages from the beginning of the topic.
  • It supports consuming key-value pair messages with additional parameters.
  • It ensures message ordering within a partition if messages have the same key.

Understanding Message Ordering in Kafka

Kafka topics are divided into multiple partitions. When a producer sends a message, Kafka determines the partition using a hash of the message key. If a key is not provided, Kafka distributes messages across partitions using a round-robin algorithm.

Ensuring Message Order

  • Messages with the same key will always go to the same partition and be read in order.
  • Messages without a key will be distributed randomly across partitions, making their read order unpredictable.

To maintain message order, always use the same key when sending related messages.

Using Kafka Console Consumer

To start consuming messages from a Kafka topic, follow these steps:

Step 1: Open a Terminal and Navigate to Kafka Directory

Ensure that your Kafka server is running before using the console consumer. Open a terminal window and navigate to the Kafka installation directory:

cd /path/to/kafka

Once inside the Kafka directory, navigate to the bin folder where Kafka CLI scripts are located:

cd bin

Step 2: Run the Console Consumer Script

Use the following command to consume messages from a topic:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

Replace my-topic with the actual name of the Kafka topic you want to read messages from.

Step 3: Read Messages from the Beginning

If you want to read all messages from the beginning of the topic, add the --from-beginning flag:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

This command will display all past and new messages sent to the topic.

Step 4: Reading Only New Messages

If you only want to read new messages without retrieving old ones, do not use the --from-beginning flag. Use the following command:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

This will connect to the Kafka cluster and wait for new messages to arrive. Any previous messages stored in the topic will not be displayed.

Step 5: Running Multiple Consumers

Kafka allows multiple consumers to read from the same topic. Open a new terminal window and run another consumer script:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic

Each consumer will receive messages independently and will not interfere with each other.

Step 6: Producing Messages to the Topic

To see how consumers react to new messages, start a producer in another terminal window:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic

Type a message and hit Enter. The message will be immediately consumed and displayed by all running consumers.

Step 7: Consuming Key-Value Pair Messages

Kafka allows sending messages as key-value pairs. To enable key-value pair support on the producer, use the following command:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic --property "parse.key=true" --property "key.separator=:"

This command sets parse.key=true to enable key-value pairs and key.separator=: to define the separator between the key and value.

Example:

userId:1
first_name:John
last_name:Doe

By default, consumers display only the value part of the message. To display both key and value, add the following property to the consumer command:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --property "print.key=true"

If you want to print both key and value explicitly, you can also add:

--property "print.value=true"

If you only want to see the keys, you can set print.value=false.

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --property "print.key=true" --property "print.value=false"

Step 8: Observing Consumer Behavior

If a consumer script is running and waiting for new messages, you can send messages from the producer script, and the consumer will immediately display them. Kafka retains messages even after they are consumed, allowing multiple consumers to read the same messages.

Step 9: Verifying Message Ordering

To verify that messages with the same key are read in order, send multiple messages with the same key:

kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-topic --property "parse.key=true" --property "key.separator=:"

Now, send messages:

1:first message
1:second message
1:third message

Then, start a consumer and observe the order:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --property "print.key=true" --from-beginning

You will see:

1:first message
1:second message
1:third message

However, if messages have different keys, their order may vary across partitions.

Real-World Applications

In real-world applications, Kafka consumers are not just CLI scripts but can be Spring Boot microservices subscribed to Kafka topics. When a new message arrives, these microservices will receive and process the message accordingly.