The kafka-console-producer.sh script is a built-in Kafka command-line tool used to send messages to a Kafka topic. It connects to the Kafka cluster via the specified bootstrap servers and allows users to input messages interactively. This tool is useful for testing Kafka producers and debugging message flows.

Before proceeding with this lesson, ensure that your Kafka server is running.

Step 1: Navigate to the Kafka Directory

1. Open a terminal window.
2. Change the directory to your Kafka installation folder.
3. Navigate to the bin directory where Kafka command-line scripts are located.

Step 2: Sending a Message Using Kafka Console Producer

To send a message to a Kafka topic, we use the kafka-console-producer.sh script.

./kafka-console-producer.sh --bootstrap-server <server1>:9092,<server2>:9092 --topic my-topic

• Replace <server1> and <server2> with your actual Kafka broker addresses.
• The --topic parameter specifies the topic where messages will be sent.

Handling Non-Existing Topics

If the specified topic does not exist, Kafka will generate an error message but still allow sending messages. By default, Kafka is configured to automatically create a new topic when a message is sent to a non-existing topic. However, it is best practice to create topics manually to control their configuration, such as:

• Number of partitions
• Replication factor

Step 3: Sending Messages

Once the command is executed, the terminal waits for input. Each line represents one record, and pressing Enter sends the message. Example:

Hello, world!
Hello, world! 2
Hello, world! 3

To stop sending messages, press Ctrl + C.

Step 4: Sending Messages with Keys

Kafka messages are stored as key-value pairs. If a message is sent without a key, the key will be null. Messages with the same key are stored in the same partition, ensuring that they are read in order.

To send a message with a key, use the following command:

./kafka-console-producer.sh --bootstrap-server <server1>:9092 --topic my-topic --property parse.key=true --property key.separator=:

• --property parse.key=true enables key-value message support.
• --property key.separator=: specifies the separator between key and value.

Sending Key-Value Messages

Once the command is executed, you can send messages in the following format:

first_name:John
last_name:Smith

• The part before : is the key.
• The part after : is the value.

If a message is sent without a valid separator, Kafka will return an error:

No key separator found on line number 2: test message

This ensures that key-value formatting is enforced properly.

Best Practices

• Always create topics manually before sending messages to define necessary configurations.
• Use multiple bootstrap servers for better fault tolerance.
• Use message keys to maintain message order in a partition.