1. Introduction In a Kafka-based microservices system, consumers continuously read and process messages from topics. However, sometimes the consumers can encounter problematic messages: If we don’t handle these bad messages carefully, they can crash the consumer or block it into an endless retry loop. One way to deal with such issues is by using a…

Introduction In real-world event-driven applications, message formats can change or become inconsistent, especially when multiple microservices publish to the same Kafka topic. If a Kafka consumer expects a specific message format but receives a message in an unexpected format, deserialization will fail. When deserialization fails, it causes a critical issue: In this tutorial, you will…

Introduction In this tutorial, we will build a Spring Boot microservice that acts as a Kafka consumer. The primary use case is to consume messages from a Kafka topic published by another microservice Kafka Consumer Characteristics Configuring application.properties for Kafka Consumer Now that your project is set up, the next step is to configure Kafka…

Kafka producers and brokers communicate over a network, which means there is always a risk of network failure, broker issues, or delayed acknowledgements. These issues can lead to duplicate messages in Kafka topics, especially when retries are enabled. To solve this, Kafka offers idempotent producers, which ensure each message is delivered only once, even during…

Introduction In the previous lesson, we explored how the retries property of the Kafka producer works and how retry behavior can be configured. In this lesson, we will dive into a more recommended and modern approach: using the delivery.timeout.ms configuration property instead of explicitly setting the number of retries. Kafka’s documentation encourages developers to rely…

What Is min.insync.replicas? When a Kafka producer sends a message, the broker writes the message to a partition. If that partition is replicated across multiple brokers (replication factor > 1), Kafka defines a set of “in-sync replicas” (ISRs)—these are replicas that are fully caught up with the leader. The min.insync.replicas property defines the minimum number…

In a previous lesson, you learned how Kafka producers can be configured to wait for acknowledgments from brokers to ensure the durability and reliability of messages. This is particularly useful in distributed systems where message loss is unacceptable. Now, let’s take it a step further by discussing what happens when a broker becomes unavailable, and…

Introduction When developing applications that communicate with Apache Kafka, it’s essential to understand how message delivery works, especially when reliability and durability are important. Kafka provides a range of options that allow you to fine-tune how the producer waits for acknowledgements from the brokers. This tutorial dives deep into how acknowledgements work in Kafka and…

In this tutorial, we’ll create a Kafka producer in Spring Boot. Dependency To work with Kafka in Spring Boot, add the following dependency: Configuration Add the following configuration in the properties file, application.properties: Explaination: 2. Key Serializer 3. Value Serializer Implementation Following is the Following is the important piece of code to send event: Note:…

Introduction In this tutorial, you will learn how to use the Apache Kafka Console Consumer to read messages from a Kafka topic using the Apache Kafka command-line interface (CLI). Introduction to Kafka Console Consumer Kafka Console Consumer is a built-in tool provided by Apache Kafka that allows you to read messages from a specified Kafka…