When designing a Kafka consumer application, you must decide how to handle failures.Failures can happen for two main reasons: Spring Kafka allows you to classify exceptions into two types: 1. Retryable Exceptions A retryable exception is an exception that happens due to a temporary issue that is likely to be fixed if we wait and…

1. Introduction In a Kafka-based microservices system, consumers continuously read and process messages from topics. However, sometimes the consumers can encounter problematic messages: If we don’t handle these bad messages carefully, they can crash the consumer or block it into an endless retry loop. One way to deal with such issues is by using a…

Introduction In real-world event-driven applications, message formats can change or become inconsistent, especially when multiple microservices publish to the same Kafka topic. If a Kafka consumer expects a specific message format but receives a message in an unexpected format, deserialization will fail. When deserialization fails, it causes a critical issue: In this tutorial, you will…

Introduction In this tutorial, we will build a Spring Boot microservice that acts as a Kafka consumer. The primary use case is to consume messages from a Kafka topic published by another microservice Kafka Consumer Characteristics Configuring application.properties for Kafka Consumer Now that your project is set up, the next step is to configure Kafka…

Kafka producers and brokers communicate over a network, which means there is always a risk of network failure, broker issues, or delayed acknowledgements. These issues can lead to duplicate messages in Kafka topics, especially when retries are enabled. To solve this, Kafka offers idempotent producers, which ensure each message is delivered only once, even during…

Introduction In the previous lesson, we explored how the retries property of the Kafka producer works and how retry behavior can be configured. In this lesson, we will dive into a more recommended and modern approach: using the delivery.timeout.ms configuration property instead of explicitly setting the number of retries. Kafka’s documentation encourages developers to rely…

What Is min.insync.replicas? When a Kafka producer sends a message, the broker writes the message to a partition. If that partition is replicated across multiple brokers (replication factor > 1), Kafka defines a set of “in-sync replicas” (ISRs)—these are replicas that are fully caught up with the leader. The min.insync.replicas property defines the minimum number…

In a previous lesson, you learned how Kafka producers can be configured to wait for acknowledgments from brokers to ensure the durability and reliability of messages. This is particularly useful in distributed systems where message loss is unacceptable. Now, let’s take it a step further by discussing what happens when a broker becomes unavailable, and…