Category: Spring-boot-apache-kafka
-
Enabling Kafka Transactions in a Spring Boot Microservice
1. Introduction Kafka transactions ensure that either all Kafka operations within a transaction succeed, or none of them are applied. This guarantees: Kafka achieves this by tagging producers with transactional IDs, enabling Kafka to keep track of the state of transactions even across application crashes or restarts. 2. Enable Transactions via application.properties To enable Kafka…
-
Introduction to Transactions in Apache Kafka with Spring Boot
In this tutorial, we will explore Kafka Transactions—what they are, why they’re needed, and how to enable them in a Spring Boot Kafka microservices architecture. Transactions in Kafka are vital for ensuring data consistency and achieving exactly-once semantics (EOS) in distributed systems. 1. Why Use Transactions in Kafka? Kafka transactions are primarily used to achieve…
-
Kafka Idempotent Consumer – Ensuring Exactly-Once Processing
Introduction In distributed systems, especially those involving message brokers like Kafka, ensuring that a message is processed exactly once is critical. In this tutorial, we will explore the concept of an idempotent Kafka consumer, which is a consumer designed to handle repeated delivery of the same message without causing side effects or data corruption. What…
-
Assigning Kafka Consumers to a Group: Annotations, Configuration, and Properties
To leverage the full power of Kafka consumer groups, each consumer instance must be correctly assigned to a group ID. This ensures coordinated consumption and enables Kafka to balance load across multiple instances. In this tutorial, we explore different ways to assign Kafka consumer microservices to a group. Why Group ID Matters A group ID…
-
Kafka Consumer – Multiple Consumers in a consumer group
Introduction Let’s understand the setup with a common microservices architecture. Components: When the products microservice publishes a ProductCreated event: Each microservice is different and needs to independently handle the event (e.g., send an email, send an SMS, etc.). The Problem with Scaling One Consumer Type Let’s say you want to scale only the email-notification microservice…
-
Installing Grafana and connect with Prometheus on Windows
1. Download Grafana for Windows 2. Install Grafana Using the MSI Installer This will: 3. Start the Grafana Server Normally, after installation: If you need to manually start or stop it: 4. Access Grafana Web UI Open the browser and go to: You will see the Grafana login page. Default login credentials: After your first…
-
Retryable vs Non-Retryable Exceptions in Kafka
When designing a Kafka consumer application, you must decide how to handle failures.Failures can happen for two main reasons: Spring Kafka allows you to classify exceptions into two types: 1. Retryable Exceptions A retryable exception is an exception that happens due to a temporary issue that is likely to be fixed if we wait and…
-
Dead Letter Topic (DLT) in Kafka with Spring Boot
1. Introduction In a Kafka-based microservices system, consumers continuously read and process messages from topics. However, sometimes the consumers can encounter problematic messages: If we don’t handle these bad messages carefully, they can crash the consumer or block it into an endless retry loop. One way to deal with such issues is by using a…
-
How to Handle Deserialization Errors in a Kafka Consumer with Spring Boot
Introduction In real-world event-driven applications, message formats can change or become inconsistent, especially when multiple microservices publish to the same Kafka topic. If a Kafka consumer expects a specific message format but receives a message in an unexpected format, deserialization will fail. When deserialization fails, it causes a critical issue: In this tutorial, you will…
-
Building a Kafka Consumer with Spring Boot
Introduction In this tutorial, we will build a Spring Boot microservice that acts as a Kafka consumer. The primary use case is to consume messages from a Kafka topic published by another microservice Kafka Consumer Characteristics Configuring application.properties for Kafka Consumer Now that your project is set up, the next step is to configure Kafka…