In Apache Kafka, you can fine-tune how producers and consumers interact with topics and partitions. This tutorial explains how to: Part 1: Controlling Kafka Producers By default, Kafka decides which partition to send a message to. However, Kafka allows you to override this behavior using the following techniques: Option 1.1: Use a Message Key (Default…

When using Apache Kafka with Avro for message serialization, the Schema Registry plays a critical role in managing, validating, evolving, and retrieving schemas for the data being sent and received. Here’s an explanation of its role—along with a practical example. What is a Schema Registry? A schema registry is a centralized service that stores and manages Avro…

1. Overview The Saga Pattern is a microservice architectural pattern for managing data consistency across multiple services in distributed transaction scenarios. Instead of using distributed transactions (which are difficult to scale), Saga breaks a transaction into a series of local transactions, where each service performs its operation and publishes an event to trigger the next…

1. Introduction Kafka transactions ensure that either all Kafka operations within a transaction succeed, or none of them are applied. This guarantees: Kafka achieves this by tagging producers with transactional IDs, enabling Kafka to keep track of the state of transactions even across application crashes or restarts. 2. Enable Transactions via application.properties To enable Kafka…

In this tutorial, we will explore Kafka Transactions—what they are, why they’re needed, and how to enable them in a Spring Boot Kafka microservices architecture. Transactions in Kafka are vital for ensuring data consistency and achieving exactly-once semantics (EOS) in distributed systems. 1. Why Use Transactions in Kafka? Kafka transactions are primarily used to achieve…

Introduction In distributed systems, especially those involving message brokers like Kafka, ensuring that a message is processed exactly once is critical. In this tutorial, we will explore the concept of an idempotent Kafka consumer, which is a consumer designed to handle repeated delivery of the same message without causing side effects or data corruption. What…

To leverage the full power of Kafka consumer groups, each consumer instance must be correctly assigned to a group ID. This ensures coordinated consumption and enables Kafka to balance load across multiple instances. In this tutorial, we explore different ways to assign Kafka consumer microservices to a group. Why Group ID Matters A group ID…

Introduction Let’s understand the setup with a common microservices architecture. Components: When the products microservice publishes a ProductCreated event: Each microservice is different and needs to independently handle the event (e.g., send an email, send an SMS, etc.). The Problem with Scaling One Consumer Type Let’s say you want to scale only the email-notification microservice…

1. Download Grafana for Windows 2. Install Grafana Using the MSI Installer This will: 3. Start the Grafana Server Normally, after installation: If you need to manually start or stop it: 4. Access Grafana Web UI Open the browser and go to: You will see the Grafana login page. Default login credentials: After your first…

When designing a Kafka consumer application, you must decide how to handle failures.Failures can happen for two main reasons: Spring Kafka allows you to classify exceptions into two types: 1. Retryable Exceptions A retryable exception is an exception that happens due to a temporary issue that is likely to be fixed if we wait and…