Learnitweb

Assigning Kafka Consumers to a Group: Annotations, Configuration, and Properties

To leverage the full power of Kafka consumer groups, each consumer instance must be correctly assigned to a group ID. This ensures coordinated consumption and enables Kafka to balance load across multiple instances. In this tutorial, we explore different ways to assign Kafka consumer microservices to a group.

Why Group ID Matters

A group ID tells Kafka which consumer instances belong to the same logical group. Kafka uses this information to distribute partitions among these instances, ensuring that each partition is read by only one consumer in the group.

Three Ways to Assign Kafka Consumers to a Group

Kafka (and Spring Kafka) offers multiple mechanisms for assigning consumers to a group:

1. Using @KafkaListener Annotation

You can directly specify the group ID in the @KafkaListener annotation within your event handler class.

@KafkaListener(topics = "product-created-events", groupId = "product-created-events")
public void handleProductCreated(ProductCreatedEvent event) {
    // process event
}

When to use:

  • When you want to quickly assign a group ID for simple microservices.
  • Good for prototyping or smaller applications where configuration is inline with the code.

2. Using a Custom Kafka Consumer Configuration Class

For more flexibility and centralized control, you can define the group ID in a dedicated Kafka consumer configuration class.

Step-by-step:

  • Open your Kafka consumer configuration class (typically named KafkaConsumerConfiguration).
  • Set the ConsumerConfig.GROUP_ID_CONFIG property in the consumerFactory() method.
@Bean
public ConsumerFactory<String, ProductCreatedEvent> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, env.getProperty("consumer.group.id"));
    // other configurations
    return new DefaultKafkaConsumerFactory<>(props);
}

When to use:

  • When you need centralized, testable, and reusable Kafka configuration.
  • Ideal for large-scale or production-grade applications.

3. Using application.properties

You can also configure the group ID via Spring Boot’s application.properties file.

Default property name:

spring.kafka.consumer.group-id=product-created-events

Custom property name (for flexibility):

consumer.group.id=product-created-events

If using a custom property name, be sure to fetch it in your configuration class:

props.put(ConsumerConfig.GROUP_ID_CONFIG, env.getProperty("consumer.group.id"));

When to use:

  • When you want to separate configuration from code.
  • Useful for managing different environments (dev, test, prod) via externalized configuration.