Introduction

In the previous lesson, we explored how the retries property of the Kafka producer works and how retry behavior can be configured. In this lesson, we will dive into a more recommended and modern approach: using the delivery.timeout.ms configuration property instead of explicitly setting the number of retries.

Kafka’s documentation encourages developers to rely more on time-based delivery configuration rather than controlling retries by count. This ensures a more flexible, responsive, and efficient system that adapts better to distributed environments with varying conditions.

What is delivery.timeout.ms?

The delivery.timeout.ms property sets the maximum amount of time the Kafka producer will wait for a message to be acknowledged by all required in-sync replicas. If this time elapses before the message is acknowledged, an error will be thrown.

Key Features of delivery.timeout.ms:

• It is a global timeout for the entire delivery process including retries, batching, waiting for acknowledgements, and internal delays.
• This setting ensures your producer won’t keep retrying indefinitely, which could otherwise lead to resource locking and operational issues.
• The default value is 120000 milliseconds (2 minutes).

Important Note:

If you modify delivery.timeout.ms to be lower than the default, it must still be greater than or equal to the sum of two other important configuration properties:

1. linger.ms
2. request.timeout.ms

These two properties determine internal waiting and communication behavior and must be accounted for within the total delivery time window.

Supporting Kafka Configuration Properties

Let’s now understand the two critical properties that influence the behavior of delivery.timeout.ms.

1. linger.ms – Time to Buffer Before Sending

In Spring Boot: spring.kafka.producer.linger-ms

This configuration controls how long the Kafka producer waits before sending a batch of records.

• Default Value: 0 milliseconds
• Purpose: It helps buffer messages and send them in batches to improve throughput and enable compression.
• Impact on Performance: When set to a non-zero value, the producer will wait for more messages to accumulate in the buffer. This improves efficiency and network utilization, especially when using compression.
• Behavior When Set to 0: The producer sends each message as soon as it arrives without waiting.

2. request.timeout.ms – Maximum Time Waiting for Broker Acknowledgment

In Spring Boot: spring.kafka.producer.request-timeout-ms

This setting controls how long the Kafka producer will wait to receive a response from the broker after sending a single request.

• Default Value: 30000 milliseconds (30 seconds)
• Purpose: If the producer does not receive a response from the Kafka broker within this time, the request is considered failed.
• Distinction from delivery.timeout.ms:
  • request.timeout.ms applies to a single request-response cycle.
  • delivery.timeout.ms applies to the entire process, including retries and all attempts to send the message successfully.

Practical Configuration Example

Here’s how you can configure your Kafka producer in a Spring Boot or plain Java application:

# Send the message immediately, no delay
linger.ms=0

# Max time the producer waits for broker response
request.timeout.ms=30000

# Total max time for the whole send operation including retries
delivery.timeout.ms=60000

In this case:

• The Kafka producer sends messages immediately.
• It will wait up to 30 seconds to get a response from a broker for each individual request.
• It allows up to 60 seconds overall for message delivery, which includes retries and response waits.

Best Practices

Here are a few best practices to consider:

• Keep delivery.timeout.ms high enough to accommodate network latency, broker delays, and restarts.
• Use batching (linger.ms > 0) if your application sends many small messages and you want to increase throughput.
• Ensure proper broker configuration like replication factor and min.insync.replicas to make full use of these producer settings.

Summary of Configuration Relationships

Let’s summarize how these configurations relate:

• linger.ms determines how long the producer will wait before sending a batch.
• request.timeout.ms sets how long the producer waits for a response for each request.
• delivery.timeout.ms controls the total duration allowed for sending the message, including retries and broker acknowledgements.

Remember:

delivery.timeout.ms >= linger.ms + request.timeout.ms