Bill Kim 김빌

Reference · Spring Ecosystem

Spring Kafka — Architecture, Message Flow, Features, Internals

A compact reference to Spring Kafka: the container hierarchy that drives @KafkaListener, the record dispatch and error-handling flow, the features that come up day-to-day (DefaultErrorHandler, @RetryableTopic, DLT, transactions, async return types, Kotlin coroutines), and the source files where the real logic lives.

  1. Architecture
  2. Message Flow
  3. Features
  4. Internals — key source files

1. Architecture

Spring Kafka layers a Spring-managed container hierarchy on top of the plain Kafka client. The application registers a KafkaListenerContainerFactory bean; the framework discovers @KafkaListener-annotated methods and asks the factory to produce a ConcurrentMessageListenerContainer per listener. That container spins up one KafkaMessageListenerContainer per concurrency unit, each of which runs a single-threaded ListenerConsumer that owns one Consumer client. ListenerConsumer drives the poll loop, hands records to a listener adapter chain, and routes failures to the configured CommonErrorHandler (most often DefaultErrorHandler with a backoff and a recoverer such as DeadLetterPublishingRecoverer).

@SpringBootApplication discovers @KafkaListener beans KafkaListenerContainerFactory + ConsumerFactory, ErrorHandler ConcurrentMessageListenerContainer (concurrency=3) KafkaMessageListenerContainer #1 ListenerConsumer thread · 1 Consumer KafkaMessageListenerContainer #2 ListenerConsumer thread · 1 Consumer KafkaMessageListenerContainer #3 ListenerConsumer thread · 1 Consumer Listener adapter chain RecordMessagingMessageListenerAdapter payload conversion · @SendTo · async callback wiring for failures @KafkaListener method user code (sync / async / suspend) CommonErrorHandler DefaultErrorHandler + BackOff seekAfterHandling · throws RecordInRetryException ConsumerRecordRecoverer DeadLetterPublishingRecoverer publish to DLT with cause headers on attempt exhaust KafkaTemplate produce path · transactions used by app + by DLT recoverer

Blue arrows: normal dispatch and produce paths. Red arrows: failure path through CommonErrorHandler to the recoverer. Each KafkaMessageListenerContainer is single-threaded — concurrency above one is achieved by running more containers, not by parallelizing inside one consumer.

2. Message Flow

The path of a single record from broker poll to user method, with both the success path (offset commit) and the failure path (backoff retry → recover to DLT) drawn in.

Broker ListenerConsumer Listener adapter User method · CommonErrorHandler 1 consumer.poll(timeout) 2 ConsumerRecords(records) 3 for each record → adapter.onMessage(rec) 4 convert payload + invoke method 5a return (sync) / CompletableFuture.complete / suspend resume 6a success → ack queued 7a commitAsync per AckMode (RECORD/BATCH/MANUAL...) failure path (5b–9b) 5b throw / CompletableFuture.completeExceptionally / suspend throws 6b sync: rethrow · async: callback → failedRecords queue 7b CommonErrorHandler.handleRemaining(rec, ex) 8b FailedRecordTracker++ · SeekUtils.doSeeks · throw RecordInRetryException 9b next poll re-delivers same offset (until BackOff exhausted) 10b attempts exhausted → recoverer.accept(rec, ex) 11b DeadLetterPublishingRecoverer → KafkaTemplate.send(DLT) · commit offset

Top half (1–7a): success path with offset commit. Bottom half (5b–11b): failure path through DefaultErrorHandler, with SeekUtils driving in-place retry until FailedRecordTracker's attempt budget is exhausted, then the recoverer publishes to a DLT and the offset commits past the bad record.

3. Features

Listener side

  • @KafkaListener on a method — declarative subscription. Supports topic literals, patterns, explicit partition assignments, and SpEL.
  • Return types — void, a value (used with @SendTo for request-reply), CompletableFuture<T>, Mono<T>, and Kotlin suspend. Async return types automatically switch AckMode to MANUAL with out-of-order commits.
  • Batch mode — set the container factory's batchListener=true and accept List<ConsumerRecord<K,V>>; the listener processes a full poll's worth of records per invocation.
  • @KafkaHandler on class-level @KafkaListener — payload-type dispatch (different methods for different message payload classes).
  • Consumer-aware injection — Consumer, Acknowledgment, ConsumerRecordMetadata can be method parameters.

Error handling and retry

  • DefaultErrorHandler — the standard CommonErrorHandler. Uses a BackOff (FixedBackOff, ExponentialBackOff) for in-place retry and a ConsumerRecordRecoverer when attempts are exhausted. Blocks the partition during retry.
  • @RetryableTopic — non-blocking retry. Failed records are published to a chain of retry topics (each with its own delay) and finally a DLT; the main partition keeps advancing. Useful when one bad record must not block its partition peers.
  • DeadLetterPublishingRecoverer — pluggable recoverer that publishes the failed record to a DLT with structured headers (cause class, stack trace, original topic / partition / offset, timestamp).
  • KafkaListenerErrorHandler — per-listener hook for translating exceptions before they reach the container's CommonErrorHandler.
  • Reply-side error handling — when using @SendTo, exceptions thrown by the listener can be converted to error replies via KafkaTemplate's setBinaryReplyHeader wiring.

Producer side

  • KafkaTemplate — main producer entrypoint. Supports send(ProducerRecord), send(topic, key, value), transactional executeInTransaction, and sendAndReceive for request-reply.
  • ReplyingKafkaTemplate — request-reply on top of KafkaTemplate + a reply container; correlates by a generated header.
  • Transactions — KafkaTransactionManager + transactional producer. Combine with JpaTransactionManager via ChainedKafkaTransactionManager for "do DB write + publish" atomicity within the same Spring transaction boundary.

Exactly-once and ordering

  • Exactly-once semantics — set enable.idempotence=true + transactional.id on the producer, use transactional KafkaTemplate, and set the consumer's isolation.level=read_committed.
  • Per-key ordering — Kafka guarantees ordering inside a partition. Spring Kafka preserves it on the blocking @KafkaListener path. Async return types (suspend / Mono / CompletableFuture) dispatch concurrently within a partition unless you wait for completion — consider this when ordering matters.

Operability

  • Observation — MessagingObservation conventions emit traces and metrics through Micrometer (producer, consumer, listener invocation spans).
  • Container lifecycle — start() / stop() / pause() / resume() at runtime; per-partition pause via ConsumerSeekAware callbacks.
  • ConsumerSeekAware — let the listener bean register seek callbacks to reposition partitions at startup / on demand (replay windows, offset reset on deploy).
  • Spring Boot auto-configuration — spring.kafka.* properties drive producer / consumer / listener / admin / streams. Listener container factory is auto-named kafkaListenerContainerFactory.

4. Internals — key source files

Where the logic actually lives. Browsing these in this order maps cleanly onto the dispatch flow above.

  • KafkaMessageListenerContainer$ListenerConsumer — the heart of the consumer side. Single-threaded poll loop. Owns the Consumer, the failedRecords deque (for async failures), the offset tracking maps (offsetsInThisBatch, deferredOffsets, lastCommits), and the calls into the listener adapter and CommonErrorHandler. handleAsyncFailure() drains failedRecords into the error handler each loop iteration.
  • ConcurrentMessageListenerContainer — supervisor that creates and lifecycles N KafkaMessageListenerContainer instances based on concurrency.
  • MessagingMessageListenerAdapter / RecordMessagingMessageListenerAdapter — the listener-adapter chain. Converts ConsumerRecord to a Spring Message<?>, invokes the user method through InvocableHandlerMethod, handles the return value (@SendTo reply, async unwrap), and routes failures to either a sync throw or the callbackForAsyncFailure hook that pushes into ListenerConsumer.failedRecords.
  • DefaultErrorHandler (extends FailedBatchProcessor) — the standard CommonErrorHandler. Coordinates the BackOff, calls SeekUtils to register seeks, decides recover-vs-retry via FailedRecordTracker, and throws RecordInRetryException to signal "this record is in retry, do not commit past it".
  • FailedRecordTracker — keyed by (topic, partition, offset). Tracks attempt count for each in-flight failure, returns the next backoff interval, and decides STOP (recover) vs CONTINUE (retry).
  • SeekUtils — utility that issues per-partition consumer.seek(...) calls to reposition the consumer back to the failing offset for the next poll-induced re-delivery.
  • DeadLetterPublishingRecoverer — implements ConsumerRecordRecoverer. Builds the DLT ProducerRecord with structured headers (original topic / partition / offset, cause class, cause message, stack trace) and publishes via an injected KafkaTemplate.
  • KafkaBackoffAwareMessageListenerAdapter / RetryTopicConfigurer / RetryTopicConfiguration — the @RetryableTopic infrastructure. Generates the retry topic chain at startup, wires a backoff-aware adapter that delays a record by the appropriate amount before invoking the user method on the next attempt.
  • KafkaTemplate — producer-side entrypoint. Wraps a Producer, supports transactions through ProducerFactoryUtils, exposes send(...) / sendAndReceive(...) / executeInTransaction(...).
  • ConsumerFactory / ProducerFactory — pluggable client factories. DefaultKafkaConsumerFactory / DefaultKafkaProducerFactory are the standard implementations; both are aware of per-instance config overrides.
  • KafkaListenerAnnotationBeanPostProcessor — discovers @KafkaListener methods at startup and asks the chosen KafkaListenerContainerFactory to create the matching container.

Related