Bill Kim 김빌

Appendix · OSS

Spring Kafka ↔ Apache Kafka — joint architecture & end-to-end sequence

A short appendix that wires the two existing pages together. One diagram shows how the Spring Boot application (KafkaTemplate on the produce side, @KafkaListener container hierarchy on the consume side) connects to a real Kafka cluster (broker, topic, partition leader / follower / ISR, consumer group). The second diagram traces a single message end-to-end across both sides — from application code calling send() through broker append + ISR replication, back through consumer fetch and the listener container, into the user's @KafkaListener method, and on to the offset commit.

Joint architecture

Two Spring Boot applications around the same Kafka cluster. The producer-side app uses KafkaTemplate (with optional transactions). The consumer-side app uses an @KafkaListener driven by Spring Kafka's container hierarchy. The cluster in the middle is the same Apache Kafka covered on the Apache Kafka page; the client-side pieces on either flank are the same ones covered on the Spring Kafka page.

Producer-side Spring Boot app Application code REST handler, scheduled job, ... KafkaTemplate send(topic, key, value) executeInTransaction(...) DefaultKafkaProducerFactory org.apache.kafka.clients .producer.KafkaProducer Idempotent / transactional enable.idempotence=true transactional.id (optional) Apache Kafka cluster (3 brokers) Broker 1 events P0 (leader) events P1 (follower) ISR: {1,2,3} Broker 2 events P1 (leader) events P2 (follower) ISR: {1,2,3} Broker 3 events P2 (leader) events P0 (follower) ISR: {1,2,3} KRaft controller quorum leaders · ISR · configs · ACLs · __transaction_state Internal topics __consumer_offsets · __transaction_state used by consumer groups + EOS Consumer-side Spring Boot app @KafkaListener method user code · @SendTo · async CompletableFuture/Mono/suspend Listener adapter chain RecordMessagingMessageListenerAdapter Container hierarchy KafkaListenerContainerFactory → ConcurrentMessageListenerContainer → KafkaMessageListenerContainer · ListenerConsumer CommonErrorHandler DefaultErrorHandler · BackOff → DeadLetterPublishingRecoverer consumer group: billing (3 members) commit

Blue arrows: produce / commit requests. Green arrows: consumer fetch (uses zero-copy on the broker side). Dashed arrows: leader-to-follower replication and the internal dispatch chain inside the listener container. The cluster in the middle is the same one drawn on the Apache Kafka page; the producer and consumer flanks are the same components covered on the Spring Kafka page.

End-to-end sequence

A single record's journey across both halves — produce path on top, consume path on the bottom, with offset commit at the end.

Producer-side app code KafkaTemplate + KafkaProducer Leader broker (partition P) Follower broker (ISR) ListenerConsumer + adapter chain Consumer-side @KafkaListener 1 kafkaTemplate.send(topic, key, value) 2 ProduceRequest (batched, compressed, acks=all) 3 append to log, assign offset N 4 FetchRequest (follower pulls) 5 FetchResponse · LEO advanced 6 advance HW · ProduceResponse(offset) 7 CompletableFuture<SendResult> completes consumer side (8 onwards) 8 consumer.poll(timeout) — FetchRequest up to HW 9 FetchResponse(records up to lastOffset) · zero-copy 10 adapter converts payload, invokes method 11 return / suspend resume / future complete 12 OffsetCommit(group=billing, offset=N+1) 13 OffsetCommitResponse — written to __consumer_offsets on failure: 10' user throws → CommonErrorHandler → SeekUtils retry → recoverer (DLT) — see Spring Kafka page, "Message Flow" section, steps 5b–11b for the full failure path

Steps 1–7: produce path through KafkaTemplate → broker leader → ISR follower → HW advance → ack. Steps 8–13: consume path through ListenerConsumer → adapter → user method → offset commit to __consumer_offsets. The failure branch is summarised at the bottom and drawn in full on the Spring Kafka — Message Flow diagram.

Where to go next