Bill Kim 김빌

Reference · Spring Ecosystem

Spring Kafka — Architecture, Message Flow, Features, Internals

A compact reference to Spring Kafka: the container hierarchy that drives @KafkaListener, the record dispatch and error-handling flow, the features that come up day-to-day (DefaultErrorHandler, @RetryableTopic, DLT, transactions, async return types, Kotlin coroutines), and the source files where the real logic lives.

  1. Architecture
  2. Message Flow
  3. Features
  4. Internals — key source files
  5. CPU / Memory / Network at Runtime
  6. Joint with Apache Kafka
  7. Source Tree (for contributors)

1. Architecture

Spring Kafka layers a Spring-managed container hierarchy on top of the plain Kafka client. The application registers a KafkaListenerContainerFactory bean; the framework discovers @KafkaListener-annotated methods and asks the factory to produce a ConcurrentMessageListenerContainer per listener. That container spins up one KafkaMessageListenerContainer per concurrency unit, each of which runs a single-threaded ListenerConsumer that owns one Consumer client. ListenerConsumer drives the poll loop, hands records to a listener adapter chain, and routes failures to the configured CommonErrorHandler (most often DefaultErrorHandler with a backoff and a recoverer such as DeadLetterPublishingRecoverer).

@SpringBootApplication discovers @KafkaListener beans KafkaListenerContainerFactory + ConsumerFactory, ErrorHandler ConcurrentMessageListenerContainer (concurrency=3) KafkaMessageListenerContainer #1 ListenerConsumer thread · 1 Consumer KafkaMessageListenerContainer #2 ListenerConsumer thread · 1 Consumer KafkaMessageListenerContainer #3 ListenerConsumer thread · 1 Consumer Listener adapter chain RecordMessagingMessageListenerAdapter payload conversion · @SendTo · async callback wiring for failures @KafkaListener method user code (sync / async / suspend) CommonErrorHandler DefaultErrorHandler + BackOff seekAfterHandling · throws RecordInRetryException ConsumerRecordRecoverer DeadLetterPublishingRecoverer publish to DLT with cause headers on attempt exhaust KafkaTemplate produce path · transactions used by app + by DLT recoverer

Blue arrows: normal dispatch and produce paths. Red arrows: failure path through CommonErrorHandler to the recoverer. Each KafkaMessageListenerContainer is single-threaded — concurrency above one is achieved by running more containers, not by parallelizing inside one consumer.

2. Message Flow

The path of a single record from broker poll to user method, with both the success path (offset commit) and the failure path (backoff retry → recover to DLT) drawn in.

Broker ListenerConsumer Listener adapter User method · CommonErrorHandler 1 consumer.poll(timeout) 2 ConsumerRecords(records) 3 for each record → adapter.onMessage(rec) 4 convert payload + invoke method 5a return (sync) / CompletableFuture.complete / suspend resume 6a success → ack queued 7a commitAsync per AckMode (RECORD/BATCH/MANUAL...) failure path (5b–9b) 5b throw / CompletableFuture.completeExceptionally / suspend throws 6b sync: rethrow · async: callback → failedRecords queue 7b CommonErrorHandler.handleRemaining(rec, ex) 8b FailedRecordTracker++ · SeekUtils.doSeeks · throw RecordInRetryException 9b next poll re-delivers same offset (until BackOff exhausted) 10b attempts exhausted → recoverer.accept(rec, ex) 11b DeadLetterPublishingRecoverer → KafkaTemplate.send(DLT) · commit offset

Top half (1–7a): success path with offset commit. Bottom half (5b–11b): failure path through DefaultErrorHandler, with SeekUtils driving in-place retry until FailedRecordTracker's attempt budget is exhausted, then the recoverer publishes to a DLT and the offset commits past the bad record.

3. Features

Listener side

  • @KafkaListener on a method — declarative subscription. Supports topic literals, patterns, explicit partition assignments, and SpEL.
  • Return types — void, a value (used with @SendTo for request-reply), CompletableFuture<T>, Mono<T>, and Kotlin suspend. Spring Kafka uses manual acknowledgment semantics internally to support asynchronous completion and out-of-order commits.
  • Batch mode — set the container factory's batchListener=true and accept List<ConsumerRecord<K,V>>; the listener processes a full poll's worth of records per invocation.
  • @KafkaHandler on class-level @KafkaListener — payload-type dispatch (different methods for different message payload classes).
  • Consumer-aware injection — Consumer, Acknowledgment, ConsumerRecordMetadata can be method parameters.

Error handling and retry

  • DefaultErrorHandler — the standard CommonErrorHandler. Uses a BackOff (FixedBackOff, ExponentialBackOff) for in-place retry and a ConsumerRecordRecoverer when attempts are exhausted. Performs blocking retries for the affected partition (@RetryableTopic takes a different, non-blocking approach via separate retry topics).
  • @RetryableTopic — non-blocking retry. Failed records are published to a chain of retry topics (each with its own delay) and finally a DLT; the main partition keeps advancing. Useful when one bad record must not block its partition peers.
  • DeadLetterPublishingRecoverer — pluggable recoverer that publishes the failed record to a DLT with structured headers (cause class, stack trace, original topic / partition / offset, timestamp).
  • KafkaListenerErrorHandler — per-listener hook for translating exceptions before they reach the container's CommonErrorHandler.
  • Reply-side error handling — when using @SendTo, exceptions thrown by the listener can be converted to error replies via KafkaTemplate's setBinaryReplyHeader wiring.

Producer side

  • KafkaTemplate — main producer entrypoint. Supports send(ProducerRecord), send(topic, key, value), transactional executeInTransaction, and sendAndReceive for request-reply.
  • ReplyingKafkaTemplate — request-reply on top of KafkaTemplate + a reply container; correlates request and reply via the KafkaHeaders.CORRELATION_ID header (UUID by default, customisable through a CorrelationIdStrategy).
  • Transactions — KafkaTransactionManager + transactional producer. For "do DB write + publish" atomicity, the historical option was ChainedKafkaTransactionManager; that class is now deprecated, and current guidance favors the outbox pattern or Spring's transaction synchronization to coordinate the DB transaction and the producer transaction.

Exactly-once and ordering

  • Exactly-once semantics — set enable.idempotence=true + transactional.id on the producer, use transactional KafkaTemplate, and set the consumer's isolation.level=read_committed.
  • Per-key ordering — Kafka guarantees ordering inside a partition. Spring Kafka preserves it on the blocking @KafkaListener path. Async return types (suspend / Mono / CompletableFuture) may process concurrently within a partition depending on listener configuration and async completion semantics — consider this when ordering matters.

Operability

  • Observation — MessagingObservation conventions emit traces and metrics through Micrometer (producer, consumer, listener invocation spans).
  • Container lifecycle — start() / stop() / pause() / resume() at runtime; per-partition pause via ConsumerSeekAware callbacks.
  • ConsumerSeekAware — let the listener bean register seek callbacks to reposition partitions at startup / on demand (replay windows, offset reset on deploy).
  • Spring Boot auto-configuration — spring.kafka.* properties drive producer / consumer / listener / admin / streams. Listener container factory is auto-named kafkaListenerContainerFactory.

4. Internals — key source files

Where the logic actually lives. Browsing these in this order maps cleanly onto the dispatch flow above.

  • KafkaMessageListenerContainer$ListenerConsumer — the heart of the consumer side. Single-threaded poll loop. Owns the Consumer, the failedRecords deque (for async failures), the offset tracking maps (offsetsInThisBatch, deferredOffsets, lastCommits), and the calls into the listener adapter and CommonErrorHandler. handleAsyncFailure() drains failedRecords into the error handler each loop iteration.
  • ConcurrentMessageListenerContainer — supervisor that creates and lifecycles N KafkaMessageListenerContainer instances based on concurrency.
  • MessagingMessageListenerAdapter / RecordMessagingMessageListenerAdapter — the listener-adapter chain. Converts ConsumerRecord to a Spring Message<?>, invokes the user method through InvocableHandlerMethod, handles the return value (@SendTo reply, async unwrap), and routes failures to either a sync throw or the callbackForAsyncFailure hook that pushes into ListenerConsumer.failedRecords.
  • DefaultErrorHandler (extends FailedBatchProcessor) — the standard CommonErrorHandler. Coordinates the BackOff, calls SeekUtils to register seeks, decides recover-vs-retry via FailedRecordTracker, and throws RecordInRetryException to signal "this record is in retry, do not commit past it".
  • FailedRecordTracker — keyed by (topic, partition, offset). Tracks attempt count for each in-flight failure, returns the next backoff interval, and decides STOP (recover) vs CONTINUE (retry).
  • SeekUtils — utility that issues per-partition consumer.seek(...) calls to reposition the consumer back to the failing offset for the next poll-induced re-delivery.
  • DeadLetterPublishingRecoverer — implements ConsumerRecordRecoverer. Builds the DLT ProducerRecord with structured headers (original topic / partition / offset, cause class, cause message, stack trace) and publishes via an injected KafkaTemplate.
  • KafkaBackoffAwareMessageListenerAdapter / RetryTopicConfigurer / RetryTopicConfiguration — the @RetryableTopic infrastructure. Generates the retry topic chain at startup, wires a backoff-aware adapter that delays a record by the appropriate amount before invoking the user method on the next attempt.
  • KafkaTemplate — producer-side entrypoint. Wraps a Producer, supports transactions through ProducerFactoryUtils, exposes send(...) / sendAndReceive(...) / executeInTransaction(...).
  • ConsumerFactory / ProducerFactory — pluggable client factories. DefaultKafkaConsumerFactory / DefaultKafkaProducerFactory are the standard implementations; both are aware of per-instance config overrides.
  • KafkaListenerAnnotationBeanPostProcessor — discovers @KafkaListener methods at startup and asks the chosen KafkaListenerContainerFactory to create the matching container.

CPU / Memory / Network at Runtime

A Spring Kafka application is an ordinary Spring Boot process. Spring Kafka adds three runtime cost centres on top of whatever the application code itself does: listener-container threads driving polls, retry / DLT infrastructure holding state for in-flight failures, and the underlying Apache Kafka client connections to the brokers. Almost nothing else is allocated unless you opt in (transactions, retry topics, request-reply replies).

CPU / threading model

  • Listener threads — ConcurrentMessageListenerContainer creates concurrency child KafkaMessageListenerContainers. Each child runs one ListenerConsumer thread that owns one KafkaConsumer instance and loops poll → dispatch → commit. CPU on this thread is application-dependent: user listener code usually dominates, but heavy deserializers (Avro / Protobuf / Jackson JSON) can contribute meaningfully, and the Spring conversion to Message<?> sits on top of that.
  • Synchronous vs async dispatch — with a void / blocking return type, the user method runs on the listener thread itself, so CPU and throughput are bound by what the user code does. With async return types (CompletableFuture, Mono, Kotlin suspend), the listener thread hands off and returns to poll immediately; CPU moves to the scheduler the user picked.
  • Producer sender thread — KafkaTemplate wraps a single Apache Kafka KafkaProducer, which runs its own internal sender thread for batching and flushing. Most apps do not see this thread directly; it is the producer's network I/O thread.
  • Retry-topic scheduler — when @RetryableTopic is enabled, Spring Kafka starts a small scheduled-executor pool (RetryTopicSchedulerLifecycle) that wakes records out of their backoff bucket and republishes them. Idle most of the time; spikes only with retry traffic.
  • Background hooks — Micrometer observation timers, transaction synchronizations, error handlers (DefaultErrorHandler) all run on the listener thread between polls. No extra threads spun up per record.

Memory layout

  • Poll buffer per container — each KafkaConsumer can hold up to max.poll.records × max(record-size) bytes worth of ConsumerRecords between polls. The Spring conversion path (record → Message<?>) allocates per record; the GC pressure here is usually the dominant Spring Kafka cost, not heap retention.
  • FailedRecordTracker — one map entry per in-flight failing (topic, partition, offset) tracking attempt count + next backoff. Bounded by how many distinct records are currently failing across all partitions of all listener containers — typically dozens, not thousands.
  • Retry topic state — when @RetryableTopic generates the retry topic chain, the in-memory structures are small (one record-list per backoff bucket while waiting). The records themselves live in Kafka, not in heap.
  • KafkaTemplate / producer batches — the underlying KafkaProducer keeps a RecordAccumulator sized by buffer.memory (default 32 MB). A backed-up producer (slow broker) is the most common heap-growth surprise; it surfaces as RecordTooLargeException or TimeoutException rather than OOM, but the buffer is real.
  • Transactional state — with transactional.id set, the producer maintains an in-flight transaction state object and one open producer per transactional ID; the consumer side holds a small offset map for the active transaction window. Negligible.

Network traffic

  • Long-lived TCP per broker per client — each ListenerConsumer's KafkaConsumer typically maintains long-lived TCP connections to the brokers it actually communicates with (leaders of its assigned partitions plus the group coordinator), opened lazily on first use. Same shape for the KafkaTemplate's producer. With concurrency=N the consumer connection count is roughly N × brokers_with_assigned_partitions.
  • Poll frequency — each listener thread sends a FetchRequest roughly every fetch.max.wait.ms (or sooner if data is available). Spring Kafka defaults enable.auto.commit=false, so the OffsetCommit cadence is driven by the container's AckMode (RECORD / BATCH / TIME / COUNT / COUNT_TIME / MANUAL / MANUAL_IMMEDIATE), not by auto.commit.interval.ms. Heartbeats to the group coordinator run on a separate background thread inside the consumer at heartbeat.interval.ms.
  • Retry / DLT republishes — every retry attempt or DLT publication is an extra ProduceRequest. With @RetryableTopic and N retry levels, a failing record turns into N+1 produces (original failure → retry-1 → retry-2 → ... → DLT). Visible in producer-side metrics; usually small unless failure rates spike.
  • Request-reply (sendAndReceive) — adds one outbound ProduceRequest on the request topic + one inbound FetchRequest on the reply topic for every call. Spring Kafka correlates by header, so the reply consumer is shared.
  • Transactions — adds InitProducerId at startup and AddPartitionsToTxn / EndTxn round trips per transaction boundary. One transactional commit = a handful of extra small requests, not per record.
APP · SPRING KAFKA ConcurrentMessageListenerContainer (concurrency = N) ListenerConsumer #1 poll → dispatch → commit owns 1 KafkaConsumer ListenerConsumer #N poll → dispatch → commit owns 1 KafkaConsumer User @KafkaListener method sync: CPU on listener thread async: hands off, listener returns to poll KafkaTemplate → KafkaProducer RecordAccumulator 32 MB 1 sender thread (batched I/O) RetryTopic scheduler backoff buckets · idle mostly wakes on retry traffic only FailedRecordTracker one entry per (topic, partition, offset) attempt count + next backoff size = # of in-flight failing records JVM · threads + heap + direct buffers THREADS • N × ListenerConsumer (poll loop) • 1 × Producer Sender (KafkaTemplate) • Async scheduler if return type is Mono / suspend / CompletableFuture • ScheduledExecutor for RetryTopic • Consumer heartbeat thread (internal) stack + thread-locals ~MB / thread, tiny vs record / accumulator buffers. HEAP (young + old gen) • Per-poll batch of ConsumerRecord max.poll.records × avg record size • RecordAccumulator (default 32 MB) grows when broker is slow • FailedRecordTracker entries • Retry backoff buckets (record lists) GC pressure ≈ per-poll ConsumerRecord churn; retained heap usually small. DIRECT (off-heap) Java NIO ByteBuffers used by socket reads/writes. Sized by -XX:MaxDirectMemorySize Bytes copied here from kernel TCP recv buffer, then into heap as records. Released after consume. NETWORK · JVM-side socket state NETWORK band • SELECTOR Java NIO multiplexer · one per KafkaConsumer / KafkaProducer poll() blocks until any socket has data — fetch + commit + heartbeat share the same Selector • PER-BROKER TCP long-lived TCP per broker · opened lazily on first request, then reused count = N × ListenerConsumer × #leaders + 1 producer × #leaders + 1 reply consumer (sendAndReceive) • IN-FLIGHT PIPELINING max.in.flight.requests.per.connection (default 5) idempotent producer caps this at ≤ 5 to preserve per-partition ordering • TLS + IDLE REAP SSLEngine + session cache (when SSL listener used) · TLS handshake at connect OS · Linux kernel CPU cores kernel scheduler timeshares every JVM thread onto cores poll loop = run-when-data, otherwise blocks on epoll (no busy-wait) Kernel RAM page cache (file-backed, unused here) + per-socket TCP send / recv buffers (net.ipv4.tcp_wmem / tcp_rmem · KB–MB each) producer batches drain JVM → TCP send buf; fetch responses fill TCP recv buf → JVM direct Net stack + NIC epoll wakes JVM I/O thread when a socket is readable NIC TX / RX queues no Spring Kafka tuning — pure OS / NIC throughput KAFKA CLUSTER (brokers, across the NIC) Broker A partition leader group coordinator Broker B partition leader tx coordinator Broker C retry / DLT topics reply topic see also: apache-kafka.html #runtime scheduled on drive sockets supply socket bytes scheduled on copy via NIO socket epoll registered fds blue = consumer side (FetchRequest, OffsetCommit, Heartbeat) · green = producer side (ProduceRequest, batched) · amber = retry / DLT republish Bytes never enter the JVM heap on the consume hot path until the record is materialised — they live in the kernel TCP buffer, then in a direct ByteBuffer, then in heap-allocated ConsumerRecord objects.

Four layers stacked. APP = Spring Kafka structures. JVM = the threads those structures run on plus the heap / off-heap buffers they allocate. OS = the cores, kernel RAM (page cache + TCP socket buffers) and NIC the JVM lives on top of. BROKERS sit across the NIC. The expensive runtime axis for Spring Kafka is user listener CPU on the sync path or scheduler choice on the async path; memory surprises usually come from a backed-up producer's RecordAccumulator; network is bounded by NIC + per-socket TCP buffer size, not by Spring Kafka itself.

Joint with Apache Kafka

The same Spring Kafka structures and the same Apache Kafka cluster drawn together, so the boundary between "client library" and "broker" is visible end-to-end. The cluster in the middle is exactly the broker shown on /apache-kafka.html; the producer and consumer flanks are the components covered on the rest of this page.

Joint architecture

Producer-side Spring Boot app Application code REST handler, scheduled job, ... KafkaTemplate send(topic, key, value) executeInTransaction(...) DefaultKafkaProducerFactory org.apache.kafka.clients .producer.KafkaProducer Idempotent / transactional enable.idempotence=true transactional.id (optional) Apache Kafka cluster (3 brokers) Broker 1 events P0 (leader) events P1 (follower) ISR: {1,2,3} Broker 2 events P1 (leader) events P2 (follower) ISR: {1,2,3} Broker 3 events P2 (leader) events P0 (follower) ISR: {1,2,3} KRaft controller quorum leaders · ISR · configs · ACLs · __transaction_state Internal topics __consumer_offsets · __transaction_state used by consumer groups + EOS Consumer-side Spring Boot app @KafkaListener method user code · @SendTo · async CompletableFuture/Mono/suspend Listener adapter chain RecordMessagingMessageListenerAdapter Container hierarchy KafkaListenerContainerFactory → ConcurrentMessageListenerContainer → KafkaMessageListenerContainer · ListenerConsumer CommonErrorHandler DefaultErrorHandler · BackOff → DeadLetterPublishingRecoverer consumer group: billing (3 members) commit

Blue = produce / commit requests. Green = consumer fetch (zero-copy on the broker side). Dashed = leader-to-follower replication and the internal dispatch chain inside the listener container.

End-to-end sequence

A single record's journey across both halves — produce path on top, consume path on the bottom, with offset commit at the end.

Producer-side app code KafkaTemplate + KafkaProducer Leader broker (partition P) Follower broker (ISR) ListenerConsumer + adapter chain Consumer-side @KafkaListener 1 kafkaTemplate.send(topic, key, value) 2 ProduceRequest (batched, compressed, acks=all) 3 append to log, assign offset N 4 FetchRequest (follower pulls) 5 FetchResponse · LEO advanced 6 advance HW · ProduceResponse(offset) 7 CompletableFuture<SendResult> completes consumer side (8 onwards) 8 consumer.poll(timeout) — FetchRequest up to HW 9 FetchResponse(records up to lastOffset) · zero-copy 10 adapter converts payload, invokes method 11 return / suspend resume / future complete 12 OffsetCommit(group=billing, offset=N+1) 13 OffsetCommitResponse — written to __consumer_offsets on failure: 10' user throws → CommonErrorHandler → SeekUtils retry → recoverer (DLT) — see "Message Flow" section above, steps 5b–11b for the full failure path

Steps 1–7: produce path through KafkaTemplate → broker leader → ISR follower → HW advance → ack. Steps 8–13: consume path through ListenerConsumer → adapter → user method → offset commit to __consumer_offsets. The failure branch is detailed in the Message Flow section above.

Source Tree (for contributors)

Official source: github.com/spring-projects/spring-kafka. Build system is Gradle (./gradlew build); JDK 17+ for the 3.x line, JDK 17+ for 4.x. Contribution flow is the standard CONTRIBUTING.adoc — fork → branch off main (next major) or a X.Y.x release branch → PR. CI runs on GitHub Actions.

CONTRACT LIBRARY CONSUMERS spring-kafka-bom/ version contract (managed deps) consumed by Boot, Cloud Stream spring-kafka/ org.springframework.kafka.* listener · template · errors · retry 90% of feature/bugfix PRs land here spring-kafka-test/ EmbeddedKafkaBroker KafkaTestUtils · @EmbeddedKafka separately published test artifact samples/ runnable examples used as living docs / fixtures spring-kafka-docs/ Antora / AsciiDoc → docs.spring.io/spring-kafka src/ (root) checkstyle · copyright Gradle convention scripts manages manages depends on

Solid arrow = version managed by; dashed arrow = depends on. A typical bugfix PR touches one file under spring-kafka/src/main/java/... and one test under spring-kafka/src/test/java/.... A new test utility goes to spring-kafka-test/; a release bumps spring-kafka-bom/.

Top-level modules

  • spring-kafka/ — the main module. Everything in org.springframework.kafka.*: listener containers, KafkaTemplate, error handlers, retry topics, transactions, support classes. Sources under src/main/java, unit tests under src/test/java (JUnit 5 + Mockito), integration tests under src/test/java/... with @EmbeddedKafka.
  • spring-kafka-test/ — test utilities published as a separate artifact. The single most-used class is EmbeddedKafkaBroker (programmatic in-process Kafka cluster); @EmbeddedKafka is its JUnit 5 hook. Also ships KafkaTestUtils for record polling and offset assertions.
  • spring-kafka-bom/ — Bill of Materials. A single pom.xml-equivalent that other Spring projects (Spring Boot, Spring Cloud Stream) import to lock the Spring Kafka + transitively-managed versions.
  • spring-kafka-docs/ — Antora-based reference documentation. Adoc sources are here; the rendered site lives at docs.spring.io/spring-kafka/reference. Doc PRs touch this module only.
  • samples/ — runnable example projects covering listener concurrency, retry topics, transactions, Kafka Streams binding, etc. Used as living documentation and as regression fixtures for behaviour-change PRs.
  • src/ (top-level) — checkstyle config, copyright headers, Gradle convention scripts shared by the modules above.
  • build.gradle / settings.gradle / gradle.properties — Gradle project setup. The version is in gradle.properties; bump there to cut a release.

Version history

  • 2.x (2018 – 2022) — built on Spring Framework 5 / Spring Boot 2 / Java 8+. Introduced @KafkaListener's concurrency model, transactional KafkaTemplate, the original error-handler hierarchy, and the first retry-topic implementation.
  • 3.0 (Nov 2022) — Jakarta EE 9 / Spring Boot 3 / Java 17+ baseline. Package rename for javax.* → jakarta.*. New CommonErrorHandler as the single error-handler contract; DefaultErrorHandler replaced the older SeekToCurrentErrorHandler / RecoveringBatchErrorHandler.
  • 3.1 – 3.3 (2023 – 2024) — @RetryableTopic hardening, observability via Micrometer, Kafka client 3.5 → 3.7 upgrades, KRaft test support in EmbeddedKafkaBroker.
  • 4.0 (Jun 2025) — Spring Framework 7 / Spring Boot 4 / Java 17+. Native compilation (GraalVM) support tightened. spring-kafka-streams behaviour aligned with Apache Kafka 4.0 (KRaft-only).
  • 4.1 (latest at the time of writing) — async listener path overhaul (suspend / Mono / CompletableFuture return-types funneled through a single async-failure callback), strict partition-ordered retry across async boundaries.

Branching & release model

main tracks the next major. Each minor (3.3.x, 4.0.x, 4.1.x) gets its own long-lived branch; in practice contributor PRs target the branch the maintainer asks for (often the actively maintained one), and a maintainer handles back/forward-porting across the supported branches when the diff doesn't apply cleanly. Release tagging convention is vX.Y.Z; milestones (X.Y.0-M1, X.Y.0-RC1) precede every .0.

Related