Streaming platforms

Apache Kafka is a distributed, durable, ordered log. Producers append records to topics; consumers read them by tracking an offset per partition. Unlike a queue, reading does not remove data - retention is time- or size-based, so many independent consumers can read the same record, and a consumer can rewind and re-read history. That one property - replay - is what makes Kafka the backbone for pipelines: a consumer that falls behind catches up, and a new consumer can build its own view of the world from the beginning.

Partitions: the unit of everything

A topic is divided into partitions, each an independent ordered log. Partitions are the unit of parallelism and the boundary of ordering:

• Ordering is per-partition only. Records with the same key hash to the same partition, so ordering within an entity holds only if you key by that entity. “Kafka guarantees order” is wrong without the qualifier.
• Partition count caps consumer parallelism. A partition can be read by only one consumer per group at a time, so ten partitions mean at most ten active consumers. The count is set at topic creation and increasing it later reshuffles key-to-partition mapping for new records - choose it for the throughput you expect to grow into, not what you need today.
• Keys decide locality. No key means round-robin; a skewed key (one hot tenant, one hot device) creates a hot partition that no amount of consumer scaling fixes.

Consumer groups and lag

A consumer group is a set of consumers that share the work of a topic - the group’s assignment spreads partitions across members, and a rebalance redistributes them when members join or leave. Consumer lag - how far behind the log’s end each partition’s committed offset sits - is the health metric: rising lag means consumers cannot keep up, and the record will age out of retention before it is read if nothing changes.

Replication and durability

Each partition has a leader and follower replicas; the in-sync replica (ISR) set is the followers currently caught up. A write is considered committed when min.insync.replicas have it - which is where the durability contract actually lives:

• acks=1 acknowledges after the leader writes, before followers confirm - a leader crash right then loses the record.
• acks=all waits for the ISR; combined with min.insync.replicas=2 and replication.factor=3, one broker loss costs nothing.
• unclean.leader.election.enable=false prevents a lagging replica from becoming leader - trading availability for not losing acknowledged writes. The default has been false since Kafka 0.11.

The defaults are correct for most workloads; the failures come from overriding them for throughput without realising what was traded.

Delivery semantics

• At-least-once is the natural mode: process then commit the offset, and a crash between them replays the record. Consumers must be idempotent.
• At-most-once: commit before processing - the record is lost on crash.
• Exactly-once exists via idempotent producers and transactions that commit offsets and output writes atomically (read-process-write pipelines like Kafka Streams). It is not a blanket property - it holds inside the transactional boundary, and downstream of the transaction you are back to idempotency.

KRaft: no more ZooKeeper

Kafka 4.0 removed ZooKeeper entirely. Metadata now lives in the cluster itself via the KRaft consensus protocol, run by dedicated controller nodes (or combined broker-controller nodes in small clusters). 3.x releases support both modes for migration; there is no ZooKeeper path in 4.x. The practical change for operators: controller quorum health replaces ZooKeeper health in your monitoring, and kafka-metadata-quorum replaces the old ZooKeeper shell tooling.

Failure modes and diagnostics

• Under-replicated partitions. A broker lagging or dead shrinks the ISR; writes still succeed if min.insync.replicas is met, but the next failure loses data. Alert on UnderReplicatedPartitions and OfflinePartitionsCount, not just broker liveness.
• Consumer group stuck rebalancing. Slow processing or session timeouts below the processing time cause members to drop and rejoin in a loop. Tune max.poll.interval.ms against real processing time rather than shrinking batch sizes blindly.
• Retention biting slow consumers. A topic retaining seven days deletes the oldest segments as it grows; a consumer rejoining after eight days finds its offset out of range and either resets to earliest (re-processing everything) or fails, depending on auto.offset.reset. Retention is an operational decision, not a default.
• Hot partitions. One key dominating the partition’s throughput shows as uneven lag across partitions in the same group - the fix is a better key, not more brokers.

What this is not for

Kafka is not a database: there are no queries, no secondary indexes, and reading “the current value” means replaying the log (compacted topics help but are not point lookups). It is not an RPC mechanism - request-response over Kafka works but is awkward, because the platform is built for durable ordered delivery, not synchronous answers. And it is not a queue with per-record acknowledgement: offsets move forward; a single poisoned record blocks the partition unless you handle it explicitly.