etcd

etcd is a key-value store whose entire value proposition is that a write either reaches a majority of members or does not happen. Its own documentation describes it as “a consistent distributed key-value store” that is “designed to hold small amounts of data that can fit entirely in memory”. Both halves of that sentence matter. The consistency is what makes it usable as the source of truth for cluster membership and leadership, and the small-data assumption is what makes almost every operational surprise a storage surprise.

The mechanism that produces the consistency is Raft, and the cost of Raft is an fsync on a majority of members before a write is acknowledged. That single fact explains the hardware advice, the timeout tuning, the monitoring, and most of the outages.

Raft, in practical terms

One member is the leader. It accepts every write that requires consensus; a write sent to a follower is forwarded to the leader automatically, so clients do not need to know who the leader is. The leader appends the write to its Raft log, replicates the entry to followers, and once a majority have persisted it the entry is committed and applied to the state machine.

The leader keeps its position by sending heartbeats. A follower that hears nothing for an election timeout starts an election. During an election the cluster cannot process writes; requests are queued until a new leader exists. Writes that reached the old leader but were never committed may be lost, because the new leader is entitled to overwrite uncommitted entries. No committed write is ever lost.

Quorum is (n/2)+1, integer division, so a three-member cluster needs two and a five-member cluster needs three. The consequence is the whole reason odd sizes are recommended:

Two members tolerate no failures at all. That is worse than one member, because there are now two machines that can take the cluster down. Four members tolerate exactly what three do, while adding a machine that can fail and a machine that must be written to. The documentation puts it directly: adding a node to an odd-sized cluster makes fault tolerance worse, because the same number of nodes may fail without losing quorum but there are more nodes that can fail. An odd size also guarantees that a network partition produces one side with a majority, which can keep serving.

There is no hard upper limit, but the project advises no more than seven members, and notes that Google’s Chubby lock service, comparable in role, suggests five. Five tolerates two failures, which covers a host loss during a rolling upgrade. Beyond that, write latency degrades. Every commit waits on more disks.

--pre-vote (default true since it was stabilised) has a member check whether it could win an election before disrupting the current leader, which stops a member returning from a partition from forcing a needless election. --strict-reconfig-check (default true) rejects membership changes that would cost quorum. Leave both on.

Revisions, watches and leases

etcd’s store is multi-version. A write does not overwrite; it creates a new revision of the keyspace. Revisions start at 1 and increase monotonically for the life of the cluster, one per atomic mutation, and a transaction containing several operations still produces one revision. Old revisions remain readable (etcdctl get --rev=N key) until compaction removes them. Physically this is a B+tree keyed by (major, sub, type) where major is the revision, plus an in-memory B-tree index from user keys to their modifications. This design is what makes watches cheap and correct. A watcher can ask for events from a specific revision and receive every change since, in order, instead of polling. It is the mechanism the Kubernetes API server uses to keep controllers in step, and it is why a restore that moves the revision backwards is dangerous (see disaster recovery below).

Leases are how clients express liveness. A client requests a lease with a TTL; the cluster grants a lease ID and a server-selected TTL in seconds. Keys can be attached to a lease (at most one lease per key), and when the lease expires or is revoked, every key attached to it is deleted and a delete event is generated. The client keeps the lease alive over a bidirectional LeaseKeepAlive stream.

That is the whole primitive behind distributed leader election. A candidate writes a leader key with a lease attached, using a transaction that only succeeds if the key does not already exist. While the candidate keeps renewing, the key persists and it is the leader. If the process dies, is partitioned away, or is simply not scheduled for long enough, the lease expires, etcd deletes the key, and the watching candidates see the delete event and race for it.

Patroni uses exactly this, which is why its ttl parameter is a lease TTL and why its timing arithmetic is really arithmetic about how many etcd round trips fit inside the lease. One etcd behaviour is worth knowing on that page’s behalf: when a new etcd leader is elected, it extends the timeouts of all leases automatically, so a lease does not expire merely because etcd itself was busy holding an election. That is protection against etcd-side unavailability; it is not protection against a client that failed to renew.

Disk latency is the whole game

The documentation is blunt: fast disks are the most critical factor for etcd performance and stability, and etcd is “very sensitive to disk write latency”. A majority of members must write every request down to disk before it commits, so commit latency is network round-trip time plus the time fdatasync needs.

The IOPS numbers come with a warning that is easy to miss and expensive to ignore: cloud providers advertise concurrent (random) IOPS figures that can be an order of magnitude higher than the sequential write performance etcd actually needs. A volume advertised at 3000 IOPS may deliver a few hundred sequentially. Benchmark with fio before you believe a number.

This is why shared and network storage causes instability, not merely slowness. etcd’s failure mode under fsync latency is not a slow response, it is a missed heartbeat. The leader often has to persist metadata before sending heartbeats; if the disk stalls, heartbeats are late, followers conclude the leader is dead and start an election, and the cluster stops accepting writes for the duration. A shared virtualised disk gives you tail latency you do not control, and etcd converts tail latency into leader elections. The same argument applies to running etcd on a node alongside anything that does bursty I/O, including, notably, the database whose failover it is coordinating.

Two warnings in the logs are the ones to recognise:

Applying a request should normally take under 50 ms even on a slow mechanical or network-attached disk; etcd warns when the average exceeds 100 ms, configurable via --experimental-warning-apply-duration (default 100ms). If both fsync metrics look healthy, the next suspects are CPU starvation and expensive client requests that touch very large numbers of keys.

Three mitigations from the tuning guide, in order of usefulness: give etcd a dedicated disk; raise its I/O priority with ionice -c2 -n0 -p $(pgrep etcd); and prioritise peer traffic on port 2380 over client traffic on 2379 with tc if followers log dropped MsgProp ... sending buffer is full.

Timeouts, and when to change them

The defaults suit a low-latency local network. Across data centres, raise both, keep the 10× ratio, and set identical values on every member: mismatched values disrupt cluster stability. Also note that election timeout is a budget for disk as well as network: each timeout is the total time from request to response, and a slow disk on either side consumes it.

On --snapshot-count the etcd documentation contradicts itself: the v3.6 configuration reference lists the default as 10000, while the maintenance guide states the default changed from 10,000 to 100,000 in v3.2. Check etcd --help on the binary you are running instead of trusting either page. The trade-off is clear regardless: a higher count holds more Raft entries in memory, which gives a slow follower more time to catch up before the leader compacts and has to ship it a full snapshot, at the cost of memory and Go GC pressure.

The space quota: the outage you will actually have

Because etcd keeps every revision, the backend database grows with write volume, not with the size of the live keyspace. A key rewritten a million times is a million revisions. If nothing compacts them, the database reaches its quota, and etcd raises a cluster-wide NOSPACE alarm that puts the cluster into a maintenance mode accepting only reads and deletes. Writes fail with:

Error: rpc error: code = 8 desc = etcdserver: mvcc: database space exceeded

For a Kubernetes cluster or a Patroni cluster, “no writes to etcd” means no scheduling, no leader lock renewals, and demotion of the PostgreSQL primary. The alarm does not clear on its own, and restarting etcd does not clear it.

The quota is --quota-backend-bytes, whose default is 0, meaning a low default quota; the FAQ states that default as 2 GB. It also warns that 8 GB is the suggested maximum for normal environments and that etcd warns at startup above that, and that the machine should have at least as much RAM as the quota to avoid swapping.

Recovery is four steps, in order:

# 1. Confirm the alarm and see the database size per member
etcdctl --write-out=table endpoint status
etcdctl alarm list

# 2. Compact away history up to the current revision
rev=$(etcdctl endpoint status --write-out=json | grep -o '"revision":[0-9]*' | grep -o '[0-9].*')
etcdctl compact "$rev"

# 3. Return the freed space to the filesystem, one member at a time
etcdctl defrag --cluster

# 4. Disarm the alarm
etcdctl alarm disarm

Two things about that sequence bite people.

Compaction does not free disk space. It drops superseded revisions inside the backend, leaving gaps. The space is available to etcd but not to the filesystem, so db size does not fall and the alarm does not clear. Only defragmentation returns it, and defragmenting a live member blocks reads and writes on that member while it rebuilds its state. The request is not replicated, so it must be issued per member. --cluster walks them for you, but sequentially and deliberately, to avoid a cluster-wide latency spike. On a member that is stopped, etcdutl defrag --data-dir <path> does the same job offline.

Then stop it happening again by enabling automatic compaction, which is off by default (--auto-compaction-retention defaults to 0).

The documented guidance: a short window such as 30m or 1h where the same keys are updated frequently, 24h - 72h where updates are rare, and 10h as a general-purpose default. The retention window is not free: it is how long a slow watcher has to catch up before it gets mvcc: required revision has been compacted and must resynchronise from scratch.

Automatic compaction still leaves fragmentation, so schedule a rolling defrag as well, per member, outside peak hours. Watch two metrics: etcd_mvcc_db_total_size_in_use_in_bytes is usage after compaction, and etcd_debugging_mvcc_db_total_size_in_bytes (renamed to etcd_mvcc_db_total_size_in_bytes from v3.4) includes free space awaiting defragmentation. When both approach the quota, compaction is overdue.

One sharp edge worth knowing: a Put, Txn or LeaseGrant can return ErrGRPCNoSpace and still have succeeded in the backend, because etcd checks the quota at both the API layer and the internal apply layer, and the apply layer only raises the alarm without blocking the transaction.

Monitoring that predicts failure

Metrics are on /metrics on the client port, or on --listen-metrics-urls if you want them reachable without client TLS. --metrics defaults to basic; extensive adds server-side gRPC histograms. Since v3.4 there are also /livez and /readyz, and ?verbose prints the individual checks.

The metrics that lead rather than lag:

A three-node configuration

Three members on 10.0.3.11 - .13. This is /etc/etcd/etcd.conf.yml on the first; on the others change name and the four URL fields. The config file is a YAML map whose keys are the flag names with the -- prefix removed, and if a config file is supplied, command-line flags and environment variables are ignored entirely.

name: etcd1
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd-wal

listen-peer-urls: https://10.0.3.11:2380
listen-client-urls: https://10.0.3.11:2379,https://127.0.0.1:2379
initial-advertise-peer-urls: https://10.0.3.11:2380
advertise-client-urls: https://10.0.3.11:2379

initial-cluster: etcd1=https://10.0.3.11:2380,etcd2=https://10.0.3.12:2380,etcd3=https://10.0.3.13:2380
initial-cluster-token: appdb-etcd-2026-08
initial-cluster-state: new

heartbeat-interval: 100
election-timeout: 1000
snapshot-count: 10000

quota-backend-bytes: 8589934592
auto-compaction-mode: periodic
auto-compaction-retention: "10h"

strict-reconfig-check: true
pre-vote: true

listen-metrics-urls: http://10.0.3.11:2381
metrics: basic
log-level: info
log-format: json

client-transport-security:
  cert-file: /etc/etcd/pki/server.crt
  key-file: /etc/etcd/pki/server.key
  trusted-ca-file: /etc/etcd/pki/ca.crt
  client-cert-auth: true

peer-transport-security:
  cert-file: /etc/etcd/pki/peer.crt
  key-file: /etc/etcd/pki/peer.key
  trusted-ca-file: /etc/etcd/pki/ca.crt
  client-cert-auth: true

Points that are not decoration. wal-dir on a separate device isolates sequential WAL fsyncs from the backend’s random I/O, which is the single most effective change on marginal storage. initial-cluster-token prevents a member from accidentally joining a different cluster that shares addresses. Give every cluster a distinct one, and a new one on every restore. client-cert-auth should be on: the configuration reference recommends client certificate authentication to prevent attacks from unauthenticated clients, naming CVE-2023-44487 (the HTTP/2 rapid-reset denial of service) as the reason. initial-cluster-state: new applies only to the first boot; change it to existing when adding a member to a running cluster.

listen-metrics-urls on a separate port matters once client-cert-auth is on, because a scraper would otherwise need a client certificate. That is the documented purpose of the flag.

Disaster recovery

etcd tolerates (N-1)/2 permanent failures. Lose more and quorum is gone irrecoverably; the only route back is snapshot and restore. Take snapshots on a schedule and store them off the cluster.

# Take a snapshot from a live member (etcdctl)
etcdctl --endpoints=https://10.0.3.11:2379 snapshot save /backup/etcd-$(date +%F-%H%M).db

# Inspect it (etcdutl)
etcdutl --write-out=table snapshot status /backup/etcd-2026-08-22-0300.db

From v3.6 the split is enforced: etcdctl takes snapshots, etcdutl restores them. A snapshot taken with etcdctl snapshot save carries an integrity hash that restore verifies; a file copied out of member/snap/db does not, needs --skip-hash-check, and may be missing data still only in the WAL.

Restoring creates fresh data directories and overwrites the member and cluster IDs, so the restored members lose their old identity. That is deliberate: it stops them rejoining a cluster that still exists. Restore the same snapshot on every member:

etcdutl snapshot restore /backup/etcd-2026-08-22-0300.db \
  --name etcd1 \
  --data-dir /var/lib/etcd \
  --initial-cluster etcd1=https://10.0.3.11:2380,etcd2=https://10.0.3.12:2380,etcd3=https://10.0.3.13:2380 \
  --initial-cluster-token appdb-etcd-restore-2026-08-22 \
  --initial-advertise-peer-urls https://10.0.3.11:2380 \
  --bump-revision 1000000000 --mark-compacted

--bump-revision and --mark-compacted are not optional in practice if anything watches the keyspace, and the Kubernetes API server does. A restore moves the revision backwards by however much the cluster advanced after the snapshot was taken. Clients holding cached state built from watches (Kubernetes informers, for example) will not necessarily notice, and controllers then act on a view that no longer matches reality. --bump-revision adds a fixed offset so revisions never decrease; the documentation suggests 1,000,000,000 covers a week-old snapshot at under 1500 writes per second. --mark-compacted marks everything up to and including the bump as compacted, which terminates all watches and forces caches to resynchronise.

Avoid --force-new-cluster. The documentation calls it strongly discouraged: it panics if members of the previous cluster are still alive.

Diagnostics

Start with endpoint health:

etcdctl endpoint health --cluster
etcdctl endpoint status --cluster -w table

The status table shows each member’s Raft term, index, and whether it is the leader. All members should be on the same term; a member on a different term has been partitioned.

High fsync latency. The single most common operational issue. etcd commits every write to WAL with fdatasync. Check with:

etcdctl check perf --load="s"
# or directly:
fio --name=etcd-wal --rw=write --fdatasync=1 --ioengine=sync --bs=2300 \
    --size=22m --directory=/var/lib/etcd/member/wal

If 99th-percentile fdatasync exceeds 10ms, the disk is too slow. Move the WAL directory to a dedicated, low-latency volume (NVMe, not network-attached). On cloud providers, use provisioned IOPS volumes and verify sequential write performance, not the advertised random IOPS figure.

Space quota alarm. etcd rejects writes when it exceeds its space quota (default 2GB). The alarm persists even after compaction until explicitly disarmed:

# Check current database size
etcdctl endpoint status -w table
# Compact to the current revision
etcdctl compact $(etcdctl endpoint status -w json | jq '.[0].Status.header.revision')
# Defragment to reclaim space
etcdctl defrag --cluster
# Disarm the alarm
etcdctl alarm disarm

If this recurs, either the quota is too small for the workload, or automatic compaction is not configured. Set --auto-compaction-retention to a value appropriate for your watch consumers (typically 1 hour for Patroni/Kubernetes).

Leader elections cycling. Frequent leader changes visible in etcdctl endpoint status (term incrementing rapidly) indicate network instability between members or disk latency causing heartbeat timeouts. Increase --heartbeat-interval and --election-timeout proportionally, but note that higher values increase failover time. The documented relationship is election-timeout >= 10 * heartbeat-interval.

Member not catching up. A member that stays behind on Raft index after rejoining may need a snapshot transfer. Check logs for database snapshot. If it keeps failing, remove the member and re-add it:

etcdctl member remove <id>
etcdctl member add <name> --peer-urls=https://<ip>:2380

Connection refused from Patroni. If Patroni logs etcdserver: request timed out or connection refused, confirm the etcd client endpoints in Patroni’s configuration match the running etcd cluster. Check TLS certificate validity if using --cert-file / --key-file.

Where etcd is the wrong tool

etcd is for small amounts of infrequently updated, high-value metadata that must be agreed on. It is explicitly not a general-purpose database, and the failure is not graceful. Exceeding the quota takes writes away from every client at once.

• Application data, queues, sessions or caches. Every write costs a majority fsync and adds a revision that must later be compacted and defragmented. Use PostgreSQL, or Redis for a cache.
• Large values or blobs. --max-request-bytes defaults to 1.5 MiB and --max-txn-ops to 128; the store is designed to fit in memory. Store a pointer, not the object.
• High write throughput. Throughput is bounded by the slowest disk in the quorum. Adding members makes writes slower, not faster.
• Metrics or events. Every sample would be a revision. That is a time-series problem; see Prometheus.
• A cluster spread across regions for latency. Cross-region members improve failure-domain independence at the cost of adding inter-region RTT to every commit. The documented preference is a single data centre, or the nearest second one.
• Two or four members. Not “suboptimal” - no better than one and three respectively, with more hardware that can fail.

Where etcd is right, it is right for a specific reason: it can answer “did this write reach a majority?”, and only a system that can answer that can distinguish a dead primary from an unreachable one. That is why it underpins the control plane in Kubernetes and why it is the default distributed configuration store for Patroni. It also means the etcd cluster inherits the availability requirements of everything above it, and must be operated accordingly: dedicated hosts, dedicated disks, automatic compaction on, scheduled defragmentation, snapshots off-cluster, and alerts on fsync latency rather than on the outage that follows it.