Patroni

PostgreSQL can replicate, and it can be promoted. What it cannot do is decide, on its own and safely, which node should be the primary after the current one stops answering. That decision needs a third party that survives the failure and that every node agrees with. Patroni is that decision maker.

Its own documentation calls it “a template for high availability (HA) PostgreSQL solutions using Python” and is explicit that it “is far from being a one-size-fits-all or plug-and-play replication system”. Take that literally. Patroni is a Python daemon running beside a PostgreSQL instance it starts, stops, configures and monitors. PostgreSQL still does the replication; Patroni writes postgresql.conf, pg_hba.conf and the recovery settings, watches the local postmaster, and records what it sees in a distributed configuration store shared by every node. Supported PostgreSQL versions are 9.3 to 18.

Every node runs the same loop: read cluster state from the store, look at the local Postgres, act. One node holds a lease on a key that says “I am the leader”. The others do not, so they run as replicas and watch that key. That is the entire mechanism, and almost every operational question about Patroni reduces to a question about that lease.

The distributed configuration store

Patroni keeps cluster state in a DCS: etcd, Consul, ZooKeeper (directly or via Exhibitor), or the Kubernetes API. A Raft implementation embedded in Patroni also exists but is marked deprecated in the configuration reference; do not build on it. Under a namespace (default /service) and a scope (the cluster name) the store holds the members, the leader key, the dynamic configuration, the sync state, and the failsafe member list.

The store is not a convenience; it is where the safety comes from. Consensus stores answer “did this write succeed against a majority?”, and only a system that can answer that can distinguish “the primary is dead” from “I cannot currently see the primary”. A node cannot make that distinction locally, which is why a script that pings the primary and promotes a replica produces two primaries.

Patroni and DCS nodes are decoupled, so there is no minimum number of database nodes: a primary-and-one-standby cluster is documented as “perfectly fine”, with no redundancy during a failover until the failed node rejoins. The DCS is different: it “has to run with 3 or 5 nodes for proper consensus and fault tolerance”. One DCS cluster can hold state for many Patroni clusters via distinct namespace/scope pairs.

What happens when the DCS is unreachable

This is the behaviour most worth understanding before you deploy, because it looks like a bug the first time you see it.

A node is allowed to run Postgres as the primary only if it can update the leader lock in DCS. In case the update of the leader lock fails, Postgres is immediately demoted and started as read-only.

The reasoning is stated just as plainly: a lock update can fail because of network partitioning or because the DCS is down, “it is impossible to distinguish between these two from a single node, and therefore Patroni assumes the worst case - network partitioning.” If the network is partitioned, some other node may take the lock and promote. So the old primary demotes before the lock expires. You lose writes; you do not get two primaries. Patroni is choosing consistency, deliberately.

The documented escape hatch is failsafe_mode (default false), set through the dynamic configuration in the DCS. With it enabled, a primary whose lock update failed for a reason other than a version/value/index mismatch may keep running as primary if it can reach every known member via POST /failsafe on the REST API. The member list lives in a permanent /failsafe key maintained by the leader, and only members present in that key may win a leader race. Replicas treat an incoming POST /failsafe as evidence the primary is alive and cache that for ttl seconds, so they do not start an election even after the leader key has expired.

Why all members rather than a quorum? Because “the view on the quorum might be different from the perspective of DCS and Patroni”: nothing forces Patroni members to be distributed the way DCS members are, so a majority of Patroni nodes could sit on the losing side of a partition. The cost is that failsafe mode stops helping as soon as one member is unreachable: if a node is terminated while the DCS is down, the check fails on the next loop and Postgres is demoted until the DCS returns.

The leader lease, and the timing arithmetic

Four parameters decide how the cluster behaves in the seconds around a failure. They are dynamic configuration, stored in the DCS and applied cluster-wide, so change them with patronictl edit-config, not by editing YAML on each node.

The documentation attaches a hard rule to the first three, as a warning:

loop_wait + 2 * retry_timeout <= ttl

The defaults sit exactly on the boundary: 10 + 2 × 10 = 30. That is not slack, it is the budget. The leader must be able to lose one DCS attempt, retry for retry_timeout, sleep loop_wait, and still renew before the lock expires. Break the inequality and you have built a cluster that can lose its lock during ordinary DCS latency. The result is a demotion, or an election.

Each direction of tuning costs something specific. A shorter ttl detects failure sooner and shortens the write outage, but tolerates less: a network blip, a paused VM or a DCS leader election all start to read as a dead primary, and holding the inequality means shrinking loop_wait and retry_timeout too, which raises DCS traffic and leaves less patience for a slow store. A longer ttl inverts both. Cutting retry_timeout alone starts elections sooner after real trouble and demotes a healthy primary during brief hiccups.

Two documented bounds belong in the runbook. Worst-case failover time for a primary failure is loop_wait + primary_start_timeout + loop_wait, unless primary_start_timeout is zero, in which case it is just loop_wait. And in asynchronous mode, data loss is worst-case bounded by maximum_lag_on_failover bytes plus whatever was written in the last ttl seconds (loop_wait/2 in the average case), because the primary’s WAL position is not sampled in real time. Typical steady-state replication delay is described as well under a second.

By default the leader election ignores replica timelines; check_timeline: true prevents a node on a different timeline from the former primary becoming leader. Per-node tags control eligibility: nofailover: true (default false) or a failover_priority of zero or below keeps a node out of the race, which is how you stop a reporting replica in another data centre from being promoted.

Failover, switchover and the old primary

Patroni distinguishes the two operations by cluster health, and patronictl enforces the distinction only by convention.

failover also permits promoting an asynchronous node when synchronous mode is enabled: it is the manual override for when synchronous mode has correctly refused to promote anything. Nothing prevents failover on a healthy cluster; the documentation simply recommends switchover there.

When the old primary comes back, its timeline has diverged from the new one if it accepted any write that was not replicated. postgresql.use_pg_rewind (default false) tells Patroni to run pg_rewind on the former leader as it rejoins, rewinding it to the point of divergence so it can stream from the new primary without being rebuilt. It only works if the cluster was initialised with data page checksums (--data-checksums to initdb) and/or wal_log_hints = on. Decide at bootstrap: turning checksums on later means rewriting the cluster.

Two settings decide what happens when rewind is not possible, and both default to false:

Leaving both at false is the conservative choice (nothing deletes your data directory), but it is also how you end up with a node that reports Replica in patronictl list and never reaches streaming. Watch for that after a failover.

Synchronous mode

Asynchronous replication is the default and it can lose committed transactions: anything not yet replicated to the promoted standby “remain[s] in a ‘forked timeline’ on the primary, and are effectively unrecoverable”.

synchronous_mode (off, on, or quorum) changes the contract. With it on, the leader manages synchronous_standby_names itself, only the last known leader or one of the synchronous replicas may join the leader race, and Patroni “will not promote a standby unless it is certain that the standby contains all transactions that may have returned a successful commit status to client”. The price is stated in the same breath: “the system may be unavailable for writes even though some servers are available.”

There is a sharp edge here. With synchronous_mode: on alone, when no suitable standby is available the primary still accepts writes. It stops guaranteeing they are replicated, and if it then fails, no standby will be promoted. When a standby crashes, commits block until the next loop iteration switches the primary to standalone mode: worst case ttl seconds, average loop_wait/2. A clean shutdown or restart of a standby does not cause that interruption, because the standby signals the primary to release it from synchronous duties first.

synchronous_mode_strict closes that gap, and is what to use when a commit must be durable on two nodes. It prevents Patroni switching synchronous replication off when no synchronous candidate is available. “As a downside, the primary is not be [sic] available for writes (unless the Postgres transaction explicitly turns off synchronous_commit), blocking all client write requests until at least one synchronous replica comes up.” Plainly: strict mode converts a durability risk into an availability outage. That is the right trade for a ledger and the wrong one for a session store.

Use at least three data nodes with synchronous replication, so losing one host does not cost write availability. Note the honest caveat: even synchronous_mode_strict can lose a transaction if the backend is cancelled while waiting for the replication acknowledgement, because the change becomes visible to other backends before it is replicated. Tag slow or distant standbys nosync: true so they are never chosen as the synchronous one.

Watchdog and fencing

Demoting on lock-update failure is only a guarantee if Patroni is running well enough to demote. The documentation lists the cases where it is not: Patroni crashed, was killed, or was starved by load, a paused VM or other infrastructure trouble; or PostgreSQL is too slow to shut down. In any of those the lock expires, another node promotes, and the old primary is still accepting commits. That is split brain, and DCS correctness cannot prevent it from inside a process that is not being scheduled.

A watchdog device is the answer: a hardware or software timer that resets the machine if it stops receiving a keepalive. Patroni activates the watchdog before promoting, and with watchdog.mode: required a node refuses to become leader if activation fails. It is disabled again on demotion, and while the cluster is paused. Only the Linux watchdog device interface is supported; modprobe softdog plus chown postgres /dev/watchdog is enough for most uses, and soft_noboot=1 makes it log to dmesg instead of rebooting while you test.

The arithmetic behind the default: with loop_wait: 10 and ttl: 30, the watchdog expires 5 seconds before the lock, giving the HA loop at least ttl - safety_margin - loop_wait = 15 seconds to complete before a forced reset, and at least ttl - safety_margin - loop_wait - retry_timeout = 5 seconds to terminate all client connections when the DCS is unavailable. The documented residual risk is a Patroni process suspended at exactly the wrong moment, between the lock update and the keepalive; safety_margin: -1 removes it at the cost of a much shorter watchdog timeout, which in turn argues for a larger ttl and smaller loop_wait and retry_timeout.

For application-level fencing there is postgresql.pre_promote, a script run after the leader lock is acquired and before promotion. Use it for anything outside PostgreSQL that must be released before the new primary starts.

A three-node configuration

Nodes pg1 - pg3 on 10.0.2.11 - .13, a three-node etcd cluster on 10.0.3.11 - .13. This is /etc/patroni/patroni.yml on pg1; on the other nodes change name, restapi and postgresql addresses, and data_dir if it differs. The bootstrap section is read only when the cluster is first initialised; afterwards use patronictl edit-config.

scope: appdb
namespace: /service/
name: pg1

restapi:
  listen: 10.0.2.11:8008
  connect_address: 10.0.2.11:8008
  authentication:
    username: patroni
    password: rest-api-password

etcd3:
  hosts:
    - 10.0.3.11:2379
    - 10.0.3.12:2379
    - 10.0.3.13:2379

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    primary_start_timeout: 300
    synchronous_mode: true
    synchronous_node_count: 1
    failsafe_mode: true
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        max_connections: 300
        wal_level: replica
        hot_standby: "on"
        max_wal_senders: 10
        max_replication_slots: 10
        wal_log_hints: "on"
        wal_keep_size: 2GB
        archive_mode: "on"
        archive_timeout: 1800s
        archive_command: 'test ! -f /var/lib/pgsql/wal_archive/%f && cp %p /var/lib/pgsql/wal_archive/%f'
      pg_hba:
        - local all all peer
        - host all all 10.0.2.0/24 scram-sha-256
        - host replication replicator 10.0.2.0/24 scram-sha-256
  initdb:
    - encoding: UTF8
    - data-checksums

postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.0.2.11:5432
  data_dir: /var/lib/pgsql/18/data
  bin_dir: /usr/pgsql-18/bin
  pgpass: /var/lib/pgsql/.pgpass_patroni
  pg_ctl_timeout: 60
  remove_data_directory_on_rewind_failure: false
  remove_data_directory_on_diverged_timelines: false
  authentication:
    superuser:
      username: postgres
      password: superuser-password
    replication:
      username: replicator
      password: replication-password
    rewind:
      username: rewind_user
      password: rewind-password
  parameters:
    unix_socket_directories: /var/run/postgresql

watchdog:
  mode: required
  device: /dev/watchdog
  safety_margin: 5

tags:
  nofailover: false
  noloadbalance: false
  clonefrom: false
  nosync: false

Notes on the choices. data-checksums at initdb time plus wal_log_hints: "on" is what makes use_pg_rewind: true effective; neither can be added later without rebuilding. pg_ctl_timeout defaults to 60 seconds. watchdog.mode: required means a node with no working /dev/watchdog will not promote. That is correct for a cluster where split brain is worse than an outage, and it is worth verifying before you rely on it, because a node that silently cannot promote is a node you do not have. failsafe_mode: true requires every member to be running a current Patroni version. Validate before starting:

patroni --validate-config /etc/patroni/patroni.yml

Diagnosing with patronictl

patronictl talks to both the REST API and the DCS, so it needs the ctl/restapi authentication settings and the DCS section. On a cluster node, the daemon’s own configuration file will do.

patronictl -c /etc/patroni/patroni.yml list appdb
patronictl -c /etc/patroni/patroni.yml list appdb -e -t          # extended, timestamps
patronictl -c /etc/patroni/patroni.yml list appdb -w 2           # refresh every 2s
patronictl -c /etc/patroni/patroni.yml topology appdb            # tree view, shows cascading
patronictl -c /etc/patroni/patroni.yml history appdb
patronictl -c /etc/patroni/patroni.yml switchover appdb --leader pg1 --candidate pg2
patronictl -c /etc/patroni/patroni.yml failover appdb --candidate pg2
patronictl -c /etc/patroni/patroni.yml reinit appdb pg3 --wait
patronictl -c /etc/patroni/patroni.yml edit-config appdb -s ttl=45

list is where diagnosis starts. Its columns:

Three readings matter more than the rest. A replica in state running rather than streaming is not replicating. After a failover that usually means a rewind that did not happen. Divergent TL values across members mean timelines have forked, which is what history is for: it lists each failover and switchover with the timeline, LSN, reason taken from the Postgres .history file, and timestamp. And a large Replay Lag with a small Receive Lag is a replica that is receiving WAL but not applying it, typically blocked by a conflicting query.

reinit rebuilds a replica’s Postgres instance from scratch. It is the blunt fix for a node that will not rejoin, and with no member names given it does nothing, which is a deliberate safety. --from-leader takes the base backup from the leader directly instead of following the configured replica creation method.

Before a major version upgrade or any manual promotion, pause the cluster:

patronictl -c /etc/patroni/patroni.yml pause appdb
patronictl -c /etc/patroni/patroni.yml resume appdb

While paused, Patroni stops changing PostgreSQL’s state: it keeps member keys and the leader lock updated, does not start a stopped Postgres, does not stop Postgres when Patroni itself stops, and does not promote anything if there is no leader lock. It warns about parallel primaries without demoting one, and the watchdog is disabled. Check list before resume.

Where Patroni sits

Patroni decides which node is the primary. It does not tell clients where that node is. Each tool here solves exactly one problem:

The conventional layout is Patroni for failover, HAProxy routing on Patroni’s health endpoints, keepalived keeping the HAProxy address available, and PgBouncer for pooling. The REST API is the seam: it publishes each node’s role over HTTP, which is how a proxy that cannot read the PostgreSQL wire protocol still routes writes correctly. The endpoint table (/primary, /replica, /health, /leader and the traps in choosing between them) is on the HAProxy page. Read it before pointing any health check at port 8008, because /health returns 200 on every healthy node regardless of role.

Put HAProxy in front of per-node PgBouncers, not a central PgBouncer in front of HAProxy, or a failover leaves pooled server connections attached to a demoted primary. Note also postgresql.proxy_address: it exists so a pooler beside Postgres can be advertised for service discovery, not to route anything. For what Patroni is managing underneath - replication slots, wal_level, checkpoints and vacuum - see PostgreSQL.

What changed, and in which release

The 4.0.0 terminology change is the one that breaks on upgrade, because it reaches outside Patroni: monitoring alerting on patroni_master, Kubernetes selectors matching role=master, and callback scripts branching on role=master all stop working. On Kubernetes, kubernetes.leader_label_value: master preserves the old label while you migrate.

If you run Patroni on Python 3.11 or newer with vm.overcommit_memory=2 (itself recommended for PostgreSQL), read the warning at the top of Patroni’s documentation index. A CPython bug can leave the REST API unresponsive while Patroni looks healthy and PostgreSQL keeps running, which behind HAProxy health checks means a primary no proxy will route to. Releases 4.1.1+ and 4.0.8+ reduce the exposure by starting threads early.

Diagnostics

The primary tool is patronictl list:

patronictl -c /etc/patroni/patroni.yml list

This shows the cluster topology: which node is Leader, which are Replicas, lag in bytes or seconds, and the timeline. If no node shows as Leader, the cluster is in a degraded state.

No leader after a restart. Check whether the DCS (etcd/Consul/ZooKeeper) is reachable. Patroni cannot promote without updating the leader key. Run etcdctl endpoint health or equivalent. If the DCS quorum is lost, Patroni deliberately refuses to promote, which is correct behaviour, not a bug.

Primary demoted unexpectedly. Check the Patroni log for INFO: Lock owner: ... and WARNING: Loop time exceeded. Common causes: DCS latency spike (network or disk on the etcd node), ttl set too low relative to actual round-trip times, or a garbage-collection pause in a JVM-based DCS client. Increase retry_timeout and ttl if the demotion was transient.

Replica not streaming. Run patronictl list and check the Lag column. Then on the replica:

patronictl -c /etc/patroni/patroni.yml show-config
# Confirm replication slots are in use
psql -c "SELECT * FROM pg_stat_replication;" -h <primary>

If the slot exists but WAL is not shipping, check max_wal_senders and whether pg_hba.conf allows replication connections from the replica IP.

Failover hangs. If patronictl failover or switchover stalls, check whether the DCS leader key is being updated. A switchover requires the old primary to voluntarily step down; if it is unreachable (or its Patroni process is dead), use patronictl failover (not switchover). If failover still does not proceed, check whether failsafe_mode is active and the DCS is degraded.

Timeline divergence after failover. After an unclean failover, the old primary may have a different timeline. Patroni handles this via pg_rewind if configured (use_pg_rewind: true). If rewind fails, the node must be re-basebackuped:

patronictl -c /etc/patroni/patroni.yml reinit <cluster> <node>

Pause mode confusion. patronictl pause disables automatic failover but does not stop PostgreSQL. Use it for maintenance (major upgrades, manual recovery). Remember to patronictl resume afterwards, or the cluster has no automatic protection.

Where Patroni is the wrong tool

• Read/write splitting or load balancing. Patroni publishes roles; it does not proxy connections. Use HAProxy against /primary and /replica, or Pgpool-II if you need per-statement routing.
• Connection pooling. Patroni does not reduce backend count. That is PgBouncer.
• Backups. Patroni can call a backup tool as a replica-creation or bootstrap method, but it takes no backups and offers no point-in-time recovery. A cluster that fails over but cannot restore is not protected against a bad DELETE.
• Moving a virtual IP. Patroni has no VRRP. If clients connect to an address instead of through a proxy, you still need keepalived or a managed load balancer.
• Deployments without a properly sized DCS. The store must run on 3 or 5 nodes. A single-node etcd makes the consensus store a single point of failure, and every DCS restart demotes the primary.
• Multi-primary or write scaling. One node accepts writes. Distributed writes mean Citus (which Patroni can manage) or a different architecture, not a tuning exercise.
• Manual, staged operations. During a major version upgrade or corruption recovery the automation works against you. Use paused mode; do not fight the HA loop.