Distributed systems
Split brain
Two nodes both believe they are primary, usually after a network partition, and both accept writes that cannot afterwards be reconciled.
Split brain is not recovered from so much as cleaned up after. Prevention is structural - a lease held in a consensus store, or a watchdog that reboots a node that cannot renew it - rather than a matter of faster detection.
Explained in depth
keepalived
VRRP master election in keepalived - a runnable two-node config, the weight arithmetic people get wrong, and why it fails on cloud networks.
PatroniHow Patroni fails PostgreSQL over - the DCS leader lock, the ttl/loop_wait/retry_timeout rule, watchdog fencing, and a runnable three-node config.
Implemented by
- Patroni
- keepalived
- etcd
- Galera Cluster