keepalived

keepalived moves an IP address between machines. That is the whole of it. Everything else on this page (election, weights, notify scripts, split-brain) exists in service of deciding which machine should currently hold that address.

It is worth fixing that sentence in mind before reading further, because the most expensive keepalived failures come from assuming it does more. It does not promote a database, it does not know whether your application is healthy, and it cannot stop two nodes from simultaneously believing they are in charge. It is a VRRP implementation with a script hook.

VRRP fundamentals

VRRP, defined for IPv4 and IPv6 in RFC 5798 (which obsoletes RFC 3768, the VRRPv2 specification), lets a set of routers on a LAN present a single virtual router to hosts. The virtual router owns one or more virtual IP addresses. Exactly one physical machine, the Master, answers for them at any moment.

The Master multicasts an advertisement every advert_int seconds. Backups listen. If a Backup hears nothing for the Master_Down_Interval it starts an election and the highest priority present wins. RFC 5798 defines that interval as:

Skew_Time           = ((256 - priority) * Master_Adver_Interval) / 256
Master_Down_Interval = (3 * Master_Adver_Interval) + Skew_Time

The skew term is deliberate: a higher-priority Backup waits a shorter time before declaring the Master dead, so it tends to win the race without an explicit tie-break. With advert_int 1 and priority 100, a Backup waits roughly 3.6 seconds. That is your floor for detection time, and it is why sub-second failover requires a fractional advert_int, not clever configuration elsewhere. keepalived allows a non-standard down_timer_adverts (default 3, per the RFCs) to change the multiplier, but every router in the virtual router must agree on it, and deviating from the protocol here is rarely worth the two seconds.

Why 255 is special

Priority 255 does not mean “very important”. RFC 5798 reserves it for the router that genuinely owns the virtual IP address as a real, configured address on its interface. Priority 0 is also reserved: a Master sends it to announce that it is giving up the role, so that a Backup can take over immediately instead of waiting out the timer. That leaves 1-254 for ordinary Backups.

keepalived enforces the consequences of address ownership. If an instance is configured with priority 255, tracking scripts cannot reduce that priority, and tracking scripts cannot raise a non-255 instance to 255. Under vrrp_strict, state MASTER may be configured if and only if the priority is 255, and accept mode is set automatically for 255 and cleared otherwise.

In practice, almost nobody runs a true address owner. The normal deployment has both nodes as non-owners with priorities such as 150 and 100, and the VIP existing only as an address keepalived adds and removes.

Preemption

By default a higher-priority node that comes back online takes the VIP away from a lower-priority node currently holding it. That is a second outage for the sake of tidiness. nopreempt disables it, letting the current Master keep the role until it actually fails. It only works if the initial state is not MASTER. preempt_delay (0 to 1000 seconds, default 0) is the middle ground: it delays preemption without preventing it, and likewise requires an initial state other than MASTER.

The trade-off is deterministic placement against churn. Use nopreempt when the two nodes are truly equivalent; use preemption when one node is genuinely better (more capacity, closer to the storage) and you would rather pay a short outage to get back to it.

A two-node configuration

Both files in full. Node A is the intended Master, node B the Backup, sharing VIP 10.20.0.100 on eth0.

/etc/keepalived/keepalived.conf on node A (10.20.0.11):

global_defs {
    router_id lb-a
    enable_script_security
    script_user keepalived_script
}

vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    timeout 1
    fall 3
    rise 2
    weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass k@l!ve1
    }

    virtual_ipaddress {
        10.20.0.100/24 dev eth0
    }

    track_script {
        chk_haproxy
    }

    notify /etc/keepalived/notify.sh
}

/etc/keepalived/keepalived.conf on node B (10.20.0.12) differs only in router_id, state and priority:

global_defs {
    router_id lb-b
    enable_script_security
    script_user keepalived_script
}

vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    timeout 1
    fall 3
    rise 2
    weight -20
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass k@l!ve1
    }

    virtual_ipaddress {
        10.20.0.100/24 dev eth0
    }

    track_script {
        chk_haproxy
    }

    notify /etc/keepalived/notify.sh
}

virtual_router_id, advert_int, auth_pass and the VIP list must match on both nodes. priority and state must differ.

Two notes on that authentication block. VRRP authentication was removed from the specification by RFC 3768 in 2004, and keepalived documents its use as non-compliant, useful mainly under unicast. Only the first eight characters of auth_pass are used, and vrrp_strict rejects the block outright. It is a mismatch guard, not a security control. Current keepalived documents a separate auth_hmac block: an HMAC-SHA256 trailer over the advert with a time- or monotonic-based anti-replay sequence, and support for rotating keys. That is a real authentication mechanism, and it is mutually exclusive with authentication. It was added to the project after 2.4.3, so check keepalived --version and your own man keepalived.conf before designing around it.

enable_script_security matters. keepalived runs scripts as user keepalived_script if it exists, otherwise as root, and it refuses at startup to run a root script that a non-root user could modify or replace.

vrrp_script and weight arithmetic

A vrrp_script block runs a command every interval seconds and records the exit status. This is the part people get wrong, so take the defaults precisely:

The default weight 0 is a distinct mode, not “no adjustment”. With weight 0, any VRRP instance tracking the script transitions to FAULT state after fall consecutive failures, and leaves FAULT after rise consecutive successes. FAULT means the instance drops out entirely and releases the VIP. That is unconditional: it does not matter what the other node’s priority is.

A non-zero weight instead adjusts priority. A positive weight is added after rise successes; a negative weight is subtracted from the initial priority after fall failures. The effective priority is clamped to [1, 254].

Work it through with the configuration above. Node A starts at 150 and tracks chk_haproxy with weight -20:

This is the trap. A dropped 20 points and is still 30 points ahead of B, so nothing happens. HAProxy is dead on the node holding the VIP and keepalived is behaving exactly as configured. The weight has to be large enough to cross the gap: with priorities 150 and 100, the weight must be more negative than −50. Set weight -60 and A falls to 90, B wins, the VIP moves.

Weights are for combining signals: three tracked scripts at −20 each so that any two failures trigger a move, or positive weights counting healthy uplinks so the node with most working paths wins. If you have one health check and one binary question (“is this node fit to serve?”), leave weight at 0 and let FAULT state do the work. It cannot be miscalculated.

timeout deserves attention too. It defaults to interval, so a script with interval 2 gets two seconds before being declared failed, and keepalived sends it SIGTERM. A health check that queries a database and occasionally takes three seconds will flap. Set timeout explicitly and shorter than interval, as above.

notify scripts

notify_master, notify_backup, notify_fault and notify_stop each fire on the corresponding transition. The generic notify script runs after those and receives four appended arguments:

DELETED applies only to instances, and only when notify_deleted is configured; otherwise a deleted instance reports FAULT. The type and state strings are always uppercase.

#!/bin/bash
# /etc/keepalived/notify.sh
TYPE=$1; NAME=$2; STATE=$3; PRIO=$4

logger -t keepalived-notify "$TYPE $NAME -> $STATE (priority $PRIO)"

case $STATE in
    MASTER)  systemctl start  myapp-writer.service ;;
    BACKUP)  systemctl stop   myapp-writer.service ;;
    FAULT)   systemctl stop   myapp-writer.service ;;
    STOP)    systemctl stop   myapp-writer.service ;;
esac

Scripts must terminate on SIGTERM. keepalived sets the parent-death signal to SIGTERM for everything it spawns, and kills scripts it is waiting on if they run too long. A notify script that blocks is a notify script that gets killed mid-way, leaving whatever it was doing half done.

There is one more hook worth knowing: notify_master_rx_lower_pri, which fires when a Master receives an advert with a lower priority than its own. That is the signature of a transient split-brain in which the other node briefly took over and grabbed some external resource; the hook exists so the rightful Master can reclaim it.

Split-brain

Both nodes holding the VIP simultaneously. Causes, in rough order of frequency:

• Advertisements blocked. VRRP is IP protocol 112 sent to multicast group 224.0.0.18 (ff02::12 for IPv6), not TCP and not UDP. A firewall rule, an IGMP-unaware switch, or a hypervisor that filters multicast is enough. Each node hears silence, each declares itself Master.
• Network partition. The nodes cannot reach each other but can still reach clients.
• Mismatched virtual_router_id. Two nodes with different VRIDs are two unrelated virtual routers that happen to be configured with the same VIP. Neither will ever see the other’s adverts as relevant.
• Authentication mismatch. Different auth_pass, or one node with an authentication block and one without. Adverts arrive and are discarded.

Diagnosing it takes two commands. First, confirm the symptom on both nodes:

ip -4 addr show dev eth0

If 10.20.0.100 appears on both, you have split-brain, not a routing problem. Then look for the adverts:

tcpdump -nn -i eth0 proto 112

On a healthy pair you see one advert per second, from exactly one source address, carrying the expected VRID and priority. Silence on a node that is not Master means it is not receiving; adverts from both source addresses means they are received and being rejected, which points at authentication or VRID mismatch. Check journalctl -u keepalived on both sides: rejected adverts and VRID mismatches are logged.

unicast_peer

The mitigation for the multicast case is to stop using multicast:

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    unicast_src_ip 10.20.0.11
    unicast_peer {
        10.20.0.12
    }
    unicast_fault_no_peer
    virtual_ipaddress {
        10.20.0.100/24 dev eth0
    }
}

Adverts are then sent as ordinary unicast IP packets to each listed peer, which survives any network that forwards unicast between the nodes. unicast_fault_no_peer is worth setting: until v2.2.4, keepalived would silently fall back to multicast if unicast keywords were present but no peers were listed, and this keyword forces the instance into FAULT state instead of failing open.

Two caveats. check_unicast_src (implied by min_ttl/max_ttl) verifies that adverts come from a configured peer. It is worth enabling, because the RFC’s TTL=255 check does not protect unicast. And vrrp_strict disallows unicast peers entirely, so the two options are mutually exclusive.

Unicast fixes transport. It does not fix partition: if the two nodes cannot exchange packets at all, both will still take the VIP.

keepalived does not do failover

This is the conceptual point that everything above leads to.

keepalived answers one question - which node currently holds this address - and it answers it from network reachability plus whatever your scripts report. It has no notion of application state. It will happily move a VIP onto a PostgreSQL replica that is 400 MB behind, because nothing in VRRP knows what a replica is.

For PostgreSQL, the tool that does know is Patroni: it holds leader election in a distributed consensus store (etcd, Consul or ZooKeeper), promotes and demotes Postgres, and fences the old primary. Because it uses consensus rather than “I stopped hearing adverts”, it can distinguish a partition from a failure and refuse to promote when it cannot establish a quorum. That is the property keepalived structurally cannot have.

A sound layering is Patroni for who is primary, PgBouncer for connection pooling, and keepalived or HAProxy for the address clients connect to. Each does one job. If you use keepalived in that stack, its health check should query Patroni’s REST API for the node’s role instead of trying to reason about Postgres itself.

Cloud networks

VRRP relies on two things a cloud virtual network does not provide: multicast for advertisements, and gratuitous ARP so that switches relearn the VIP’s location after a move. Both are documented absences:

unicast_peer removes the multicast dependency, and adverts will flow. The gratuitous ARP problem remains and is the fatal one: Azure states outright that its virtual networks have no Layer 2 semantics, so adding a VIP to an interface and shouting about it changes nothing about where the fabric delivers that address. The new Master holds an address the network will not route to it.

The alternatives are to let the platform own the address:

• A managed load balancer with a health probe. This is the default answer and usually the right one.
• A failover script that calls the provider API to reassign an elastic IP, move a secondary private address, or rewrite a route-table entry. keepalived supports this pattern deliberately: no_virtual_ipaddress suppresses the missing-VIP warning for exactly the case where “floating IP addresses are managed administratively” on cloud servers, and notify_master_rx_lower_pri exists so a Master can reclaim a secondary address that a briefly-promoted peer took.

That second option works, but be clear about what you have built: failover time is now bounded by an API call rather than by ARP, and the API can fail or rate-limit at the worst moment.

Diagnostics

keepalived logs to syslog by default. Check the journal:

journalctl -u keepalived --since "10 minutes ago" --no-pager

VIP not appearing on the expected interface. Verify the instance is in MASTER state:

ip addr show dev eth0 | grep <VIP>
# If not present:
journalctl -u keepalived | grep -i "entering"

Common causes: another node has higher priority and is also in MASTER state (split brain), or the VRRP advertisements are not reaching this node (firewall blocking protocol 112, or the interface is on a different VLAN).

Split brain: two nodes both holding the VIP. Both nodes believe they are MASTER because they cannot hear each other’s advertisements. Diagnose with:

# On each node:
tcpdump -i eth0 -n proto 112

If no VRRP packets arrive, the network between nodes is blocking multicast (224.0.0.18) or unicast VRRP. Cloud environments commonly block multicast; use unicast_peer configuration instead. Also check that both nodes use the same virtual_router_id and auth_pass.

Frequent MASTER/BACKUP transitions (flapping). Visible in logs as rapid “Entering MASTER” / “Entering BACKUP” cycling. Causes: network jitter causing missed advertisements, or the advert_int is too low for the network’s reliability. Increase advert_int (default 1 second) and verify no packet loss between nodes.

VIP not reachable from other hosts. After failover, the new MASTER sends a gratuitous ARP to update switch MAC tables. If the switch has not updated, other hosts still send traffic to the old MASTER’s MAC address. Check:

arping -c 3 <VIP>
# Verify the MAC matches the current MASTER's interface

Some managed switches and cloud environments suppress gratuitous ARP. In those cases, reduce ARP cache timeout on clients or use a different VIP mechanism (cloud load balancer, DNS failover).

Health check script not triggering failover. If vrrp_script is configured but failover does not happen when the service is down, check: is the script executable? Does it return the expected exit code (0 = healthy, non-zero = unhealthy)? Is weight set correctly? A negative weight reduces priority; a positive weight increases it. Test the script manually:

/etc/keepalived/check_postgres.sh; echo $?

Where keepalived is the wrong tool

• Across subnets or availability zones. VRRP is a single-LAN protocol. Two nodes in different broadcast domains cannot share a VIP.
• On a cloud virtual network, for the reasons above. Use the platform’s load balancer.
• As a quorum device. Two nodes and no arbiter cannot distinguish “peer is dead” from “peer is unreachable”. If both answers are dangerous, you need consensus with three or more voters, not VRRP.
• As an application health check. It runs the script you give it. If that script only checks that a process exists, that is all the health you get.
• As a database failover controller. Use Patroni, repmgr, or a managed service, and let keepalived move the address afterwards if you still need it to.
• For load distribution. A VIP lives on one node. If you want traffic on both, that is PgBouncer plus HAProxy, or keepalived’s own LVS/IPVS side, which is a separate subsystem from the VRRP configuration described here.