Skip to main content

High availability (active/standby)

Run two PBX nodes that stand in for each other. One node is active and serves phones, trunks, the web console and the background jobs. The other is standby: it serves nothing until the active node stops, then takes over on its own.

How failover works​

  • The nodes agree on which one is active through a lease in the shared Valkey. The active node renews it every few seconds.
  • When the active node stops, the standby takes the lease and starts serving: at once after a clean stop, and within the lease time (10 seconds by default) after a crash or power loss.
  • A node that loses the lease (for example because it lost its connection to Valkey) stops at once and restarts as the standby, so two nodes never serve at the same time.
  • Registrations are kept in Valkey, so registered phones keep working through the takeover.
  • Answered calls carry on when the media relay (rtpengine) is reachable from both nodes. Calls that are still ringing at the moment of failover are lost and can be dialled again.
  • When the separate media service (the one that plays music on hold and runs voice menus) restarts or one of several media services stops, answered calls carry on. A call whose media cannot stay on the same port moves to a new port of that media service, or of another one, and the phone gets the new address in a re-INVITE: callers hear a gap of about a second. A caller in the middle of a voicemail message or a recording of uploaded audio is still cut off.
  • The health check /readyz on port 9464 answers only on the active node. A floating IP address (for example with keepalived) or a load balancer uses it to send traffic to the active node.

Requirements​

  • Shared services. Both nodes use the same PostgreSQL, Valkey, ClickHouse and file storage. Use highly available versions of these services (for example a managed database, or replication with failover); a database per node does not work.
  • The same install secrets and release on both nodes. Run the first-time setup once and copy the secrets file to the second node.
  • The media relay on its own host, or on both nodes with shared state, so answered calls survive a failover.
  • One address for clients. Phones, trunks and browsers use a floating IP or a DNS name behind a health-checked load balancer. Set the SIP advertised address to it.
  • IPv6 phones. On a dual-stack install whose SIP advertised address is IPv4, set [sip] advertise6 to the PBX's IPv6 address (for example [2001:db8::5]). Phones and trunks reached over IPv6 then get that address in the SIP headers, and their media uses IPv6 while IPv4 phones keep IPv4.

Set up​

  1. On both nodes, set:

    MPBX__HA__ENABLED=true
    MPBX__HA__GROUP=primary
    MPBX__HA__LEASE_TTL_SECS=10
    MPBX__SIP__ADVERTISE=203.0.113.10

    and point both nodes at the same database, Valkey, ClickHouse and file storage.

  2. Install keepalived on both nodes. Use a health check that calls http://127.0.0.1:9464/readyz and fails unless it answers 200, and give the floating IP to the node that passes it. Use nopreempt so the address does not move back when the other node returns.

  3. Restart the PBX on both nodes. One logs "this node is now active", the other "another node is active; standing by".

  4. Run a failover drill before you go live.

SettingMeaningDefault
ha.enabledActive/standby onoff
ha.groupThe nodes that stand in for each other (lowercase letters, digits and -)primary
ha.lease_ttl_secsHow long a crashed active node holds the lease (2 to 300 seconds)10

Failover drill​

  1. Check that the console opens on the floating address and that a test phone is registered.
  2. Place an answered call between two extensions, and one over a trunk.
  3. Stop the active node cleanly (stop the service or container). The standby should serve within a few seconds.
  4. Check: the console opens again on the floating address, the test phone still makes calls, the answered calls still have audio, and a new call works.
  5. Start the stopped node again. It logs "standing by".
  6. Repeat with a hard stop (kill the container or power off the machine): the takeover then takes up to ha.lease_ttl_secs.

Maintenance and upgrades​

  • To update a node, stop the active one cleanly: the standby takes over at once. Update it, start it again (it comes back as standby), then do the same with the other node.
  • To see which node is active, look for "this node is now active" in the node's log, or check which node answers /readyz.
  • If Valkey is down, the active node keeps serving for one lease time and then stops, because it can no longer be sure it is the only active node. Keep Valkey highly available.