Split Brain: Mitigation Strategies with a MySQL HA Example (using Pacemaker)

1. What Is Split Brain?

Split brain is an abnormal state in a cluster where two or more nodes simultaneously believe they are the primary (master).
It often occurs after a short network partition when health checks misfire, allowing both sides to become active.
When the network reconnects, the result is data conflicts, corruption, and inconsistency.
This is especially critical in systems where data integrity is paramount, such as databases and storage.

2. A Typical Example

A common case is Pacemaker + Corosync + MySQL Dual Master Replication:

• MySQL Dual Master Replication allows both nodes to accept writes.
• Pacemaker manages VIP and MySQL services, handling failover in case of failure.
• Corosync provides node-to-node messaging, health checks, and quorum management.
• During a network partition, both nodes may claim the VIP and operate as primaries, triggering a split brain.

3. Why the Recovery Phase Is Dangerous

• While isolated, the nodes may look like independent systems.
• Once the network reconnects, each node’s changes collide.
• In a database environment, this leads to inconsistent transactions and broken integrity.

4. Common Causes

• Health check limitations: heartbeat-level checks may miss real service failures.
• Two-node cluster limitation: quorum ties cannot be broken without a tie-breaker.
• Fencing failure: faulty nodes are not isolated, allowing VIP and services to duplicate.
• No recovery safeguards: no mechanism to prevent conflicts when connectivity returns.

5. Prevention and Mitigation Strategies

Depending on infrastructure and operational requirements, these strategies can be applied individually or in combination.

Quorum

• Use three or more nodes (odd number) or a quorum device to enforce majority rules.
• Two-node clusters without a tie-breaker are inherently unstable.

Fence Device (STONITH)

• Use IPMI, power-off, or NIC disablement to completely isolate a faulty node.
• This is the most reliable option, though device dependencies and operational complexity are high.

Network Partition Timeout and Auto-Exclusion

• With Corosync parameters (token, consensus), automatically evict nodes unresponsive for a defined period.
• Ensure VIP always resides on a single node using constraints like migration-threshold or failure-timeout.
• Ignore short-lived glitches; trigger failover only when downtime exceeds a threshold.

Pacemaker Constraints

• Limit MySQL resources in clone mode (one per node).
• Bind VIP with network monitoring resources to reduce split-brain risk.
• Configure on-fail=standby to automatically demote failing nodes.

6. Monitoring Points

• Pacemaker: fencing logs, active VIP node.
• Corosync: ring status, quorum events.
• MySQL: replication delay, dual-master write conflicts.
• VIP: ensure no simultaneous activation across nodes.

7. Prevention Checklist

• For two-node clusters, implement a tie-breaker or auto-exclusion logic.
• STONITH is ideal, but if impractical, network-based exclusion can serve as a fallback.
• VIPs must be configured as strictly single-node resources.
• Regular chaos tests (network partition → recovery → VIP/MySQL behavior validation) are essential.

8. Conclusion

Split brain is not about the partition itself but the conflict at reconnection.
It can occur in real-world HA setups such as MySQL clusters managed by Pacemaker and Corosync.

Mitigation is not a one-size-fits-all solution—it’s a set of options to choose from:

• For environments where complexity is acceptable, use quorum and STONITH-based designs.
• Where constraints are tighter, network partition detection and strict VIP constraints can still provide a degree of protection.

Ultimately, every HA design must include at least one mechanism to prevent split brain—otherwise, the system risks becoming a data corruption engine rather than a high-availability solution.