Restarting a cluster when nodes are disconnected from each other - Adaptive Applications - BlueCat Gateway - 23.2.4

BlueCat Distributed DDNS Administration Guide
Locale
English
Product name
BlueCat Gateway
Version
23.2.4

The following procedure describes how to bootstrap (restart) a Distributed DDNS database cluster when nodes in the desired cluster are running, but are disconnected from each other. This is sometimes called a "split brain" situation.

The bootstrapping process depends on how nodes are distributed as separate clusters.

All nodes are disconnected from each other

When all nodes are disconnected from each other due to a network problem, they all become non-primary nodes that can't be read or written to. If you want to restore DDNS services before the network link is restored, you can make one of the nodes a primary. To do so, run the following commands: 
docker exec -it <node-container-name> mariadb
SET GLOBAL wsrep_provider_options='pc.bootstrap=true';
Tip: When the network links are restored, the other nodes should automatically rejoin.
Attention: Do not set this bootstrap option on multiple nodes. If you do so, you will end up with two active cluster instances, with diverging data sets.

Some nodes remain connected

When some nodes are still connected to each other and functioning as a cluster, we recommend the following:

  • If a network error prevents individual nodes from communicating with the cluster, those nodes will automatically rejoin the cluster when the network connection is restored. No additional work is required.

  • If you ended up with multiple active cluster instances (such as nodes 1 and 2 in one instance, but nodes 3 and 4 on a separate cluster instance), do the following:

    1. Choose the bigger cluster to be the final data cluster.

    2. Stop each node in other clusters. To do so, run the docker stop command for each node: 
      docker stop <Node container name>

    3. After all nodes except for those in the final cluster are stopped, start each stopped node with the docker start command:

      docker start <Node container name>

      Each node will automatically locate and sync with the active data cluster. Make sure you wait for each node to fully start and sync with the rest of the cluster before starting another node.