How to replace a failed high availability cluster node

This article describes the steps necessary to replace a failed node in a high availability (HA) cluster when one of the nodes (either the master or the slave) fails.

AMC Master Node Fails

Environment:
node1: AMC Master (fails)
node2: AMC Slave (alive)

1. Shut down the failed unit (in this case the master node).
2. Log in to AMC on the slave node and click the Assign as Master link.  The slave node is now designated as the AMC Master.
3. Disconnect all network cables from the master node.
4. Replace the master node with a new unit that has no configuration settings.
5. Connect the appropriate network cables to the new master node.
6. Power on the new master node.
7. Run through setup, using either Setup Tool (config_reset) over a serial connection to the console, or Setup Wizard in a Web browser.  If the nodes in your pair should be dual-homed, opt not to join the node into a cluster.
8. Configure the internal network interface address and netmask so that you can get to AMC on this node.
9. During setup, configure the node to join the cluster you have already created.
10. After configuration, log in to AMC and configure the external interface, if your appliances are dual-homed.
11. Install the same set of hotfixes on the master node that are already on the slave node.  Reboot if necessary.
12. On the master node, run Cluster Tool (config_reset) and enter the required information  (cluster interface IP address, subnet, cluster name, node name).
13. Reboot the unit (this occurs automatically at the end of Cluster Tool).
14. Node1 is now the AMC slave, and node2 is the AMC Master. The configuration will synchronize automatically.
15. (Optional) If you want node1 to be AMC Master, shut down node2 and click Assign as AMC Master on node1.  After doing that, power on node2.

AMC Slave Node Fails

Environment:
node1:AMC Master (alive)
node2:AMC Slave (fails)

1. Shut down the failed unit (in this case the slave node).
2. Disconnect all network cables from the slave node.
3. Replace the slave node with new unit. Note: this new unit doesn't have any configuration.
4. Connect all the appropriate network cables to the new slave node.
5. Power on the new slave node.
6. Run through setup, using either Setup Tool (config_reset) over a serial connection to the console, or Setup Wizard in a Web browser. If the nodes in your pair should be dual-homed, choose not to join this node into a cluster.
7. Configure the internal network interface address and netmask so you can get to AMC on this node.
8. During setup, configure the node to join the cluster you have already created.
9. After configuration, log in to AMC and configure the external interface, if your appliances are dual-homed.
10. Install the same set of hotfixes on the slave node that are already on the master node.  Reboot if necessary.
11. On the slave node, run cluster_tool and enter required information (cluster interface IP address, subnet, cluster name, node name).
12. Reboot the unit. (occurs automatically at the end of cluster tool).
13. Both nodes will be synchronized automatically after the reboot.