Active Bare Metal Node Failure - Unplanned

When an Active node fails in a Cluster Manager HA model, the Standby node becomes the Active node and continues to upgrade the remote clusters or the NFs. To replace the failed Active node:

  1. Remove the Active node.

    Note

    Upgrade or synchronize the remote clusters using the Standby node.

    For more information on upgrading the SMI Cluster Manager in HA mode, see Upgrading SMI Cluster Manager in HA section in UCC SMI Cluster Manager Deployment Guide.

  2. Update the Cluster Manager HA configuration to swap the backup and control-plane node types when the Active node is up and running.

    nodes active 
      k8s node-type backup #originally this would be control-plane type in the fresh deployment 
     exit 
     nodes standby 
      k8s node-type control-plane #originally this would be backup type in fresh deployment  
    exit 
  3. Run the cluster synchronization.

    clusters cluster_name actions sync run debug true 

    Example:

    SMI Cluster Deployer# clusters trysjc-ha actions sync run debug true
    This will run sync.  Are you sure? [no,yes] yes
    message accepted
  4. Verify the status of the cluster and pods when the Active and Standby nodes are up and running.

    clusters cluster_name actions k8s cluster-status 
    kubectl get pods -n namespace 
  5. Log in to the Ops Center service and verify whether the cluster configuration and CEE data are retained.

  6. Once the active node is back from maintenance, update the Cluster Manager HA configuration to swap the backup and control-plane node types.

    nodes active 
      k8s node-type control-plane #originally this would be control-plane type in the fresh deployment 
     exit 
     nodes standby 
      k8s node-type backup #originally this would be backup type in fresh deployment  
    exit 

NOTES:

  • clusters cluster_name - Specifies the K8s cluster.

  • actions sync run debug true - Synchronizes the cluster configuration.

  • actions k8s cluster-status - Displays the status of the cluster.