Worker Bare Metal Node Maintenance - Planned

This section describes the procedures involved in replacing a working primary worker Bare Metal node for maintenance. To replace a primary worker Bare Metal node for maintenance, use the following:

  1. Drain and remove the worker node which is sent for maintenance, using the following command:

    clusters cluster_name nodes worker_node actions sync drain remove-node true 

    Example:

    [installer-controlplane] SMI Cluster Deployer# clusters kali-stacked nodes cmts-worker1 actions sync drain remove-node true 
    This will run drain on the node, disrupting pods running on the node.  Are you sure? [no,yes] yes
    message accepted
  2. Verify the status of the worker node using the following command:

    clusters cluster_name nodes worker_node actions sync logs 

    Example:

    [installer-controlplane] SMI Cluster Deployer# clusters kali-stacked nodes cmts-worker1 actions sync logs 
    logs 2020-10-06 20:01:48.023 DEBUG cluster_sync.kali-stacked.cmts-worker1: Cluster name: kali-stacked 
    2020-10-06 20:01:48.024 DEBUG cluster_sync.kali-stacked.cmts-worker1: Node name: cmts-worker1 
    2020-10-06 20:01:48.024 DEBUG cluster_sync.kali-stacked.cmts-worker1: debug: false 
    2020-10-06 20:01:48.024 DEBUG cluster_sync.kali-stacked.cmts-worker1: remove_node: true 
    .
    .
    .
    .
    2020-10-06 20:02:30.057 DEBUG cluster_sync.kali-stacked.cmts-worker1: Cluster sync successful 
    2020-10-06 20:02:30.058 DEBUG cluster_sync.kali-stacked.cmts-worker1: Ansible sync done 
    2020-10-06 20:02:30.058 INFO cluster_sync.kali-stacked.cmts-worker1: _sync finished.  Opening lock 
  3. Shutdown the node.

  4. Assign the primary control plane Bare Metal node to maintenance mode in the cluster configuration using the following commands:

    configure 
      clusters cluster_name 
      nodes worker_node 
      maintenance true 
      commit 
      end 

    Example:

    [installer-controlplane] SMI Cluster Deployer# config 
    Entering configuration mode terminal
    [installer-controlplane] SMI Cluster Deployer(config)# clusters kali-stacked 
    [installer-controlplane] SMI Cluster Deployer(config-clusters-kali-stacked)# nodes cmts-worker1 
    [installer-controlplane] SMI Cluster Deployer(config-nodes-cmts-worker1)# maintenance true 
    [installer-controlplane] SMI Cluster Deployer(config-nodes-cmts-worker1)# commit
    Commit complete.
    [installer-controlplane] SMI Cluster Deployer(config-nodes-cmts-worker1)# end
  5. The node is ready for the RMA process.

    Note

    If the remaining nodes need to be upgraded or NFs need to be synchronized, run a cluster sync in this state. However, it's not a part of the RMA process.

  6. Add the node back to the cluster when it is repaired or replaced and available.

    Note

    If you add a node after it's repaired, ensure that the disks are clean by clearing the boot drive and virtual drive on the node. This step is to ensure that the virtual drive is in a clean state without the previous state before you add it back. However, removal of the virtual drive is not required for a new replacement node.

  7. Attach the new primary worker Bare Metal node and remove it from the maintenance mode in the cluster configuration using the following commands:

    configure 
      clusters cluster_name 
      nodes worker_node 
      maintenance false 
      commit 
      end 

    Example:

    SMI Cluster Deployer# config 
    Entering configuration mode terminal
    [installer-controlplane] SMI Cluster Deployer(config)# clusters kali-stacked 
    [installer-controlplane] SMI Cluster Deployer(config-clusters-kali-stacked)# nodes cmts-worker1 
    [installer-controlplane] SMI Cluster Deployer(config-nodes-cmts-worker1)# maintenance false 
    [installer-controlplane] SMI Cluster Deployer(config-nodes-cmts-worker1)# commit
    Commit complete.
    [installer-controlplane] SMI Cluster Deployer(config-nodes-cmts-worker1)# end
  8. Run the cluster synchronization using the following command:

    clusters cluster_name actions sync run debug true 

    Example:

    SMI Cluster Deployer# clusters kali-stacked actions sync run debug true
    This will run sync.  Are you sure? [no,yes] yes
    message accepted

NOTES:

  • clusters cluster_name - Specifies the K8s cluster.

  • nodes worker - Specifies primary worker Bare Metal node.

  • maintenance true/false - Assigns or removes the primary control plane 1 Bare Metal mode to maintenance mode

  • actions sync run debug true - Synchronizes the cluster configuration.

  • actions k8s cluster-status - Displays the status of the cluster.