Upgrading Node-by-Node (OpenStack)

You can upgrade nodes within a K8s cluster one at a time using the node-by-node upgrade process. But this upgrade process is limited only to the node level. You cannot run the cluster synchronization and node-by-node synchronization in tandem. It is possible to run the synchronization on two independent nodes, which has been replaced and are ready to go back to the cluster, simultaneously. Also, you can run parts of the synchronization specific to that node (control plane, etcd, and worker) without running a cluster wide synchronization. You need to drain and synchronize a combination of etcd, control plane and worker nodes to begin with the upgrade process.

To upgrade node-by-node, use the following configurations:

  1. Drain the node.

    clusters cluster_name nodes node_name actions sync drain 

    Example:

    SMI Cluster Manager# clusters test1 nodes etcd1 actions sync drain 
    
  2. Verify the status of the drained node.

    clusters cluster_name nodes node_name actions sync status 

    Example:

    SMI Cluster Manager# clusters test1 nodes etcd1 actions sync status 
    
  3. Verify the pod status of the drained node.

    clusters cluster_name nodes node_name actions k8s pod-status 

    Example:

    SMI Cluster Manager# clusters test1 nodes etcd1 actions k8s pod-status 
    
  4. Run a synchronization on the node.

    clusters cluster_name nodes node_name actions sync run 

    Example:

    SMI Cluster Manager# clusters test1 nodes etcd1 actions sync run 
    
    Note

    The host OS is upgraded during cluster synchronization automatically on Bare Metal deployments.

  5. Verify the cluster status. Ensure that the parameter all-ok is true before proceeding to the next node.

    clusters cluster_name actions k8s cluster-status 

    Example:

    SMI Cluster Manager# clusters test1 actions k8s cluster-status 
    	pods-desired-count 26 
    	pods-ready-count 26 
    	pods-desired-are-ready true 
    	etcd-healthy true 
    	all-ok true 
    
  6. Verify the status of the pods on a specific node before proceeding to the next node.

    clusters cluster_name nodes node_name actions k8s pod-status show-pod-details 

    Example:

    SMI Cluster Manager# clusters test1 nodes cmts-worker1 actions k8s pod-status show-pod-details 
    	Value for 'show-pod-details' [false,true]: true 
    	pods { 
    		name calico-node-c65zf 
    		namespace kube-system 
    		owner-kind DaemonSet 
    		owner-name calico-node 
    		ready true 
    	} 
    	pods { 
    		name coredns-6db4464669-k6pqz 
    		namespace kube-system 
    		owner-kind ReplicaSet 
    		owner-name coredns-6db4464669 
    		ready true 
    	} 
    	pods { 
    		name kube-proxy-tfxcq 
    		namespace kube-system 
    		owner-kind DaemonSet 
    		owner-name kube-proxy 
    		ready true 
    	} 
    	pods { 
    		name nginx-ingress-controller-6f8f8c4cc7-q5b7c 
    		namespace nginx-ingress 
    		owner-kind ReplicaSet 
    		owner-name nginx-ingress-controller-6f8f8c4cc7 
    		ready true 
    	} 
    	pods { 
    		name keepalived-cmbnf 
    		namespace smi-vips 
    		owner-kind DaemonSet 
    		owner-name keepalived 
    		ready true 
    	} 
    pods-count 5 
    pods-available-to-drain-count 2 
    

Notes:

  • nodes node_name – Specifies the nodes present in the cluster. node_name is the name of the node.

  • sync drain – Drains or cordons the selected node in preparation for an upgrade.

  • sync status – Shows the status of the drained node.

  • k8s pod-status – Shows the status of the k8s pods scheduled on the node.

  • sync run – Upgrades or synchronizes the node.

  • k8s cluster-status – Shows an overall status of the cluster including pod and etcd based statistics.

  • show-pod-details true – Shows the list of pods in addition to the counts.