Upgrading Node-by-Node (OpenStack)
You can upgrade nodes within a K8s cluster one at a time using the node-by-node upgrade process. But this upgrade process is limited only to the node level. You cannot run the cluster synchronization and node-by-node synchronization in tandem. It is possible to run the synchronization on two independent nodes, which has been replaced and are ready to go back to the cluster, simultaneously. Also, you can run parts of the synchronization specific to that node (control plane, etcd, and worker) without running a cluster wide synchronization. You need to drain and synchronize a combination of etcd, control plane and worker nodes to begin with the upgrade process.
To upgrade node-by-node, use the following configurations:
-
Drain the node.
clusters cluster_name nodes node_name actions sync drain
Example:
SMI Cluster Manager# clusters test1 nodes etcd1 actions sync drain
-
Verify the status of the drained node.
clusters cluster_name nodes node_name actions sync status
Example:
SMI Cluster Manager# clusters test1 nodes etcd1 actions sync status
-
Verify the pod status of the drained node.
clusters cluster_name nodes node_name actions k8s pod-status
Example:
SMI Cluster Manager# clusters test1 nodes etcd1 actions k8s pod-status
-
Run a synchronization on the node.
clusters cluster_name nodes node_name actions sync run
Example:
SMI Cluster Manager# clusters test1 nodes etcd1 actions sync run
NoteThe host OS is upgraded during cluster synchronization automatically on Bare Metal deployments.
-
Verify the cluster status. Ensure that the parameter all-ok is true before proceeding to the next node.
clusters cluster_name actions k8s cluster-status
Example:
SMI Cluster Manager# clusters test1 actions k8s cluster-status pods-desired-count 26 pods-ready-count 26 pods-desired-are-ready true etcd-healthy true all-ok true
-
Verify the status of the pods on a specific node before proceeding to the next node.
clusters cluster_name nodes node_name actions k8s pod-status show-pod-details
Example:
SMI Cluster Manager# clusters test1 nodes cmts-worker1 actions k8s pod-status show-pod-details Value for 'show-pod-details' [false,true]: true pods { name calico-node-c65zf namespace kube-system owner-kind DaemonSet owner-name calico-node ready true } pods { name coredns-6db4464669-k6pqz namespace kube-system owner-kind ReplicaSet owner-name coredns-6db4464669 ready true } pods { name kube-proxy-tfxcq namespace kube-system owner-kind DaemonSet owner-name kube-proxy ready true } pods { name nginx-ingress-controller-6f8f8c4cc7-q5b7c namespace nginx-ingress owner-kind ReplicaSet owner-name nginx-ingress-controller-6f8f8c4cc7 ready true } pods { name keepalived-cmbnf namespace smi-vips owner-kind DaemonSet owner-name keepalived ready true } pods-count 5 pods-available-to-drain-count 2
Notes:
-
nodes node_name – Specifies the nodes present in the cluster. node_name is the name of the node.
-
sync drain – Drains or cordons the selected node in preparation for an upgrade.
-
sync status – Shows the status of the drained node.
-
k8s pod-status – Shows the status of the k8s pods scheduled on the node.
-
sync run – Upgrades or synchronizes the node.
-
k8s cluster-status – Shows an overall status of the cluster including pod and etcd based statistics.
-
show-pod-details true – Shows the list of pods in addition to the counts.