Aller au contenu

Lab 08: Upgrading an OKS Cluster

By the end of this lab, you will be able to:

  • Identify the current Kubernetes version of an OKS cluster
  • Identify the next available version for the upgrade
  • Upgrade the Control Plane of an OKS cluster
  • Upgrade the Worker Nodes
  • Understand the upgrade order of the different cluster components
  • Verify the cluster status before and after the upgrade

Before starting this lab, make sure you have the following:

Important: a cluster upgrade must be performed one minor version at a time.

For example, to upgrade Kubernetes from 1.32 to 1.34, the upgrades must be performed successively:

1.32 → 1.33 → 1.34

For each version, the Control Plane is upgraded first, then the Worker Nodes must be upgraded to the same version before moving to the next minor version.

Before starting the upgrade, check the Kubernetes version currently used by the cluster.

Run:

Terminal window
kubectl version

Example output:

Terminal window
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.32.13

The Server Version field corresponds to the Kubernetes version of the Control Plane.

In this example, the Control Plane is running Kubernetes 1.32.13.

Then check the status and version of the Worker Nodes:

Terminal window
kubectl get nodes -o wide

Example:

Terminal window
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
ip-10-50-44-36 Ready <none> 38m v1.32.13 10.50.44.36 <none>
ip-10-50-62-191 Ready <none> 60m v1.32.13 10.50.62.191 <none>

Before proceeding, make sure that:

  • all Worker Nodes are in the Ready state
  • the Control Plane and Worker Nodes are using the same Kubernetes minor version

In our example:

Control Plane : v1.32.13
Worker 1 : v1.32.13
Worker 2 : v1.32.13

The cluster is ready for the upgrade.

The upgrade starts with the Control Plane.

The Worker Nodes will be upgraded in a second step.

Retrieve the OKS cluster information:

Terminal window
oks-cli --profile <PROFILE> cluster get \
-p <PROJECT_NAME> \
-c <CLUSTER_NAME>

In the output, look for the available_upgrade field:

"statuses": {
"status": "ready",
"available_upgrade": "1.33"
}

The available_upgrade field indicates the next minor version available for the cluster upgrade.

In this example:

Current version : 1.32
Available version : 1.33

Start the upgrade to the version indicated by available_upgrade:

Terminal window
oks-cli --profile <PROFILE> cluster update \
-p <PROJECT_NAME> \
-c <CLUSTER_NAME> \
--version <VERSION>

For example, to upgrade Kubernetes from 1.32 to 1.33:

Terminal window
oks-cli --profile <PROFILE> cluster update \
-p <PROJECT_NAME> \
-c <CLUSTER_NAME> \
--version 1.33

Only the major.minor version is specified.

OKS automatically selects the corresponding available patch version.

For example:

1.32.13 → 1.33.13

During the upgrade, regularly check the cluster status:

Terminal window
oks-cli --profile <PROFILE> cluster list \
-p <PROJECT_NAME>

Example:

+-----------+--------------------+---------------------+---------------+---------------+----------+
| CLUSTER | PROFILE | REGION | CREATED | UPDATED | STATUS |
+-----------+--------------------+---------------------+---------------+---------------+----------+
| cluster02 | default | cloudgouv-eu-west-1 | an hour ago | a minute ago | ready |
+-----------+--------------------+---------------------+---------------+---------------+----------+

Wait until the cluster returns to the following state:

ready

Then check the cluster information:

Terminal window
oks-cli --profile <PROFILE> cluster get \
-p <PROJECT_NAME> \
-c <CLUSTER_NAME>

Relevant output:

{
"version": "1.33",
"expected_version": "1.33",
"statuses": {
"status": "ready",
"available_upgrade": "1.34"
}
}

The OKS cluster is now using version 1.33, and version 1.34 is offered as the next upgrade.

Do not immediately upgrade to the next version. The Worker Nodes must first be upgraded to the same version as the Control Plane.

After upgrading the Control Plane, refresh the kubeconfig to retrieve an up-to-date access configuration:

Terminal window
oks-cli --profile <PROFILE> cluster kubeconfig \
-p <PROJECT_NAME> \
-c <CLUSTER_NAME> \
--refresh \
--ttl <TTL>

Then verify that access to the cluster works:

Terminal window
kubectl version

Example:

Terminal window
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.33.13

The Server Version field confirms that the Control Plane is now running Kubernetes 1.33.13.

At this stage, the Control Plane is using the new Kubernetes version, but the Worker Nodes may still be using the previous version.

Check their status:

Terminal window
kubectl get nodes

Example:

Terminal window
NAME STATUS ROLES AGE VERSION
ip-10-50-44-36 Ready <none> 38m v1.32.13
ip-10-50-62-191 Ready <none> 60m v1.32.13

We temporarily have two Kubernetes versions in the cluster:

Control Plane : v1.33.13
Worker 1 : v1.32.13
Worker 2 : v1.32.13

This state is temporary.

The Worker Nodes must now be upgraded to the same version as the Control Plane.

The manual upgrade of the Worker Nodes is performed one node at a time.

For each Worker Node:

  1. the node is drained.
  2. the pods running on the node are evicted.
  3. the node is deleted from the cluster.
  4. the NodePool automatically creates a new Worker Node.
  5. the new Worker joins the cluster with the new Kubernetes version.

Display the Worker Nodes:

Terminal window
kubectl get nodes

Example:

Terminal window
NAME STATUS ROLES AGE VERSION
ip-10-50-44-36 Ready <none> 38m v1.32.13
ip-10-50-62-191 Ready <none> 60m v1.32.13

Choose the first Worker Node to upgrade.

Before deleting a Worker Node, it must be drained.

Draining prevents new pods from being scheduled on the node and evicts workloads that can be moved.

Run:

Terminal window
kubectl drain <NODE_NAME> \
--ignore-daemonsets \
--delete-emptydir-data

For example:

Terminal window
kubectl drain ip-10-50-44-36 \
--ignore-daemonsets \
--delete-emptydir-data

Example output:

Terminal window
node/ip-10-50-44-36 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/cilium-envoy-xxxxx, kube-system/cilium-xxxxx, ...
node/ip-10-50-44-36 drained

The --ignore-daemonsets option allows pods managed by DaemonSets to be ignored during the drain.

The --delete-emptydir-data option allows the eviction of pods using emptyDir volumes.

Then check the node status:

Terminal window
kubectl get nodes

Example:

Terminal window
NAME STATUS ROLES AGE VERSION
ip-10-50-44-36 Ready,SchedulingDisabled <none> 39m v1.32.13
ip-10-50-62-191 Ready <none> 61m v1.32.13

The SchedulingDisabled status indicates that the node has been cordoned and no longer accepts new pods.

Only delete the Worker Node once the drain has completed successfully.

Once the node has been successfully drained, delete it from the cluster:

Terminal window
kubectl delete node <NODE_NAME>

For example:

Terminal window
kubectl delete node ip-10-50-44-36

The NodePool detects that the Worker has disappeared and automatically creates a new node to replace it.

Monitor the Worker Nodes:

Terminal window
kubectl get nodes -w

A new Worker Node should gradually join the cluster.

Once the new node is in the Ready state, exit watch mode with Ctrl+C.

Then check the versions:

Terminal window
kubectl get nodes

Example:

Terminal window
NAME STATUS ROLES AGE VERSION
ip-10-50-36-2 Ready <none> 31s v1.33.13
ip-10-50-62-191 Ready <none> 64m v1.32.13

The new Worker is now using the same Kubernetes version as the Control Plane:

Control Plane : v1.33.13
Worker 1 : v1.33.13
Worker 2 : v1.32.13

The first Worker has been successfully upgraded.

Repeat the same procedure for each Worker that is still using the previous Kubernetes version.

Start by identifying the remaining nodes:

Terminal window
kubectl get nodes

For each Worker concerned, perform the following steps:

Terminal window
kubectl drain <NODE_NAME> \
--ignore-daemonsets \
--delete-emptydir-data

Verify that the drain completes successfully, then delete the node:

Terminal window
kubectl delete node <NODE_NAME>

Then wait for its replacement to be created:

Terminal window
kubectl get nodes -w

Before moving to the next Worker, verify that the new node:

  • is in the Ready state
  • is using the same Kubernetes version as the Control Plane

Worker Nodes must be replaced one at a time. Wait until the new Worker is operational before starting the upgrade of the next one.

Once all Worker Nodes have been replaced, check the Control Plane version:

Terminal window
kubectl version

Example:

Terminal window
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.33.13

Then check the Worker Nodes:

Terminal window
kubectl get nodes -o wide

Example:

Terminal window
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
ip-10-50-36-2 Ready <none> 4m56s v1.33.13 10.50.36.2 <none>
ip-10-50-51-19 Ready <none> 56s v1.33.13 10.50.51.19 <none>

The Control Plane and all Worker Nodes must now be using the same Kubernetes version:

Control Plane : v1.33.13
Worker 1 : v1.33.13
Worker 2 : v1.33.13

Finally, check the status of the cluster components:

Terminal window
kubectl get pods -A

Make sure that the system pods are operational and that no component remains in an abnormal state.

The upgrade is complete when:

  • the Control Plane is using the new Kubernetes version
  • all Worker Nodes are using the same version
  • all Worker Nodes are in the Ready state
  • the Kubernetes cluster components are operational

If multiple minor versions need to be applied, repeat the entire procedure for each version.

For example, to upgrade Kubernetes from 1.32 to 1.35:

1.32
↓
Control Plane 1.33
↓
Worker Nodes 1.33
↓
Control Plane 1.34
↓
Worker Nodes 1.34
↓
Control Plane 1.35
↓
Worker Nodes 1.35

Never start the upgrade to the next minor version until the Worker Nodes have reached the same version as the Control Plane.