Lab 01: Managing Taints and Tolerations in Kubernetes

Objectives

• Manage workloads using specialized Node Pools (e.g., GPU vs CPU nodes, or test vs production workloads).
• Configure affinity and anti-affinity rules for pods.
• Use labels and taints to control pod scheduling onto nodes.
• Test deployments to validate pod behavior based on configured rules.

Prerequisites

1. A running Kubernetes cluster provided by OKS.
2. Access to the kubectl CLI.
3. Administrator privileges on the cluster.

Retrieve the kubeconfig file

oks-cli cluster kubeconfig --cluster-name my-cluster --project-name my-project > kubeconfig.yaml
export KUBECONFIG=./kubeconfig.yaml

Verify cluster access

Make sure you have the required permissions to access the target cluster. You must have a user or role that allows you to export or retrieve the cluster configuration.

kubectl get nodes
kubectl describe node <node-name>

This command provides detailed information about the specified Kubernetes node, including labels, taints, capacities, current conditions, and the pods running on it.

Apply a taint to all nodes

kubectl get nodes -o custom-columns=":metadata.name" | xargs -I {} kubectl taint node {} dedicated=workload:NoSchedule

To confirm that taints are applied to each node, use:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Example expected output:

NAME              TAINTS
ip-10-91-35-253   dedicated=workload:NoSchedule
ip-10-91-39-104   dedicated=workload:NoSchedule

Create a Pod without toleration

kubectl run pod-without-toleration --image=nginx --port=80 --dry-run=client -o yaml | kubectl apply -f -

Check Pod status

Use the following command to check the pod status:

kubectl get pods

The pod will be in a Pending state because no available node accepts pods without toleration due to the applied taints.

Remove the taint from a node

Remove the taint from one of the nodes to allow the pod to be scheduled on it.

Command to remove a taint:

kubectl taint nodes <node-name> dedicated:NoSchedule-

Verify taint removal

Confirm that the taint has been removed:

kubectl describe node ip-10-91-35-253 | grep Taints

Verify that the Pod is now deployed

Check the pod status again to confirm it is now running on the node where the taint was removed.

Command:

kubectl get pods -o wide

Conclusion

This lab demonstrated how Kubernetes uses taints and tolerations to control pod placement on specific nodes. When a node is tainted, only pods with matching tolerations can be scheduled onto it.