EKS 1.22 Upgrade Tutorial

👋 Welcome to my upgrade guide to EKS version 1.22.

EKS has recently started supporting version 1.22 of Kubernetes. If you want to upgrade, you should know this: Version 1.22 has LOTS of REMOVED APIs. Previously they were deprecated, now they are outright REMOVED, so any workload trying to run using those removed versions will simply fail to run. This means that this upgrade is very dangerous for your production environment.

NEW! Now you can use Datree to prevent the use of deprecated APIs in your resources. Find the policy here.

In this tutorial we will learn how to upgrade correctly. We will:

1. Identify the removed APIs in the target cluster.
2. Convert the removed APIs to new versions.
3. Upgrade the EKS Control Plane to version 1.22.
4. Upgrade Node Groups to AMI version 1.22.6.

⏳Identify the removed APIs in the target cluster

So let’s scan  our cluster to see if we are currently running any of those removed APIs. You can find the full list of removed APIs at the bottom of this blog post.

First, we will fetch all of the manifests for the following namespace hn-app and output it as YAML

kubectl get all -n hn-app -o yaml|grep apiVersion

Now, in order to find those removed APIs we have two options. One, manually cross-referencing the output with the list of removed APIs. Or two, using Datree to automatically identify those removed APIs.

Datree is an open source project built by me and by a bunch of talented engineers in order to prevent misconfigurations in Kubernetes environments, and it has a Kubectl plugin that can come in handy in this case. The plugin connects to our kubectl configured cluster, pulls the currently running manifests, and checks to see if their version is compatible with the future version that we will upgrade our cluster to.

In order to use the plugin we will first install it using Krew (the package manager for Kubectl plugins)

kubectl krew install datree

Now we will scan the hn-app workspace using the Datree with the target version 1.22.6

kubectl datree test -s 1.22.6 -- --namespace hn-app

And here are the results:

As you can see, we have an issue with our Ingress controller as it is using the networking.k8s.io/v1beta1 version.

And indeed, if we’ll check the manifest we’ll see the removed API in line #2:

# Output
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
 annotations:
   alb.ingress.kubernetes.io/scheme: internet-facing
   alb.ingress.kubernetes.io/target-type: ip
   kubernetes.io/ingress.class: alb
 creationTimestamp: null
 labels:
   app: hn-ingress
 name: hn-ingress
 namespace: hn-app
spec:
 rules:
 - host: hn.datree.io
 - http:
     paths:
     - backend:
         serviceName: service-hn
         servicePort: 80
       path: /
       pathType: Prefix
status:
 loadBalancer: {}

If we try to use this API Version after we’ve upgraded the cluster to version 1.22 we will get the following error:

error: unable to recognize "hn-ingress-2.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"

🔁 Converting the removed API versions to new versions

Now that we have identified the removed API version using Datree’s Kubectl plugin, we will use an official Kubernetes kubectl plugin called convert in order to migrate our API version

#installing kubectl convert on linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl-convert" && sudo install -o root -g root -m 0755 kubectl-convert /usr/local/bin/kubectl-convert

#installing kubectl convert on macOS
# Intel
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl-convert" && chmod +x ./kubectl-convert
# Apple Sillicon
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl-convert" && chmod +x ./kubectl-convert

sudo mv ./kubectl-convert /usr/local/bin/kubectl-convert && sudo chown root: /usr/local/bin/kubectl-convert

Now let's convert the file to the new API networking.k8s.io/v1

kubectl convert -f hn-ingress-2.yaml --output-version networking.k8s.io/v1

#Output
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
   alb.ingress.kubernetes.io/scheme: internet-facing
   alb.ingress.kubernetes.io/target-type: ip
   kubernetes.io/ingress.class: alb
 creationTimestamp: null
 labels:
   app: hn-ingress
 name: hn-ingress
 namespace: hn-app
spec:
 rules:
 - host: hn.datree.io
 - http:
     paths:
     - backend:
         service:
           name: service-hn
           port:
             number: 80
       path: /
       pathType: Prefix
status:
 loadBalancer: {}

Now we have the up-to-date and supported API versions! 🎉 We are safe to perform the upgrade 🔥

🤞 Upgrading the EKS cluster:

In order to perform the upgrade process for the Control Plane and Node Groups, we will use the EKSCTL toolkit

Upgrading the Control Plane:

eksctl upgrade cluster -r us-east-1 -n k8s21to22d-k8sdemo4 --approve

Upgrading the node group:

eksctl upgrade nodegroup -r us-east-1 -c k8s21to22d-k8sdemo4 --name k8s21to22d-k8sdemo4-node_group

⛔️ Removed APIs:

• Beta versions of the ValidatingWebhookConfiguration and MutatingWebhookConfiguration API (the admissionregistration.k8s.io/v1beta1 API versions)
  error: unable to recognize "deployment.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "apiregistration.k8s.io/v1beta1"
  error: unable to recognize "deployment.yaml": no matches for kind "MutatingWebhookConfiguration" in version "apiregistration.k8s.io/v1beta1"‍
• The beta CustomResourceDefinition API (apiextensions.k8s.io/v1beta1)
  error: unable to recognize "deployment.yaml": no matches for kind "CustomResourceDefinition" in version "apiregistration.k8s.io/v1beta1"‍
• The beta APIService API (apiregistration.k8s.io/v1beta1)
  error: unable to recognize "deployment.yaml": no matches for kind "APIService" in version "apiregistration.k8s.io/v1beta1"‍
• The beta TokenReview API (authentication.k8s.io/v1beta1)
  error: unable to recognize "deployment.yaml": no matches for kind "TokenReview" in version "networking.k8s.io/v1beta1"‍
• Beta API versions of SubjectAccessReview, LocalSubjectAccessReview, SelfSubjectAccessReview (API versions from authorization.k8s.io/v1beta1)
  error: unable to recognize "deployment.yaml": no matches for kind "SubjectAccessReview" in version "networking.k8s.io/v1beta1"
  error: unable to recognize "deployment.yaml": no matches for kind "LocalSubjectAccessReview" in version "networking.k8s.io/v1beta1
  error: unable to recognize "deployment.yaml": no matches for kind "SelfSubjectAccessReview" in version "networking.k8s.io/v1beta1"‍
• The beta CertificateSigningRequest API (certificates.k8s.io/v1beta1)
  error: unable to recognize "deployment.yaml": no matches for kind "CertificateSigningRequest" in version "networking.k8s.io/v1beta1"‍
• The beta Lease API (coordination.k8s.io/v1beta1)
  error: unable to recognize "deployment.yaml": no matches for kind "Lease" in version "coordination.k8s.io/v1beta1"‍
• All beta Ingress APIs (the extensions/v1beta1 and networking.k8s.io/v1beta1 API versions)
  error: unable to recognize "deployment.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
  error: unable to recognize "deployment.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"

Additional Resources:

1. https://sysdig.com/blog/kubernetes-1-22-whats-new/
2. https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/
3. https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/
4. https://kubernetes.io/docs/reference/using-api/deprecation-guide/
5. https://youtu.be/FVGEvbLX46M