Kubernetes Troubleshooting - Fixing Error CreateContainerConfig

{{text-cta}}

CreateContainerConfigError as the name suggests there is a problem with the kubernetes container when it's going from Pending to Successful Running state. But to be precise you are missing Kubernetes Secrets in the deployment manifest. 

If the kubernetes Pod is in Pending state which means -

1. The Pod has been accepted by the kubernetes cluster

2. Docker container image has been downloaded and ready to run

But before the container goes into the Running state it goes through the deployment configuration and validates all the necessary configurations which are needed for the successful deployment of the docker container. 

So during this validation phase it tries to find the secretKeyRef with the name, key but it could not find those secrets available inside the kubernetes cluster leading to CreateContainerConfigError

In this guide first we are going to replicate the issue on the development kubernetes cluster and later on we will create the kubernetes secret to fix it. Here are the steps -

1. Replicate the error CreateContainerConfigError
2. How to Check the container logs for finding the missing Kubernetes Secret
3. Create and apply the correct kubernetes secret
4. Re-run the kubernetes deployment after fixing the kubernetes secret
5. Conclusion

1. Replicate the error CreateContainerConfigError?

To understand this issue, we are going to replicate this issue by deploying a mysql docker container inside the kubernetes cluster. 

Why mysql docker container? - Because to setup a mysql you need to provide an username and password which can be stored inside kubernetes secret.

List of  kubernetes artifacts needed for deploying mysql:

1. Persistent Volume (test-pv.yaml)
2. Persistent Claim (test-pvc.yaml)
3. Kubernetes Secret (test-secrets.yaml) 
4. Deployment (test-deployment.yaml)
   

(Note*- To replicate the issue we are going to skip the creation of Kubernetes secret)

Persistent Volume

First create the persistent volume with the storage of  1Gi.

Here is the configuration for persistent volume which can saved as a test-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
 name: test-pv
spec:
 capacity:
   storage: 1Gi
 volumeMode: Filesystem
 accessModes:
   - ReadWriteOnce
 persistentVolumeReclaimPolicy: Retain
 storageClassName: local-storage
 local:
   path: /home/vagrant/storage
 nodeAffinity:
   required:
     nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
               - node1

Apply the above configuration by running the following command:

$ kubectl apply -f test-pv.yaml
persistentvolume/test-pv created

Verify the status of persistent volume by running the following command:

$ kubectl get pv

NAME  	  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS  	CLAIM   STORAGECLASS   REASON   AGE
test-pv   1Gi        RWO            Retain           Available          local-storage           84s

Persistent Volume claim

After creating the persistent Volume claim, let us create a persistent volume claim for the test-pv which we have created in the previous step.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: test-pvc
spec:
 volumeName: test-pv
 storageClassName: local-storage
 volumeMode: Filesystem
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: 1Gi

$ kubectl apply -f test-pvc.yaml
persistentvolumeclaim/jhooq-pvc created

Verify the status of the persistent volume claim(test-pvc) by running the following command:

$ kubectl get pvc
NAME   	   STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS    AGE
test-pvc   Bound    test-pv   1Gi        RWO            local-storage   113s

{{text-cta}}

Kubernetes Secret

We are not going to create the kubernetes secret inside the kubernetes cluster. Because we want to replicate the CreateContainerConfigError.

Deployment

At last create the deployment for mysql docker container image. Use the following deployment manifest (test-deployment.yaml).

apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
spec:
 selector:
   matchLabels:
     app: mysql
 strategy:
   type: Recreate
 template:
   metadata:
     labels:
       app: mysql
   spec:
     containers:
     - image: mysql:5.6
       name: mysql
       env:
       - name: MYSQL_ROOT_PASSWORD
         valueFrom:
           secretKeyRef:
             name: mysql-secret
             key: password
       ports:
       - containerPort: 3306
         name: mysql
       volumeMounts:
       - name: mysql-persistent-storage
         mountPath: /home/vagrant/storage
     volumes:
     - name: mysql-persistent-storage
       persistentVolumeClaim:
         claimName: test-pvc

Apply the above deployment configuration by running the following kubectl command:

$ kubectl apply -f test-deployment.yaml
deployment.apps/mysql created

Verify the status of the pod using kubectl get all command:

$ kubectl get all
NAME                        READY   STATUS                       RESTARTS   AGE
pod/mysql-95d6b45b5-n4lz5   0/1     CreateContainerConfigError   0          3m7s

NAM                  TYPE        CLUSTER-IP   EXTERNAL-IP       PORT(S)   AGE
service/kubernetes   ClusterIP   10.233.0.1   <none>    	443/TCP   25d

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   0/1     1            0           3m7s

NAME                          	  DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-95d6b45b5   1         1         0       3m7s

2. How to Check the container logs for finding the missing Kubernetes Secret

Now in the Step-1 we replicated the CreateContainerConfigError, but to find the exact root cause we need to check the POD logs.

Run the following command to list all the PODs running inside the kubernetes cluster.

$ kubectl get pod
NAME                    READY   STATUS                       RESTARTS   AGE
mysql-95d6b45b5-8bcpz   0/1     CreateContainerConfigError   0          114s

Copy the container POD name from the above output .i.e. - mysql-95d6b45b5-8bcpz

Run the kubectl describe command with POD name as follow -

$ kubectl describe pod mysql-95d6b45b5-8bcpz

The above mentioned describe command will show a very long log file but carefully scroll down to the bottom of the log and look for following message

Events:
  Type     Reason          Age                     From                   Message
  ---- 	   ------      	   ----                	   ----           	  -------
  Normal   Scheduled   	   6m37s               	   default-scheduler      Successfully assigned default/mysql-95d6b45b5-8bcpz to node1
  Normal   SandboxChanged  6m36s               	   kubelet        	  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed      	   4m53s (x12 over 6m37s)  kubelet        	  Error: secret "mysql-secret" not found
  Normal   Pulled      	   90s (x28 over 6m37s)	   kubelet        	  Container image "mysql:5.6" already present on machine

You can easily identify the error “mysql-secret” not found and due to this error POD was not getting deployed and resulting in CreateContainerConfigError.

3. Create and apply the correct kubernetes secret

Let’s create the kubernetes secret (test-secret.yaml) for mysql and apply it.

apiVersion: v1
kind: Secret
metadata:
 name: mysql-secret
type: kubernetes.io/basic-auth
stringData:
 password: test1234

Apply the above test-secret.yaml configuration with kubectl command:

$ kubectl apply -f test-secret.yaml
secret/mysql-secret created

Verify the secret by running the following kubectl command:

$ kubectl get secret mysql-secret
NAME           TYPE                       DATA   AGE
mysql-secret   kubernetes.io/basic-auth   1      17s

4.Re-run the kubernetes deployment after fixing the kubernetes secret

Now we have fixed the kubernetes secret, the next step would be delete the old kubernetes deployment of mysql and re-deploy it.

Run the following kubectl command to delete the old deployment mysql:

$ kubectl delete deployment mysql
deployment.apps "mysql" deleted

Re-deploy the mysql deployment (test-deployment.yaml) manifest again:

$ kubectl apply -f test-deployment.yaml
deployment.apps/mysql created

Check the Pod status again after the re-deployment:

$ kubectl get pod
NAME                      READY   STATUS    RESTARTS   AGE
mysql-95d6b45b5-59fcx     1/1     Running   0          98s

Check the Pod logs for using the kubectl describe:

$ kubectl describe pod mysql-95d6b45b5-jxzb9

5. Conclusion

Always start the debugging of the CreateContainerConfigError by looking into the POD logs using the kubectl describe Pod command because those POD logs will help you to identify the name of the missing kubernetes secret inside the kubernetes cluster.