CreateContainerConfigError as the name suggests there is a problem with the kubernetes container when it's going from Pending to Successful Running state. But to be precise you are missing Kubernetes Secrets in the deployment manifest.
If the kubernetes Pod is in Pending state which means -
1. The Pod has been accepted by the kubernetes cluster
2. Docker container image has been downloaded and ready to run
But before the container goes into the Running state it goes through the deployment configuration and validates all the necessary configurations which are needed for the successful deployment of the docker container.
So during this validation phase it tries to find the secretKeyRef with the name, key but it could not find those secrets available inside the kubernetes cluster leading to CreateContainerConfigError
In this guide first we are going to replicate the issue on the development kubernetes cluster and later on we will create the kubernetes secret to fix it. Here are the steps -
- Replicate the error CreateContainerConfigError
- How to Check the container logs for finding the missing Kubernetes Secret
- Create and apply the correct kubernetes secret
- Re-run the kubernetes deployment after fixing the kubernetes secret
1. Replicate the error CreateContainerConfigError?
To understand this issue, we are going to replicate this issue by deploying a mysql docker container inside the kubernetes cluster.
Why mysql docker container? - Because to setup a mysql you need to provide an username and password which can be stored inside kubernetes secret.
List of kubernetes artifacts needed for deploying mysql:
- Persistent Volume (test-pv.yaml)
- Persistent Claim (test-pvc.yaml)
- Kubernetes Secret (test-secrets.yaml)
- Deployment (test-deployment.yaml)
(Note*- To replicate the issue we are going to skip the creation of Kubernetes secret)
First create the persistent volume with the storage of 1Gi.
Here is the configuration for persistent volume which can saved as a test-pv.yaml
Apply the above configuration by running the following command:
Verify the status of persistent volume by running the following command:
Persistent Volume claim
After creating the persistent Volume claim, let us create a persistent volume claim for the test-pv which we have created in the previous step.
Verify the status of the persistent volume claim(test-pvc) by running the following command:
We are not going to create the kubernetes secret inside the kubernetes cluster. Because we want to replicate the CreateContainerConfigError.
At last create the deployment for mysql docker container image. Use the following deployment manifest (test-deployment.yaml).
Apply the above deployment configuration by running the following kubectl command:
Verify the status of the pod using kubectl get all command:
2. How to Check the container logs for finding the missing Kubernetes Secret
Now in the Step-1 we replicated the CreateContainerConfigError, but to find the exact root cause we need to check the POD logs.
Run the following command to list all the PODs running inside the kubernetes cluster.
Copy the container POD name from the above output .i.e. - mysql-95d6b45b5-8bcpz
Run the kubectl describe command with POD name as follow -
The above mentioned describe command will show a very long log file but carefully scroll down to the bottom of the log and look for following message
You can easily identify the error “mysql-secret” not found and due to this error POD was not getting deployed and resulting in CreateContainerConfigError.
3. Create and apply the correct kubernetes secret
Let’s create the kubernetes secret (test-secret.yaml) for mysql and apply it.
Apply the above test-secret.yaml configuration with kubectl command:
Verify the secret by running the following kubectl command:
4.Re-run the kubernetes deployment after fixing the kubernetes secret
Now we have fixed the kubernetes secret, the next step would be delete the old kubernetes deployment of mysql and re-deploy it.
Run the following kubectl command to delete the old deployment mysql:
Re-deploy the mysql deployment (test-deployment.yaml) manifest again:
Check the Pod status again after the re-deployment:
Check the Pod logs for using the kubectl describe:
Always start the debugging of the CreateContainerConfigError by looking into the POD logs using the kubectl describe Pod command because those POD logs will help you to identify the name of the missing kubernetes secret inside the kubernetes cluster.