ResourcesMore Than Filled IconKubernetes Troubleshooting - Fixing ImagePullBackOff State Error

Kubernetes Troubleshooting - Fixing ImagePullBackOff State Error

Kubernetes Troubleshooting - Fixing ImagePullBackOff State Error

The ImagePullBackOff is an error state that occurs when a Pod startup fails due to an inability to pull the image. The state is displayed when running a $ kubectl get pods command for viewing/finding resources, as shown here:


NAME                  READY   STATUS               RESTARTS   AGE
nginx-5263354bf62     1/2     ImagePullBackOff     0          3h

A request to pull an image may include multiple elements in the path, each with a potential to be misconfigured. For each type of misconfiguration, Kubernetes returns a slightly different error, as described below. In short, there are two types of requests: 

  • Full path request (e.g. gcr.io/google-samples/pubsub-sample:v1) - this type of request specifies the registry (gcr.io), repository (google-samples), image name (pubsub-sample), and image tag (v1). Images in a private repository require the full path.   
  • Partial path request (e.g. pubsub-sample:v1) - in this case, the only elements included in the request are the image name (pubsub-sample) and the image tag (v1). In terms of the missing registry, if you don’t define a registry, the default registry is Docker hub. And in terms of the repository, there are two options:
  • 1. For verified images in a public repository - you don’t need to specify the registry and the repository (e.g. nginx)
  • 2. For non-verified images in a public repository - you need to specify the repository and the image name, but you don't need to specify the registry (e.g. rancher/nginx).

Reasons for ImagePullBackOff error

The ImagePullBackOff state can happen for the following reasons:

  • Misconfiguration of path elements - whether for a full path or a partial path, the request itself may have errors in any of the elements. However, as described below, the error messages vary for each type of mistake and it often doesn’t indicate the specific issue. 
  • Access configuration issue - the registry/repository is private, and you are not authorized to access it.
  • The registry is inaccessible - it’s possible that there’s a network issue between your Kubernetes node and the registry.

How to fix the ImagePullBackOff error?

Fixing misconfiguration of path elements

To make sense of the error messages concerning misconfigured path elements, I have generated examples for each misconfiguration option.

Misconfigured image name

Private registry


Type 	   Reason         Age                	    From                Message
----       ------ 	  ----               	    ----           	-------
Normal     Pulling	  29m (x4 over 31m)  	    kubelet        	Pulling image "gcr.io/google-samples/fake-app:1.0"
Warning    Failed 	  29m (x4 over 31m)  	    kubelet        	Failed to pull image "gcr.io/google-samples/fake-app:1.0": 
rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/google-samples/fake-app:1.0 not found: manifest unknown: Failed to fetch "1.0" from request "/v2/google-samples/fake-app/manifests/1.0".
Warning    Failed 	  29m (x4 over 31m)  	    kubelet        	Error: ErrImagePull
Warning    Failed 	  23s (x107 over 31m)       kubelet             Error: ImagePullBackOff
Normal     BackOff	  86s (x129 over 31m)	    kubelet        	Back-off pulling image "gcr.io/google-samples/fake-app:1.0"

In the above example, we can see a request to pull - gcr.io/google-samples/fake-app:1.0. In this case the registry (gcr.io), exists. The repository (google-samples) exists. But the image name (fake-app) has been misconfigured.

Now let's look at the error that was returned:

	
manifest for gcr.io/google-samples/fake-app:1.0 not found: manifest
unknown: Failed to fetch "1.0" from request "/v2/google-samples/fake-app/manifests/1.0".
	

You would expect that the error would indicate that the problem is in the image-name level, but, as you can see, the error message indicates that it has failed to fetch the image tag, although the misconfiguration is in the image-name level. This means that when this error appears, it could be a misconfiguration of both the image-name and the image-tag

Public registry

	
Type  	 Reason         Age                  From           	Message
----     ------ 	----                 ----           	-------
Normal   Pulling	45s (x4 over 2m28s)  kubelet        	Pulling image "fake-app:1.0"
Warning  Failed 	42s (x4 over 2m25s)  kubelet        	Failed to pull image "fake-app:1.0": 
rpc error: code = Unknown desc = Error response from daemon: pull access denied for fake-app, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning  Failed 	42s (x4 over 2m25s)  kubelet        	Error: ErrImagePull
Normal   BackOff	28s (x6 over 2m25s)  kubelet        	Back-off pulling image "fake-app:1.0"
Warning  Failed 	15s (x7 over 2m25s)  kubelet        	Error: ImagePullBackOff
	

In the above example, the request fake-app:1.0 did not indicate the registry or the repository. In this case, the system assumes that this is a verified image from a public repository.

Now let's look at the error that was returned:

	
Failed to pull image "fake-app:1.0": rpc error: code = Unknown desc = Error response from daemon: 
pull access denied for fake-app, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
	

The error indicates that either the repository does not exist or that you don't have access to Docker. This means that when this error appears:

  • It’s possible that this is not a verified image in a public repository, so you should have indicated the registry and repository before the image name and image tag;
  • It’s possible that that image tag does not exist;
  • Or it’s possible that you don’t have access to the repository.

Misconfigured image tag

Private registry

	
Type 	 Reason         Age   From           	Message
----     ------ 	----  ----           	-------
Normal   Pulling	14s   kubelet        	Pulling image "gcr.io/google-samples/hello-app:0.0.1"
Warning  Failed 	10s   kubelet        	Failed to pull image "gcr.io/google-samples/hello-app:0.0.1": 
rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/google-samples/hello-app:0.0.1 not found: manifest unknown: Failed to fetch "0.0.1" from request "/v2/google-samples/hello-app/manifests/0.0.1".
Warning  Failed         10s   kubelet           Error: ErrImagePull
Normal   BackOff	10s   kubelet        	Back-off pulling image "gcr.io/google-samples/hello-app:0.0.1"
Warning  Failed 	10s   kubelet        	Error: ImagePullBackOff
	

In the above example, we can see a request to pull - gcr.io/google-samples/hello-app:0.0.1. In this case the registry (gcr.io), exists. The repository (google-samples) exists. The image name (hello-app) exists. But the image tag (0.0.1) has been misconfigured.

Now let's look at the error that was returned:

	
Failed to pull image "gcr.io/google-samples/hello-app:0.0.1": 
rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/google-samples/hello-app:0.0.1 not found: 
manifest unknown: Failed to fetch "0.0.1" from request "/v2/google-samples/hello-app/manifests/0.0.1".
	

In this case, the error rightfully indicates that it has failed to fetch the specific image tag (0.0.1).

Public registry

	
Type 	 Reason           Age   From           	   Message
----     ------ 	  ----  ----           	   -------
Normal   Scheduled        19s   default-scheduler  Successfully assigned default/bad-tag to docker-desktop
Normal   Pulling	  18s   kubelet            Pulling image "nginx:fake-tag"
Warning  Failed 	  8s	kubelet            Failed to pull image "nginx:fake-tag": 
rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:fake-tag not found: manifest unknown: manifest unknown
Warning  Failed 	  8s	kubelet            Error: ErrImagePull
Normal   BackOff	  7s	kubelet            Back-off pulling image "nginx:fake-tag"
Warning  Failed 	  7s	kubelet            Error: ImagePullBackOff
	

The short version of the request Pulling image nginx:fake-tag doesn't indicate the registry or the repository. In this case, the error has rightfully indicated that the manifest for the tag was not found.

Misconfigured repository

Private registry

	
Type 	 Reason   Age                	From 	Message
----     ------   ----               	---- 	-------
Normal   Pulling  59m (x4 over 61m)  	kubelet  Pulling image "gcr.io/samples/hello-app:1.0"
Warning  Failed   59m (x4 over 61m)  	kubelet  Failed to pull image "gcr.io/samples/hello-app:1.0": 
rpc error: code = Unknown desc = Error response from daemon: Head https://gcr.io/v2/samples/hello-app/manifests/1.0: unknown: Project 'project:samples' not found or deleted.
Warning  Failed   59m (x4 over 61m)  	kubelet  Error: ErrImagePull
Normal   BackOff  6m (x239 over 61m)    kubelet  Back-off pulling image "gcr.io/samples/hello-app:1.0"
Warning  Failed   70s (x261 over 61m)	kubelet  Error: ImagePullBackOff
	

In the above example, we can see the following request - gcr.io/samples/hello-app:1.0. The request lists an incorrect repository, samples, instead of google-samples. In this case the error message indicates the project:samples was not found. This error is used to indicate that the repository was not found.

Public registry


Type 	 Reason           Age                	   From           	  Message
----     ------ 	  ----               	   ----           	  -------
Normal   Scheduled        5m11s              	   default-scheduler       Successfully assigned default/bad-repository to docker-desktop
Normal   Pulling	  3m41s (x4 over 5m11s)    kubelet        	  Pulling image "nginxx/nginx:1.14.0"
Warning  Failed 	  3m38s (x4 over 5m5s)     kubelet        	  Failed to pull image "nginxx/nginx:1.14.0": 
rpc error: code = Unknown desc = Error response from daemon: pull access denied for nginxx/nginx, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning  Failed 	  3m38s (x4 over 5m5s)     kubelet        	  Error: ErrImagePull
Warning  Failed 	  3m13s (x6 over 5m5s)     kubelet        	  Error: ImagePullBackOff
Normal   BackOff	  3m (x7 over 5m5s)  	   kubelet        	  Back-off pulling image "nginxx/nginx:1.14.0"

In this example, the request nginxx/nginx:1.14.0 includes a misconfigured repository - nginxx. The error message correctly indicates that the repository either doesn’t exist or requires login.  

Misconfigured registry

Private registry


Type 	 Reason         Age            	     From               Message
----     ------ 	----           	     ----           	-------
Normal   Pulling	14s (x2 over 26s)    kubelet        	Pulling image "worng-registry.io/google-samples/hello-app:1.0"
Warning  Failed 	14s (x2 over 25s)    kubelet        	Failed to pull image "worng-registry.io/google-samples/hello-app:1.0":
rpc error: code = Unknown desc = Error response from daemon: Get https://worng-registry.io/v2/: Service Unavailable
Warning  Failed 	14s (x2 over 25s)    kubelet        	Error: ErrImagePull
Normal   BackOff	3s (x2 over 25s)     kubelet        	Back-off pulling image "worng-registry.io/google-samples/hello-app:1.0"
Warning  Failed 	3s (x2 over 25s)     kubelet        	Error: ImagePullBackOff

In this example, the request has a misconfigured registry namewrong-registry. The error indicates, "Service Unavailable".  When this type of error appears, it means the system was unable to find the registry because it doesn’t exist.

Public registry


Type   	 Reason      	 Age          	  From           	Message
----     ------      	 ----         	  ----           	-------
Normal   Scheduled   	 7s           	  default-scheduler   Successfully assigned default /bad-registry to docker-desktop
Normal   Pulling     	 7s           	  kubelet        	Pulling image "worng-registry.io/ngnix:1.14.0" 
Warning  Failed      	 7s           	  kubelet        	Failed to pull image "worng-registry.io/ngnix:1.14.0": 
rpc error: code = Unknown desc = Error response from daemon: Get https://worng-registry.io/v2/: Service Unavailable
Warning  Failed      	 7s           	  kubelet        	Error: ErrImagePull
Normal   SandboxChanged  6s           	  kubelet        	Pod sandbox changed, it will be killed and re-created.
Normal   BackOff     	 4s (x3 over 5s)  kubelet        	Back-off pulling image "worng-registry.io/ngnix:1.14.0"
Warning  Failed      	 4s (x3 over 5s)  kubelet        	Error: ImagePullBackOff

When checking it against the public image registry, kubernetes returns the same "Service Unavailable" message, indicating that the registry itself was not found.

Tips for troubleshooting path misconfigurations

The examples of error messages mentioned above should help you identify whether the ImagePullBackOff state was caused by a path misconfiguration. But in any case there are some additional things you can do to solve this “mystery”: 

Try pulling the docker image locally using a generic tag - if you were not able to pull the docker image, try pulling it using a generic latest tag. If you’re successful, it means that the registry is correct.


$ docker pull (registry/repo/)image:latest

Do you see any errors?

👉 [YES] There are two possible reasons:

  1. There’s an access configuration issue between your local machine and the registry.
  2. The image doesn’t exist on the repository. If your registry has a web GUI, you can use it to see the valid images. 

👉 [NO] Add the image tag:


$ docker pull (registry/repo/)image:1.0.0

Do you see any errors?

👉 [YES] you now know that your image tag doesn’t exist in your registry. If your registry has a web GUI, you can use it to see the valid tags.

👉 [NO] You now know it is an access configuration issue between the pod and the registry because the image and tag are valid.

Fixing access configuration issue

For security reasons the error will not specifically indicate the lack of permissions to access the container registry name. However, if the names in the path are correct, perhaps you don't have permissions to access the registry/repository. In fact, the same error message saying that the repository doesn’t exist or has no pull access, will apply in this case. 

  1. Check if you can access the container locally - as a first step you can try pulling the same image from your local machine.
  2. Add the secret to Kubernetes - if you’re trying to pull an image from a private repository that requires authentication, you need to add the required registry secret into Kubernetes. In the following example the secret name is registry-secret:

kubectl -namespace [YOUR NAMESPACE] \ 
create secret docker-registry registry-secret \
--docker-server=https://index.docker.io/v1/ \
--docker-username=[THE USERNAME] \
--docker-password=[THE PASSWORD] \
--docker-email=example@example.com

  1. Refer to the imagePullSecrets on a Pod - after adding the secret, you should add the imagePullSecrets reference so that your pod knows to use it. This needs to be done for each pod using a private registry.

apiVersion: v1
kind: Pod
metadata: null
name: foo
namespace: awesomeapps
spec: null
containers:
  - name: foo
image: 'johndoe/awesomeapp:v1'
imagePullSecrets:
  - name: registry-secret

Fixing inaccessible registry

In case there’s an issue that prevents access to the registry itself, you should do the following:

  1. Log into the Kubernetes node - you’ll need to check that this is the node to which the container was assigned by checking that you file /tmp/runbooks_describe_pod.txt contains the host name.
  2. Run container runtime to download and run by hand - run the container runtime command to download and run it manually as described above.
Eyar Zilberman
Chief Product Officer & Co-founder
Datree

Test your first config files to prevent misconfiguration

Try it now!

Prevent Kubernetes Misconfigurations NOW!

Other resources
No items found.