"Failed to pull image" error on mirror images

INFORMATION

I have many CronJobs using same image and tag as the main API service. My Gitlab CI/CD is configured to update apps image tags and deploy them when a build is successful.
On Qovery, I verified tags are updated: everything is fine.

To update and deploy, I use Qovery CLI:

qovery container update --tag xxx [...]
qovery lifecycle update --tag xxx [...]
qovery cronjob update --tag xxx [...]

qovery environment deploy [...]

ISSUE

Sometimes (very often), Kubernetes can’t pull mirror images, especially for CronJobs. It happens also on services.

Failed to pull image "rg.fr-par.scw.cloud/qovery-mirror-[...]: rpc error: code = NotFound desc = failed to pull and unpack image

On Scaleway’s cockpit, when I look for the image reference, the namespace exists but there is no image in it. It looks like the mirroring feature is a bit buggy.

If there is no solution to this, is there a way to disable mirroring? I’m not using external image registry which could rate limit. I’m using the Scaleway registry where mirror images are kept.

Full error message:

Failed to pull image "rg.fr-par.scw.cloud/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1:<redacted>.<redacted>.d1f0be3b.c5650a31-c405-4b8c-9468-a7baf57f3ef1":
rpc error: code = NotFound desc = failed to pull and unpack image "rg.fr-par.scw.cloud/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1:<redacted>.<redacted>.d1f0be3b.c5650a31-c405-4b8c-9468-a7baf57f3ef1":
failed to resolve reference "rg.fr-par.scw.cloud/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1:<redacted>.<redacted>.d1f0be3b.c5650a31-c405-4b8c-9468-a7baf57f3ef1":
rg.fr-par.scw.cloud/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1/qovery-mirror-c5650a31-c405-4b8c-9468-a7baf57f3ef1:<redacted>.<redacted>.d1f0be3b.c5650a31-c405-4b8c-9468-a7baf57f3ef1: not found

This has been resolved introducing a change to workaround docker buildx / SCW registry bug.

Please let me know if you need anything else.

Cheers

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.