Retroactivity of the registry.image_retention_time option

Questions and Answers
1

Hello,

I haven’t changed the registry.image_retention_time option on my cluster, and as it’s one year by default, I have a huge backlog of images on my ECR (which is costing us a lot of money!).

I am going to change its value, but I’d like to know what will happen to the qovery-mirror-*** repositories that are older than my new value. Will they be removed automatically by Qovery or should I clean them myself because the change is not retroactive?

Thanks,

2

Hi @Talal ,

From our experience, storing those images does not cost so much money, that’s why we set the default value to one year (and to avoid also unexpected behavior if you don’t redeploy your apps so often).

Can you tell me how much that costs you?

It’s not retroactive but if you really to clean up those images you can feel free to delete them - but make sure they are not used by your apps if you are running production workloads.

3

Hello @rophilogene,

The qovery-mirror-*** images represent about 8 to 9To of data, so it is costing us more than $800/month.

For the manual deletion, is there an easy way to know which images are used by Qovery? And if I delete one that is used, will it be re-generated?

4

If you are not using Qovery in production feel free to drop all images and redeploy your environments - they will be regenerated.

Out of curiosity : what’s the average size of your images? Did you try to reduce the size?

5

Ps: even if you have production apps, you can safely delete all the images if you redeploy right away your apps to force them to be regenerated by Qovery. But do this when your workload is low.

6

@rophilogene We are using Qovery in production.

Just to be completely sure: you’re telling me that I can delete all the qovery-mirror-*** images on the ECR repository, and if I redeploy the cluster right away (when the workload is low), they will be regenerated safely. Is that right?

7

Let me double check with the team for a better procedure. Because I don’t want to provide one that will cause some downtime. Even minor downtime.

If it’s ok for you to get potential minor downtime, the quickest and simplest (from my perspective) would be to:

  1. Change your image retention time from cluster advanced settings.
  2. Apply changes by updating your cluster and wait the end of the update.
  3. Find and delete all container registry repositories from this cluster.
  4. Redeploy all your environments running on your cluster.

@Pierre_Mavro would you recommend another safest and simple approach?

8

hello @Talal ,

Image retention setting is not retroactif, it will set only for newly created ecr repository.

Regarding purging everything, you can do it, but you need to re-deploy all your environments just after it.

There will be no downtime, because image will be cached on the node of the cluster and currently running apps will be fine.
But if a new node is scaled-up in the cluster while you have deleted the image cache, it will fail to schedule the app.

So you can:

  • Delete all the qovery-mirror-*** ecr registry
  • change the image retention advance settings with the value you want
  • just after re-deploy all your environments for qovery to re-create the mirror with the new settings
9

Thank you @Erebe, I did what you suggested and it worked well.

10

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.