After looking at the response to this topic and browsing through the linked engine code, I have a couple of questions regarding the way Qovery builds images.
Before I get to them, a brief overview:
We have multiple logical services (by logical service I mean a Qovery project and a corresponding Qovery environment)
This logical service/application has multiple ‘sidecars’ (i.e. Qovery services). These sidecars are usually the web server, a couple of background job processor servers, etc
All of these sidecars use the same application code, the only difference between them is the entry point, so ideally the containers should be created from the same image
The problem is that deployments take a huge amount of time because of the image build process, which takes around ~5-10 minutes. This is due to the way an image name is generated (not saying that it’s incorrect).
As evident from the code, one part of the image name is Qovery application’s name, which is different for each Qovery service.
Question 1:
Is it possible to somehow circumvent this and force services to use the same image name-tag combination, without resorting to creating services with the ‘Container registry’ source instead of ‘Git provider’ set, and building images ourselves?
Question 2 - follow-up:
If using the ‘Container registry’ source is the only option, does this mean that image builds have to happen outside of Qovery? According to this part of Qovery documentation, it appears that an external actor has to notify Qovery about the new image hash every time it’s built. If an external CI provider would be used to build images, it would most likely incur additional costs due to data being ingested from outside the AWS VPC, which we would like to avoid if possible. I assume there’s no way to build images from within Qovery itself, because it leads to the ‘chicken and the egg’ problem?
As of now the solution proposed on the question 1 is not something we can easily provide given our current implementation. Plus this has been done this way (Each application has its repository within the registry) to be sure we delete the image when the associated service is deleted.
Moving to a container approach is a more viable approach and you need to:
change the registry mode of your cluster toCluster so that every image will be pushed in the same ECR repository and shared across environments. Make sure to set the right image_registry_retention_time so that images won’t stay in the registry forever, adding more registry cost.
build the images on your side and push them on a container registry
instrument your CI to trigger the deployment of the new version. you can find an example with github actions here
regarding the cost: the network cost will decrease. If you store the built images on a registry located for example on github registry, we will mirror that image to your local registry (the one running on your aws account) just when the image is not available (i.e. the first time). Every other deployment, using the same image, will just pull the data from the local registry without going outside. Data transfer between ECR and other AWS services located in the same region is free.
you have more details on how the image mirroring works here
Thanks for the response, this mostly makes sense.
I’d like to clarify the specific steps that are required in more detail, could you please validate them?
We DON’T have to create any extra registries through the Qovery UI in the organisation settings.
We update the mentioned registry configuration parameters in the existing cluster, including setting the mirroring mode to ‘cluster’.
We have to recreate existing Qovery services in order to set their application source parameters to ‘Container registry’.
When picking the registry from the dropdown menu, we select the already automatically created registry named ‘registry-{hash}’ that corresponds to the cluster.
Then, we instrument the CI with an image build step and Qovery deployment. My outstanding question is how does Qovery learn about the registry it should mirror? Documentation on image mirroring does not seem to mention it, nor the CLI commands that make Qovery aware of a new image tag specify the credentials/URL.
As I was writing this, I realised that we’re supposed to add a registry via Qovery UI that was created beforehand, and that’s how Qovery learns about where to mirror images from, is that right? Therefore the mirror mode parameter is orthogonal to the registry that is used as source for the application, correct?
If my understand is correct and the registry that is listed in the Qovery UI as registry-{hash} is the registry of a corresponding AWS account to which the Qovery cluster is bound to, then yes, we’d like to use that ECR from the same account.
My outstanding question is how does Qovery learn about the registry it should mirror?
Since you have chosen the registry registry-{hash} in step 4, it will be seen as any other registry. Qovery will use the image name to know which repository should be used to pull the image.
Let’s say you pus the images of an application Backend to the registry-{hash} . You can:
push the built images on a repository backend with an incremental tag
configure the application Backend on Qovery side with source registry = registry-{hash} , image name = backend
Qovery will pull the image from the specified repository (image name) and clone it within a shared repository across the cluster
note: we could definitively optimize this step and avoid to clone on the same registry. I’ve added this point to our backlog
