Deployment Time went up from ~30 min to ~4,5 hours

Questions and Answers Deployment
1

Deployment Time went up from ~30 min to ~4,5 hours

2

Hi Thomas, by looking at your deployments, I’ve seen multiple things related to Qovery that makes your deployment sometimes (less than 10% of the time) longer than expected.

Long deployment time

I’ve noticed that for the same environment (E.g. your production), sometimes a deployment will take 15 minutes and sometimes up to 45 minutes, and in a rare case more than 50 minutes (cf screenshot with a logarithmic scale on your deployment time).

Screenshot 2022-06-08 at 9.06.29 AM
Screenshot 2022-06-08 at 9.06.29 AM1903×686 37.3 KB

The problem comes from our deployment engine (the part in charge of deploying your infrastructure and your apps) that is running on our infrastructure. This part will download the container cache layers of all your apps that are redeployed from your AWS ECR (Container Registry) and it takes a lot of time (I’ve seen ~11 minutes on average).

Solutions

There are two solutions to solve this issue:

  • Deploy the Qovery engine to run on your infrastructure instead of our infrastructure.
  • Get rid of the build phase on Qovery and deploy directly a container that you have built on your CI system. (not possible yet with Qovery - coming soon early July 2022).

Sequential deployment → Parallel deployment

Qovery deploys your applications one by one in a sequential way. Which leads to a long deployment time.

Solution

I am looking with the team on how we can make parallel deployments available within an environment.

Slow DNS resolution while using CDN

You use custom DNS with CDN for some of your apps. Resulting in domain resolutions taking 5 minutes to be checked and being skipped since your DNS is behind a CDN.

Screenshot_2022-06-08_at_8_53_53_AM
Screenshot_2022-06-08_at_8_53_53_AM1211×668 70 KB

Solution

I am looking with my team to provide an option to skip DNS resolution when using a CDN.

Conclusion

Those issues are not directly related to your apps but to how Qovery works. I will come back to you with all the solutions once they are implemented, tested and available. Then, you will be able to deploy in less than 10 minutes.

Thank you

1 Like
3

Hi @rophilogene thanks for the detailed response :slight_smile:

Regarding your suggestion to run build & deploy on our own instances: Currently we would not like to take this step, it’s a good separation of responsibilities right now, between GH, Qovery and our AWS account. There is one more suggestion I’d like to make here: If it is possible to have a dedicated instance that runs builds within your infrastructure, let’s talk :slight_smile:

Parallel deployments sounds reasonable, can you communicate an ETA?

DNS resolution: Same here: Anything you can communicate already?

I personally like the goal of deploying in less than 10 minutes, looking forward to get there with you :slight_smile:

4

Hello @FlorianSuchan ,

It makes sense to me and it’s doable - let me check what we can concretely propose to you asap.

We have a ticket for this optimization but there is no release date associated. I see with the team and I come back to you.

cc @Florian_Lepont

PR done :tada: → Add multiple app advanced settings by evoxmusic · Pull Request #728 · Qovery/engine · GitHub ← you can expect this to be released tomorrow or max early next week

Challenge accepted :handshake: Let’s make this happen :rocket:

5

@rophilogene nice, thanks a lot :clap: :clap: Looking forward to see progress on the open points :slight_smile:

6

Hi @rophilogene,

are there any updates about the parallel deployment ticket? Did you had the chance to have a look with the team and maybe already a rough ETA? :slight_smile:

7

I come back to you this week for this. cc @a_carrano @Florian_Lepont

1 Like
8

Hi, I know it’s an old thread - but here are some updates on how to optimize the deployment time of your pipeline.