Deployment takes a long time

Hello Qovery,

One issue we are running into is that our environment takes a long time to deploy. It takes about 8-10 minutes to deploy whereas, in our current pipeline, it takes less than 4 minutes even with less resources. I suspect It may be related to how Qovery handles the deployment queues and the probs.

Do you happen to have any tips to make this speedier?

1 Like

Hi @markvantage can I give a try to clone one of your environments to run from scratch a full deployment and run an analysis of what’s taking some time?

Yes. Thanks for looking into it.

Thank you

Hi @markvantage :wave: , thank you - I run some tests and will come back to you.

Side Note: We’ll add some tooling to the CLI to better troubleshoot the deployment time of an environment and which steps take most of the deployment time. I’ll keep you posted here.

Hi @markvantage ,

I added a command inside the CLI to troubleshoot the deployment time (not released yet). Here is the output for my test inside your project.

 go build && ./qovery-cli environment deployment explain --organization "[REDACTED]" --project "[REDACTED]" --environment "test-romaric" --level "step"
.
└── Environment: test-romaric [duration: 10 minutes and 57 seconds]
    β”œβ”€β”€ Stage 1: Dependencies [duration: 1 minute and 37 seconds]
    β”‚   β”œβ”€β”€ redis [duration: 1 minute and 33 seconds]
    β”‚   β”‚   └── Step 1: Deploy [duration: 50 seconds]
    β”‚   β”œβ”€β”€ postgres [duration: 1 minute and 33 seconds]
    β”‚   β”‚   └── Step 1: Deploy [duration: 1 minute and 1 seconds]
    β”‚   β”œβ”€β”€ elasticsearch [duration: 1 minute and 33 seconds]
    β”‚   β”‚   └── Step 1: Deploy [duration: 1 minute and 26 seconds]
    β”‚   └── mailhog [duration: 1 minute and 32 seconds]
    β”‚       └── Step 1: Deploy [duration: 1 minute and 0 seconds]
    β”œβ”€β”€ Stage 2: Essentials [duration: 3 minutes and 38 seconds]
    β”‚   β”œβ”€β”€ db-migration-job [duration: 3 minutes and 25 seconds]
    β”‚   β”‚   └── Step 1: Deploy [duration: 3 minutes and 8 seconds]
    β”‚   β”œβ”€β”€ fe-app [duration: 3 minutes and 35 seconds]
    β”‚   β”‚   β”œβ”€β”€ Step 1: Deploy [duration: 3 minutes and 35 seconds]
    β”‚   β”‚   └── Step 2: Deployed [duration: 1 minute and 46 seconds]
    β”‚   └── fe-admin [duration: 3 minutes and 35 seconds]
    β”‚       β”œβ”€β”€ Step 1: Deploy [duration: 3 minutes and 35 seconds]
    β”‚       └── Step 2: Deployed [duration: 1 minute and 56 seconds]
    β”œβ”€β”€ Stage 3: CONTAINER DEFAULT [duration: 1 minute and 33 seconds]
    β”‚   β”œβ”€β”€ api-ruby [duration: 1 minute and 30 seconds]
    β”‚   β”‚   β”œβ”€β”€ Step 1: Deploy [duration: 1 minute and 30 seconds]
    β”‚   β”‚   └── Step 2: Deployed [duration: 36 seconds]
    β”‚   β”œβ”€β”€ node-api [duration: 1 minute and 30 seconds]
    β”‚   β”‚   β”œβ”€β”€ Step 1: Deploy [duration: 1 minute and 30 seconds]
    β”‚   β”‚   └── Step 2: Deployed [duration: 13 seconds]
    β”‚   └── worker-que [duration: 1 minute and 21 seconds]
    β”‚       └── Step 1: Deploy [duration: 36 seconds]
    β”œβ”€β”€ Stage 4: APPLICATION DEFAULT [duration: 1 minute and 11 seconds]
    β”‚   β”œβ”€β”€ hasura-external [duration: 1 minute and 9 seconds]
    β”‚   β”‚   β”œβ”€β”€ Step 1: Deploy [duration: 1 minute and 9 seconds]
    β”‚   β”‚   └── Step 2: Deployed [duration: 19 seconds]
    β”‚   β”œβ”€β”€ worker-sidekiq [duration: 1 minute and 0 seconds]
    β”‚   β”‚   └── Step 1: Deploy [duration: 37 seconds]
    β”‚   └── hasura [duration: 1 minute and 7 seconds]
    β”‚       β”œβ”€β”€ Step 1: Deploy [duration: 1 minute and 7 seconds]
    β”‚       └── Step 2: Deployed [duration: 11 seconds]
    └── Stage 5: ALL OTHER [duration: 2 minutes and 43 seconds]
        β”œβ”€β”€ hasura-external-migration-job [duration: 2 minutes and 39 seconds]
        β”‚   └── Step 1: Deploy [duration: 2 minutes and 25 seconds]
        β”œβ”€β”€ hasura-migration-job [duration: 2 minutes and 40 seconds]
        β”‚   └── Step 1: Deploy [duration: 2 minutes and 28 seconds]
        └── node-worker [duration: 2 minutes and 40 seconds]
            └── Step 1: Deploy [duration: 1 minute and 39 seconds]

By quickly looking, I do think you should take a look at the following services to see if you can optimize them:

  • db-migration-job β†’ it takes 3 minutes and 8 seconds to be deployed
  • fe-admin β†’ it also takes 3 minutes to be deployed
  • fe-app β†’ same for this one

But when I looked at the details of the Stage 2 (Essentials) and those services… I see that they are most of their time waiting to get a new Kubernetes node available.

    β”œβ”€β”€ Stage 2: Essentials [duration: 3 minutes and 38 seconds]
    β”‚   β”œβ”€β”€ db-migration-job [duration: 3 minutes and 25 seconds]
    β”‚   β”‚   └── Step 1: Deploy [duration: 3 minutes and 8 seconds]
    β”‚   β”‚       β”œβ”€β”€ πŸ”“ Login to registry docker.io as user [REDACTED]devops
    β”‚   β”‚       β”œβ”€β”€ πŸͺž Mirroring image to private cluster registry to ensure reproductibility
    β”‚   β”‚       β”œβ”€β”€ πŸš€ Deployment of Job at tag [REDACTED]/api-ruby:master is starting with a timeout/max duration of 5.00 minutes
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃  |__ Pod job-zbe271275-db-migration-job-9cqst is STARTING
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃  |__ Pod job-zbe271275-db-migration-job-9cqst is STARTING
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃  |__ Pod job-zbe271275-db-migration-job-9cqst is STARTING
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃  |__ Pod job-zbe271275-db-migration-job-9cqst is STARTING
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Successfully pulled image "busybox:1.34.1" in 728.168337ms (728.186637ms including waiting)
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Created container qovery-wait-container-output
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Started container qovery-wait-container-output
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃  |__ Pod job-zbe271275-db-migration-job-9cqst is STARTING
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Successfully pulled image "[REDACTED].dkr.ecr.us-east-1.amazonaws.com/qovery-mirror-be271275-27c4-4cb1-b4df-[REDACTED]:[REDACTED].api-ruby.master.be271275-27c4-4cb1-b4df-[REDACTED]" in 17.882343497s (17.882366331s including waiting)
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Created container job-zbe271275-db-migration-job
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Successfully pulled image "busybox:1.34.1" in 728.168337ms (728.186637ms including waiting)
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 1 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ Waiting for the job container job-zbe271275-db-migration-job to be processed...
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 1 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 1 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ 
    β”‚   β”‚       β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚       β”œβ”€β”€ ┃ Job at tag [REDACTED]/api-ruby:master execution is in progress ⏳, below the current status:
    β”‚   β”‚       β”œβ”€β”€ ┃
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃ πŸ›° Job at new tag [REDACTED]/api-ruby:master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚       β”œβ”€β”€ ┃  |__ Pod job-zbe271275-db-migration-job-9cqst is STARTING
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Started container job-zbe271275-db-migration-job
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Successfully pulled image "[REDACTED].dkr.ecr.us-east-1.amazonaws.com/qovery-mirror-be271275-27c4-4cb1-b4df-[REDACTED]:[REDACTED].api-ruby.master.be271275-27c4-4cb1-b4df-[REDACTED]" in 17.882343497s (17.882366331s including waiting)
    β”‚   β”‚       β”œβ”€β”€ ┃     |__ ℹ️ Created container job-zbe271275-db-migration-job
    β”‚   β”‚       └── ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”œβ”€β”€ fe-app [duration: 3 minutes and 35 seconds]
    β”‚   β”‚   β”œβ”€β”€ Step 1: Deploy [duration: 3 minutes and 35 seconds]
    β”‚   β”‚   β”‚   β”œβ”€β”€ πŸ”“ Login to registry docker.io as user [REDACTED]devops
    β”‚   β”‚   β”‚   β”œβ”€β”€ πŸͺž Mirroring image to private cluster registry to ensure reproductibility
    β”‚   β”‚   β”‚   β”œβ”€β”€ πŸš€ Deployment of Container `z0a4c5630` at tag/commit [REDACTED]/[REDACTED]-frontend:fe_app-ci-master is starting: You have 0 pod(s) running, 0 service(s) running, 0 network volume(s)
    β”‚   β”‚   β”‚   β”œβ”€β”€ 
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master deployment is in progress ⏳, below the current status:
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ 
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master deployment is in progress ⏳, below the current status:
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃  |__ Pod app-z0a4c5630-fe-app-7cd5776d4b-d9zqq is STARTING
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ 
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master deployment is in progress ⏳, below the current status:
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃  |__ Pod app-z0a4c5630-fe-app-7cd5776d4b-d9zqq is STARTING
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ 
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master deployment is in progress ⏳, below the current status:
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃  |__ Pod app-z0a4c5630-fe-app-7cd5776d4b-d9zqq is STARTING
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ 
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master deployment is in progress ⏳, below the current status:
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_app-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃  |__ Pod app-z0a4c5630-fe-app-7cd5776d4b-d9zqq is STARTING
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃     |__ ℹ️ Successfully assigned env-z4efc517f-test-romaric/app-z0a4c5630-fe-app-7cd5776d4b-d9zqq to ip-10-31-85-124.ec2.internal
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃     |__ ℹ️ Pulling image "[REDACTED].dkr.ecr.us-east-1.amazonaws.com/qovery-mirror-0a4c5630-6e0d-4edb-aa9b-652bb13d7097:[REDACTED].[REDACTED]-frontend.fe_app-ci-master.0a4c5630-6e0d-4edb-aa9b-652bb13d7097"
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┃     |__ ℹ️ pod triggered scale-up: [{eks-qovery-2023080317311382170000000a-50c4de3a-0f2f-faf7-a120-6abc3d3f7f63 6->7 (max: 15)}]
    β”‚   β”‚   β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚   β”‚   β”‚   β”œβ”€β”€ πŸš€ Deployment of router `z41103054` is starting
    β”‚   β”‚   β”‚   β”œβ”€β”€ 🌍 Checking DNS Ip resolution for domain z4efc517f-z41103054-gtw.[REDACTED]. Please wait, it can take some time...
    β”‚   β”‚   β”‚   β”œβ”€β”€ 🌍 Waiting domain z4efc517f-z41103054-gtw.[REDACTED] resolve to an Ip address...
    β”‚   β”‚   β”‚   └── ✨ Domain z4efc517f-z41103054-gtw.[REDACTED] resolved to ip 52.55.34.109
    β”‚   β”‚   └── Step 2: Deployed [duration: 1 minute and 46 seconds]
    β”‚   β”‚       β”œβ”€β”€ βœ… Deployment of Container succeeded
    β”‚   β”‚       └── βœ… Deployment of router succeeded
    β”‚   └── fe-admin [duration: 3 minutes and 35 seconds]
    β”‚       β”œβ”€β”€ Step 1: Deploy [duration: 3 minutes and 35 seconds]
    β”‚       β”‚   β”œβ”€β”€ πŸ”“ Login to registry docker.io as user [REDACTED]devops
    β”‚       β”‚   β”œβ”€β”€ πŸͺž Mirroring image to private cluster registry to ensure reproductibility
    β”‚       β”‚   β”œβ”€β”€ πŸš€ Deployment of Container `z3105c137` at tag/commit [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master is starting: You have 0 pod(s) running, 0 service(s) running, 0 network volume(s)
    β”‚       β”‚   β”œβ”€β”€ 
    β”‚       β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master deployment is in progress ⏳, below the current status:
    β”‚       β”‚   β”œβ”€β”€ ┃
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ 
    β”‚       β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master deployment is in progress ⏳, below the current status:
    β”‚       β”‚   β”œβ”€β”€ ┃
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃  |__ Pod app-z3105c137-fe-admin-848bdd6848-d7x9w is STARTING
    β”‚       β”‚   β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚       β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ 
    β”‚       β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master deployment is in progress ⏳, below the current status:
    β”‚       β”‚   β”œβ”€β”€ ┃
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃  |__ Pod app-z3105c137-fe-admin-848bdd6848-d7x9w is STARTING
    β”‚       β”‚   β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚       β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ 
    β”‚       β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master deployment is in progress ⏳, below the current status:
    β”‚       β”‚   β”œβ”€β”€ ┃
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃  |__ Pod app-z3105c137-fe-admin-848bdd6848-d7x9w is STARTING
    β”‚       β”‚   β”œβ”€β”€ ┃     |__ ⚠️ 0/6 nodes are available: 6 Insufficient cpu. preemption: 0/6 nodes are available: 6 No preemption victims found for incoming pod..
    β”‚       β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ 
    β”‚       β”‚   β”œβ”€β”€ ┏━━ πŸ“ Deployment Status Report ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ ┃ Container at tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master deployment is in progress ⏳, below the current status:
    β”‚       β”‚   β”œβ”€β”€ ┃
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at old version has 0 pods: 0 running, 0 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃ πŸ›° Container at new tag [REDACTED]/[REDACTED]-frontend:fe_admin-ci-master has 1 pods: 0 running, 1 starting, 0 terminating and 0 in error
    β”‚       β”‚   β”œβ”€β”€ ┃  |__ Pod app-z3105c137-fe-admin-848bdd6848-d7x9w is STARTING
    β”‚       β”‚   β”œβ”€β”€ ┃     |__ ℹ️ Successfully assigned env-z4efc517f-test-romaric/app-z3105c137-fe-admin-848bdd6848-d7x9w to ip-10-31-85-124.ec2.internal
    β”‚       β”‚   β”œβ”€β”€ ┃     |__ ℹ️ Pulling image "[REDACTED].dkr.ecr.us-east-1.amazonaws.com/qovery-mirror-3105c137-4dde-4010-a8f9-4bc35dff1f1a:[REDACTED].[REDACTED]-frontend.fe_admin-ci-master.3105c137-4dde-4010-a8f9-4bc35dff1f1a"
    β”‚       β”‚   β”œβ”€β”€ ┃     |__ ℹ️ pod triggered scale-up: [{eks-qovery-2023080317311382170000000a-50c4de3a-0f2f-faf7-a120-6abc3d3f7f63 6->7 (max: 15)}]
    β”‚       β”‚   β”œβ”€β”€ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    β”‚       β”‚   β”œβ”€β”€ πŸš€ Deployment of router `z648238af` is starting
    β”‚       β”‚   β”œβ”€β”€ 🌍 Checking DNS Ip resolution for domain z4efc517f-z648238af-gtw.[REDACTED]. Please wait, it can take some time...
    β”‚       β”‚   β”œβ”€β”€ 🌍 Waiting domain z4efc517f-z648238af-gtw.[REDACTED] resolve to an Ip address...
    β”‚       β”‚   └── ✨ Domain z4efc517f-z648238af-gtw.[REDACTED] resolved to ip 52.55.34.109
    β”‚       └── Step 2: Deployed [duration: 1 minute and 56 seconds]
    β”‚           β”œβ”€β”€ βœ… Deployment of Container succeeded
    β”‚           └── βœ… Deployment of router succeeded

Let’s do 2 things to improve the deployment time here:

  1. Increase your cluster resources to ensure that when a new environment is deployed, the Kubernetes cluster does not claim an additional node (it can take some time - here, we see that it takes more than a minute).
  2. Adjust your apps’ CPU and RAM to avoid wasting resources and unnecessarily triggering the cluster node auto scaler. >> To do it - consider using Kubecost or a monitoring solution to know how much resources your apps really need.

How does that sound? Let me know if you have any questions.

Romaric.

FYI - I published this article as well