Timeout and concurrency issues with BYOK setup

Hello,

We recently migrated from Qovery Managed to a BYOK setup and have encountered several issues:

  • It appears we are unable to build multiple environments simultaneously within a single cluster. Everything seems to be queued, causing subsequent deployments to be postponed until the first one is completed.
  • The deployments at the front of the queue randomly fail with the error: đź’€ Failed to get an answer from the engine in 10 min, aborting deployment. Please retry later!. However, upon reinitiating, they generally succeed.

Thank you for your attention to these matters.

Hello @markkkkas,

This is actually expected, you’ve got one Engine running on this cluster by default, horizontal pod autoscaler is not activated because it requires prometheus to be deployed to work.

What you can do is to increment Engine replicas setting min replicas from 1 to the value you want in the engine values file here. This will allow to spawn more engines on your side, 1 deployment = 1 engine.

Let me know if it helps.

Cheers

Hi @bchastanier, thanks for a good suggestion.

I think you meant metrics server is required for autoscaling or is it really Prometheus?

Do we understand correctly that Qovery Engine pods on our infra will be building our images and executing the deployment actions? You also mentioned, 1 deployment = 1 engine, what if we have 3 concurrent builds in 1 deployment, does it mean it 3 engines?

Hi @prki,

I think you meant metrics server is required for autoscaling or is it really Prometheus?

To be confirmed but to me you need Prometheus and Prometheus adapter in order to enable HPA on the engine. Right now we do not support deploying it by default on BYOK, but you should be able to build something on your side to enable HPA on engine based on a metric of your choice.

Do we understand correctly that Qovery Engine pods on our infra will be building our images and executing the deployment actions? You also mentioned, 1 deployment = 1 engine, what if we have 3 concurrent builds in 1 deployment, does it mean it 3 engines?

Deployment actions are handled by Engine itself indeed, builds are handled by docker builders spawned in kubernetes as well on demand. So let’s say you have one deployment having 3 apps to be built, it will requires one engine (handling the pipleline and the deployment) and will spawn 3 docker builders in parallel.

Hope it’s clearer, let me know if it’s not.

Cheers

Thanks, @bchastanier, it’s more clear now. We thought the build happens on Qovery infra and that’s why the pricing was based on deployment minutes. What’s the reason pricing is associated with deployment time then?

1 Like