Environment stuck in deployment state

Hello there :leaves:

Our application is stuck in deployment state while there is no current deployment.
It means that it’s actually impossible for us to deploy another version, re-deploy and unexpectedly to cancel deployment.

There are no deployment logs, no “deploying” label on any app in the stack.

How do you guys think we could fix this unless deleting the stack and creating another one (which will not be possible since it’s in deploying state I guess) ?

Jason.

Hi @json , thanks for the report - @Melvin_Zottola (backend engineer @ Qovery) is taking a look at what’s going on. He will respond through this thread.

Hello @json,
We stopped your environment deployment, it’s in status DEPLOYMENT_ERROR now.
I’m investigating why it was still running.

Melvin.

Hello !
Thank you. :slight_smile:

Here are the remaining logs, just to be sure u got everything.

Here are updates.

Tried to delete env, database naturally goes first but is stuck in deleting state.

I upgraded the cluster to a 4-12 t3a.large scale group.
The stack still encounters stuck deployment issues.

Hello @json ,
Sorry for the delay.
We’re still fixing the issue, I let you know once it is resolved.

About your last message, I can see that the deployment of your application “api-server” is making a timeout on our engine side: you should see in the error logs Readiness probe failed. We have a timeout of 10minutes for deploying the application.

1 Like

Thanks a lot for your very quick replies!

Feel free to ask any details if needed.
Actually we’re deploying this ‘api-server’ service in less than 15 seconds locally. It’s not supposed to last 10 minutes.

Okay thank you for the precision :+1:

Was there a recent change with your Dockerfile application port ?
We’ve seen that the port used by your current running application is 8880 and the application that fails is 8080.
If yes, can you re-create a 2nd application “api-server-2” in the same environment with the good port and deploy this new application ?

No recent change on Dockerfile ports.

Locally the stack is running correctly on docker compose with the same ports.

The failing app is crossbar on 8880, which is a websocket node server.
And ‘api-server’ is on 8080, yes.

Okay, indeed I thought it was the same application at first sight, my bad.

We are experiencing environment deployment errors since this afternoon. I let you know once this is resolved.

Hi @json , it has been resolved now. Let me know if you experienced any similar issues :pray:

Hello.
I am having the same issue here.

Hi @vmc , I don’t think you have the same issue, can you open another thread? (This issue is fixed)

Hi there, I think there’s still an issue.
Still having same stuck deployment state, with more informations about not found pod controllers. Maybe I have to update my cluster or something like this ? Let me know if I have to handle ops by my side.

Can you show your application logs?

Got no applications logs. :grimacing:

I tried to create a new env, which is now deployed and working.
The existing environments are broken and had to delete them.

So I can confirm the issue is now fixed. If you eventually want precise details, logs for investigations don’t hesitate to ask.

1 Like