503 Service Temporarily Unavailable

Bugs
1

Hi,

We have cloned production environment to preproduction one. Then configure CNAME subdomain on both qovery and our provider. DNS CNAME is correctly propagated.
Deployment log of our application is succesfull for our application and router

But our website subdomain return 503 Service Temporarily Unavailable

I didn’t mention url here for security purpose.

Thanks
Jérémy.

2

Hi,

Please ensure your application is running correctly. Looks like Nginx can’t contact your application endpoint.

Pierre

3

Hi Pierre,

Of course app is running and available successfully thought rustrocks.cloud url

4

Ok, can you please paste the console URL of the application.

5
6

I confirm what you did is good. The deployed configuration looks good as well. Investigating why it happens

7

Ok @JrmyDev ,

I think I got the issue. Here is what I see: your old and new applications are using the same custom domain. Nginx side it’s not possible, so it’s using the first declared one (redirecting to the stopped instance).

Now here is what I’m guessing and would like your confirmation:

  1. You removed the custom domain on Qovery from the old app
  2. You did not redeployed your app
  3. You stopped it instead (so your app is not live anymore but we keep nginx config to keep TLS certificates and other things that should not change)
  4. You added the same custom domain to your new app

And this is why I see 2 declarations of the same domain. Please tell me if it’s what you did.

What I suggest you in this case is to redeploy your old app, so the custom domain will be released from the load balancer, then stop it again. Redeploy your new backend and it should be good.

Thanks

8

It works !!

Thanks a lot Pierre
Have a nice day

1 Like
9

You’re welcome :slight_smile: . We’re going to improve our product to avoid this in the future. A ticket has been opened internally.

10

Hi @Pierre_Mavro, was this issue resolved? We also started seeing “503 Service Temporarily Unavailable” on one of the applications, but we do not have 2 applications with the same domain as in this case. Are there any other conditions for this issue to appear? We do have 2 external domains on that particular application.

11

Hi,

Can you please paste the console link to the applications you’re talking about please.

Thanks

12

It’s this one: Qovery Web Console | Deploy and Manage On-demand Environments on AWS, Remarkably Fast

We’re getting 503 every time we deploy now.

13

Hi,

Everything looks ok; sites are accessible. If you encounter anytime you deploy on the first seconds/minutes, I encourage you to set custom liveness and readiness probes. More info here: Troubleshoot | Docs | Qovery

Pierre

14

Yes it happens on the first minute or so. The initial delays are increased already. However, it’s weird the old application is kicked off before the new one is ready to accept requests. The delay would only help if the probes were failing.

15

From what you’re saying, I advise you to tune liveness and readiness probe, it will solve your problem. Kubernetes needs to know how your application works (when it’s up and running and when it can send traffic to it). Those elements will help you to solve your issue.

16

Hi @Pierre_Mavro, could you elaborate on what should we tune? AFAIK, Kubernetes should not terminate the existing pod until a new one has started and passed liveness and readiness checks. If there’s a problem with liveness/readiness probes, then Kubernetes will try restarting the new pod, and if not successful then eventually, deployment will crash. We don’t have these problems but occasionally see 503 errors. What’s even more interesting is that we don’t encounter these errors on every deployment.

17

Hi @prki,

Sounds like there might be an issue with your probes, indeed this should not happen.
Did you follow those instructions: Service Advanced Settings | Docs | Qovery ?

It smells like readiness probe for you app is somehow not working properly, it should wait until your app is ready to handle requests.

Can you share more info about your app and checks your are performing for those probes?

Cheers

18

Hi @bchastanier, our application is rather simple, and we used the default settings from Qovery. I now tweaked it to use HTTP instead of TCP to see if it makes any impact.

19

Does your app has a status check or anything?
Is it loading external resources / querying a DB / performing any action like migration before starting?

20

We check health by hitting the root path “/” which is a simple redirect. We are performing migrations before starting the server so the port is not getting opened right away, and the health check doesn’t pass initially. We had to increase the probe delay to allow extra time for migrations.

1 Like