Our Qovery production cluster suddenly went down and is returning “403 Forbidden” errors. Was there any change deployed from the Qovery side?
Hello !
Do you have any application returning 403? If so, can you please share Qovery console URL? Any logs / monitoring showing those 403 so I can have a deeper look?
Cheers
We narrowed it down to “network.ingress.whitelist_source_range” option. It no longer detected incoming IP addresses correctly and we had to turn it off. I see there was a cluster deployment today, was there a relevant change on Qovery side?
1 Like
From what we saw in your NGINX logs, some 403 comes for IP 13.40.94.198 for example, which wasn’t in the network.ingress.whitelist_source_range. Any changes on that front? What’s the service calling this app?
Our IP whitelist is set to Cloudflare IP range. It did not change. I think the problem is that IPs are not reported correctly to nginx and likely contain IPs of the AWS Load Balancer or other component along the path to the nginx.
We found a lead, we are working on it and will get back to you ASAP.
Hi just to inform that I have the same problem on my production cluster
Any updates on resolving the issue? We are close to peak traffic hours.
Thanks,
Simon
We are preparing a patch and rollout will happen soon.
That’s great, thanks for the update!
It should be fixed, you can re-set your network.ingress.whitelist_source_range.
Let us know if it is not the case.
We are still looking to understand how this happened, as the change concerned should not produce this impact/make ip whitelist break.
Please let us know the full story once you get to the bottom of it.
I would also like to request a feature to allow setting IP whitelist at the LB security group level. Since we set the whitelist on all our applications anyway, I would find it a lot more reliable if it was applied to the load balancer rather than inside nginx config. It would also serve the purpose even better since the blacklisted traffic wouldn’t even reach nginx.
Hello @prki ,
The problem has been solved.
You can find a first status here: 403 received Nginx customer side - Incident details - Qovery - Status
We will provide more information on this incident as soon as we are done collecting all the details.
Regards,
Charles-Edouard