I’ve been noticing some strange behavior in Qovery across multiple different environments the last few weeks. I believe that it is just dropping some of our env vars and I also noticed it even swapped the resources (vcpu and memory) of 2 applications within the same environment. I have checked the audit logs and there are no entries showing that a member of our team changed any of these values (we don’t use CLI).
Below I have attached a screenshot of the latest instance of this. This particular variable was showing it had “last changed” 4 months ago in the dashboard. However, over the weekend it was just changed to an empty string somehow. No indication of any manual activity in the audit logs. As soon as we changed the env var to what we set 4 months ago and redeployed, the application started working again. Unfortunately, this timing doesn’t add up with other redeployments. This environment was last redeployed 5 days ago, but started having an issue 2 days ago.
As you can probably understand, this is quite concerning behavior, so I will gladly provide any more evidence if needed to assist in tracking this down. Thanks.
All 3 of those environments had separate instances of variable values disappearing. It seems like it just happens to random environments. It’s not always the same ones. Specifically for Env3, it occurred on Friday Aug 9, 2024. The environment secret was used in our application at the very minimum Aug 5 - Aug 8 with no issues. Then on Aug 9 our application started failing because the variable was changed somehow. As you can see from the attached screenshot of our Audit logs, we did not change that variable since July 29th. This specific env var was a project level one, but this has also happend for Application scoped env vars as well.
