Qovery dropping environment variables and changing configs

Hello,

I’ve been noticing some strange behavior in Qovery across multiple different environments the last few weeks. I believe that it is just dropping some of our env vars and I also noticed it even swapped the resources (vcpu and memory) of 2 applications within the same environment. I have checked the audit logs and there are no entries showing that a member of our team changed any of these values (we don’t use CLI).

Below I have attached a screenshot of the latest instance of this. This particular variable was showing it had “last changed” 4 months ago in the dashboard. However, over the weekend it was just changed to an empty string somehow. No indication of any manual activity in the audit logs. As soon as we changed the env var to what we set 4 months ago and redeployed, the application started working again. Unfortunately, this timing doesn’t add up with other redeployments. This environment was last redeployed 5 days ago, but started having an issue 2 days ago.

As you can probably understand, this is quite concerning behavior, so I will gladly provide any more evidence if needed to assist in tracking this down. Thanks.

It just happened again.

Hi @colin ,

can you please share the Qovery URL of the application having this issue?

It’s happened on at least these 3 envs.

on those 3 environments the variable value disappeared? is it always the same environment?

if we focus on one of those 3, do you have also a timeframe for when the issue happened?

thanks, I’m trying to gather more info for the investigation

All 3 of those environments had separate instances of variable values disappearing. It seems like it just happens to random environments. It’s not always the same ones. Specifically for Env3, it occurred on Friday Aug 9, 2024. The environment secret was used in our application at the very minimum Aug 5 - Aug 8 with no issues. Then on Aug 9 our application started failing because the variable was changed somehow. As you can see from the attached screenshot of our Audit logs, we did not change that variable since July 29th. This specific env var was a project level one, but this has also happend for Application scoped env vars as well.
image

Hi Colin,

sorry for my late reply.

the team has investigated this point but we couldn’t find any trace of the rootcause… is it something you have seen again recently?

No. I haven’t seen it since August 9th.

Ok thanks.

Unfortunately, I can’t provide you with more information on what happened and I’m really sorry for this.

What I propose is to close this topic but feel free to ping us again on the forum if you find the issue and we will investigate as a top priority.

Alessandro

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.