Canceled deployment every time I try to redeploy them

[fv-kuba (Kuba A)]

Seems like that worked, and all users are restored in the configmap

[fv-kuba (Kuba A)]

We just managed to do a deployment as well and it worked.

Do you have any indication as to what caused the issue in the backend? My assumption is that the IAM user for the service got deleted before the service deployment successfully finished and this caused the issue?

[bchastanier (Benjamin Chastanier)]

That’s great !

Indeed, your assumption is good, as explained above:

@bchastanier

Last week we released a change in our stack allowing to fully move Qovery stack to role base instead of user base authentication, see this thread.
To rollout it out, a cluster update was needed in order to:

1. Remove iam-eks-user-mapper user from IAM
2. Create new role for iam-eks-user-mapper in IAM
3. Deploy the new app allowing to grant IAM users from Admins group access to EKS cluster

As to what’s going on on your cluster, I guess it boils down to EKS aws-auth configmap being empty or at least not having stephane’s account anymore hence we cannot access to it via credentials from this account.
The only root cause I can see of this to happen is if somehow there was an issue during your cluster update between step 1 and 2 (aws user account in IAM for the tool have been removed but the new app hasn’t been deployed) the iam-eks-user-mapper app is still the old one trying to use the old user to update the aws-auth configmap but it fails.

It’s likely the update failed somewhere after old user for iam-eks-user-mapper got deleted and before new iam-eks-user-mapper app version got deployed. It should not happen, I will investigate on our end and patch if I find anything which may leads to this situation.