Terraform error when planing

hosmelq · June 19, 2023, 7:41pm

INFORMATION

Relevant information to this issue:

OS: MacOS

ISSUE

I’m getting an error when trying to deploy an application, everything were working until now.

❯ terraform plan
data.qovery_cluster.main: Reading...
data.qovery_organization.prison_fellowship: Reading...
qovery_project.pf_digital: Refreshing state... [id=xxx]
data.qovery_organization.prison_fellowship: Read complete after 0s [id=xxx]
module.environment["Production"].qovery_environment.environment: Refreshing state... [id=xxx]
module.environment["Staging"].qovery_environment.environment: Refreshing state... [id=xxx]
data.qovery_cluster.main: Read complete after 0s [id=xxx]
module.environment["Staging"].qovery_database.postgres[0]: Refreshing state... [id=xxx]
module.environment["Production"].qovery_application.app: Refreshing state... [id=xxx]
module.environment["Staging"].qovery_application.app: Refreshing state... [id=xxx]

Planning failed. Terraform encountered an error while generating this plan.

│
│ Error: Provider produced invalid plan
│
│ Provider "registry.terraform.io/qovery/qovery" planned an invalid value for
│ module.environment["Staging"].qovery_application.app.environment_variables: count in plan (cty.UnknownVal(cty.Number)) disagrees with
│ count in config (cty.NumberIntVal(8)).
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
│
│
│ Error: Provider produced invalid plan
│
│ Provider "registry.terraform.io/qovery/qovery" planned an invalid value for
│ module.environment["Staging"].qovery_application.app.secrets: count in plan (cty.UnknownVal(cty.Number)) disagrees with count in
│ config (cty.NumberIntVal(8)).
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
│

pfmskeff · June 19, 2023, 8:08pm

Adding some context. Hosmel here is working on our deployments from us.
This issue and others appears to correlate with the Kubernetes upgrade. All of our environments went down. When we try to deploy our apps we get the following types of errors:

Application at commit XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX deployment is in progress , below the current status:
┃ Kubernetes load balancer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX is READY
┃
┃ Application has 2 pods. 0 starting, 0 terminating and 2 in error
┃ |__ Pod XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX is FAILING crash loop, pod is restarting too frequently. Look into your application logs
┃ |__ Pod crashed 8 times
┃ |__ Back-off restarting failed container
┃ |__ Container image “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX” already present on machine
┃ |__ Created container XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
┃ |__ Pod XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX is FAILING crash loop, pod is restarting too frequently. Look into your application logs
┃ |__ Pod crashed 6 times
┃ |__ Back-off restarting failed container
┃ |__ Container image “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX” already present on machine
┃ |__ Created container XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
┃
┃
┃ Need Help ? Please consult our FAQ to troubleshoot your deployment Troubleshoot | Docs | Qovery and visit the forum https://discuss.qovery.com/
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Helm command UPGRADE for release Error on XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX helm upgrade terminated with an error: CommandError { full_details: Some(“Helm timed out for release XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX during helm UPGRADE: Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Command still running. No output available. Waiting for next line…Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: timed out waiting for the condition: release XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX failed: timed out waiting for the conditionrelease XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX failedhelm.sh/helm/v3/pkg/action.(*Rollback).performRollback\thelm.sh/helm/v3/pkg/action/rollback.go:215helm.sh/helm/v3/pkg/action.(*Rollback).Run\thelm.sh/helm/v3/pkg/action/rollback.go:79helm.sh/helm/v3/pkg/action.(*Upgrade).failRelease\thelm.sh/helm/v3/pkg/action/upgrade.go:478helm.sh/helm/v3/pkg/action.(*Upgrade).reportToPerformUpgrade\thelm.sh/helm/v3/pkg/action/upgrade.go:343helm.sh/helm/v3/pkg/action.(*Upgrade).releasingUpgrade\thelm.sh/helm/v3/pkg/action/upgrade.go:402runtime.goexit\truntime/asm_amd64.s:1371an error occurred while rolling back the release. original upgrade error: timed out waiting for the conditionhelm.sh/helm/v3/pkg/action.(*Upgrade).failRelease\thelm.sh/helm/v3/pkg/action/upgrade.go:479helm.sh/helm/v3/pkg/action.(*Upgrade).reportToPerformUpgrade\thelm.sh/helm/v3/pkg/action/upgrade.go:343helm.sh/helm/v3/pkg/action.(*Upgrade).releasingUpgrade\thelm.sh/helm/v3/pkg/action/upgrade.go:402runtime.goexit\truntime/asm_amd64.s:1371UPGRADE FAILEDmain.newUpgradeCmd.func2\thelm.sh/helm/v3/cmd/helm/upgrade.go:202github.com/spf13/cobra.(*Command).execute\tgithub.com/spf13/cobra@v1.2.1/command.go:856github.com/spf13/cobra.(*Command).ExecuteC\tgithub.com/spf13/cobra@v1.2.1/command.go:974github.com/spf13/cobra.(*Command).Execute\tgithub.com/spf13/cobra@v1.2.1/command.go:902main.main\thelm.sh/helm/v3/cmd/helm/helm.go:87runtime.main\truntime/proc.go:225runtime.goexit\truntime/asm_amd64.s:1371: Command terminated with a non success exit status code: exit status: 1”), message_safe: “Error while executing Helm command.” }
Deployment of Application failed ! Look at the report above and to understand why.
Need Help ? Please consult our FAQ to troubleshoot your deployment Troubleshoot | Docs | Qovery and visit the forum https://discuss.qovery.com/
Deployment failed
Qovery Engine has terminated the deployment

pfmskeff · June 22, 2023, 7:54pm

It would appear that our issue stemmed from the Docker image and config we were using.
It worked on K8S 1.23 but would not work on 1.24.

More specifically there were some issues with Apache permissions which affected port binding. We also leveraged the s6-overlay in our Docker config.