CrashLoop Handling - Turn off Celery workers

[Read Only] Questions and Answers Deployment
1

Hi all,

We’ve run into a CrashLoopBackOff on our staging application, and wondering if there are any recommendations on a simpler approach or shortcut to address. Case:

  • Celery started running before Django migrations could occur
  • column django_celery_beat_periodictask.clocked_id didn’t migrate
  • Celery beat keeps trying to run, thereby throwing an err (because required table doesn’t exist), creating the crash loop

We’d like to turn off celery so migrations can run first and deploy the solution, but the celery crashloop is preventing us from doing so. We can “stop” the app in Qovery, but if we deploy or restart, celery starts running again. Similarly, connecting to the app via kubectl to shutdown celery isn’t possible (bc crash loop). We can update all our devops config and infrastructure to shut off celery workers that way, but, before we do that, wondering if there’s a shortcut or easier approach via Qovery? Is this possible in the qovery CLI? Or not possible because celery’s containerized?

  • databases: Postgres (AWS RDS)
  • Programming language and version: Python 3.8.10, Django 2.2
  • Link to your application: Qovery
2

We ended up solving this ourselves, but still would be interested if there are any faster/easier options. thanks