Deployments fails intermittently

INFORMATION

Relevant information to this issue:

ISSUE

When new deployments are run in Qovery, it fails sometimes. The same command works in GitHub actions and when run locally.

Github actions:

Local:
image

Qovery:

HOW TO REPRODUCE

We have not found a way of replicating this effectively, since it happens intermittently.

1 Like

Hello @jorgeramirezamora,

Unfortunately, there is not much information in the log to investigate the issue.
Maybe adding verbose option could help to determine what is the issue.

Regards.

Hello @Pierre_Gerbelot,

I already added the verbose option to the command described above, I also upgraded the Node version to LTS and I am having the same intermittent issue:

https://console.qovery.com/organization/2fcf38ad-3cfd-4a7d-ab4b-25d5ca78d95e/project/80907b1a-c91e-4dd8-9aa1-63562c81428e/environment/36aaf85e-a6f6-4183-8b73-7e5f5b002f26/logs/52d1e940-21f5-4f70-8ce4-f1cbaf9ac143/deployment-logs/

We had to turn-off PR environments because of the same reason.

Regards

Hello @jorgeramirezamora

After further investigation, the problem is probably due to reaching the memory limit of the builder.


You may try to limit the node.js’s memory usage

Let me know if it fix your issue.

Regards

Hello @Pierre_Gerbelot,

Thanks for the suggestion, I already tried with a 4 GB limit and still does not work.

Just wondering, where does this 8 GB limit come from? The cluster node configuration or the service resource instance configuration?

Regards

Hello @jorgeramirezamora
I’ve checked the memory usage of the builder used for building your application, and the limit of 8GB has still been reached.
It seems that the limit mechanism might not have been configured correctly.

The limit of 8GB comes from the default configuration allocated to the build processes: 4 CPUs and 8GB of RAM.

Regards,

Hello @Pierre_Gerbelot,

Now I see where is coming from: build.cpu_max_in_milli and build.ram_max_in_gib variables in Advanced Settings.

It is weird, I am using the right mechanism which is setting the NODE_OPTIONS environment variable to “–max_old_space_size=X” in my Dockerfile. I already using the same approach in our GitHub Actions and it works correctly there. It has similar specs if we compare it against Qovery ones:

Regards

Hello @jorgeramirezamora
It may be due to a typo in the option you used. It should be max-old-space-size
You can refer to the documentation here: Command-line API | Node.js v22.5.1 Documentation

Let me know if it fixes the issue.

Thank you

Hello @Pierre_Gerbelot,

I tried your suggestion and the problem continues.

Regards

Hell @jorgeramirezamora,

Here is the memory usage of the builder for two deployments. The first deployment is failing, and the second one is succeeding. Would you mind setting the –max-old-space-size to 4GiB and letting me know if the issue is still occurring?

Thank you.

Hello @Pierre_Gerbelot,

The value is already set to 6GB, and as you say, sometimes fails, sometimes succeeds:

ENV NODE_OPTIONS="--max-old-space-size=6144"

image

Regards

Hello @jorgeramirezamora,

Actually, I would like if it possible that you set 4GB instead of 6GB to be sure that the option works with the npx nx run-many command . As stated in the documentation for --max-old-space-size=SIZE (in megabytes):

On a machine with 2 GiB of memory, consider setting this to 1536 (1.5 GiB) to leave some memory for other uses and avoid swapping.

Thank you.

Hello @Pierre_Gerbelot,

Yeah sure, just updated all the services to 4GB. After that, a couple of deployments worked, and then again one failed.

Screenshot 2024-07-26 at 1.09.03 PM

Regards

Hello @Pierre_Gerbelot,

I finally fixed this, adding what I did so if somebody else has this problem can try it out:

  1. I upgraded Nx to version 19.5.x, which supports incremental builds for Vite.
  2. I transformed all packages/libraries into buildable Vite packages/libraries.
  3. I migrated all applications to use SWC as compiler, rather than babel.

If this is not enough, another possibility is to use Rspack rather than Webpack as bundler, but the plugin still is not as complete as the Webpack as of today, but is blazing fast tough.

Thanks for all the support.

Regards

Hello @jorgeramirezamora ,

Thank you for sharing your solution with everyone. I’m sure this will be valuable for others facing a similar issue in the future.

Best regards,

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.