Deployment fails missing neonctl

Relevant information to this issue:

ISSUE

I’m getting a deployment error and am at a loss on how to proceed. The first build succeeded but failed to start, probably because the env var scope was wrong (I had set them to Project instead of Environment). I fixed that, but the redeploy looks like it’s showing an old pod stuck in error state:

🛰 Job at new tag 7c8b0e64c628796beb3b7c0a35c899c7103f1d1b has 1 pods: 0 running, 0 starting, 0 terminating and 1 in error
...
❌ job failed !
Either it couldn't be executed correctly after `0` retries or its execution didn't finish after `5.00 minutes`.
Underlying error: `BackoffLimitExceeded: Job has reached the specified backoff limit`.

https://console.qovery.com/organization/0de1b24b-a858-4281-bfd1-374550d2c97e/project/5d636cd9-5abb-4a21-9e01-3337be7d12a9/environment/d4a8800e-1e01-4d9b-80b3-24e011520144/logs/08fcb5fe-cb39-4b7b-932f-4287e54c3064/deployment-logs

UPDATE:
It’s at a point where it’s blocked by neonctl. The latest deploy logs are at
https://console.qovery.com/organization/0de1b24b-a858-4281-bfd1-374550d2c97e/project/5d636cd9-5abb-4a21-9e01-3337be7d12a9/environment/d4a8800e-1e01-4d9b-80b3-24e011520144/logs/08fcb5fe-cb39-4b7b-932f-4287e54c3064/live-logs

error Command "neonctl" not found.

The Dockerfile has that and the logs show it’s installed.

RUN yarn add global neonctl@v1.20.0
`#9 [ 3/11] RUN yarn add global neonctl@v1.20.0`
`#9 0.443 yarn add v1.22.19`
`#9 0.483 info No lockfile found.`
`#9 0.529 [1/4] Resolving packages...`
`#9 2.414 [2/4] Fetching packages...`
`#9 8.511 [3/4] Linking dependencies...`
`#9 10.22 [4/4] Building fresh packages...`
`#9 10.24 success Saved lockfile.`
`#9 10.25 success Saved 91 new dependencies.`
`#9 10.25 info Direct dependencies`
`#9 10.25 ├─ global@4.4.0`
`#9 10.25 └─ neonctl@1.20.0`
...
`#9 DONE 11.0s`

HOW TO REPRODUCE

Select Deploy from the Manage deployment menu. I’ve been reading the docs and will try the Qovery CLI.

qovery log

| 2023-09-12 23:22:04.373 | job-z08fcb5fe-tapi-hkvhx | exec /tini: no such file or directory
ERRO[2023-09-13T13:09:17-05:00] connection closed by server: websocket: close 1011 (internal server error): Remote closed stream

Relevant lines from Dockerfile:

# Add Tini
## https://github.com/krallin/tini#why-tini
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]

# Start Server
CMD [ "npm", "start" ]

Why does tini fail to install? Is this the actual error or a symptom of a deeper issue?

I disabled tini in the Dockerfile to bypass that issue. It’s getting a little further and reporting

error Command "neonctl" not found.

The Dockerfile has that and the logs show it’s installed.

RUN yarn add global neonctl@v1.20.0
`#9 [ 3/11] RUN yarn add global neonctl@v1.20.0`
`#9 0.443 yarn add v1.22.19`
`#9 0.483 info No lockfile found.`
`#9 0.529 [1/4] Resolving packages...`
`#9 2.414 [2/4] Fetching packages...`
`#9 8.511 [3/4] Linking dependencies...`
`#9 10.22 [4/4] Building fresh packages...`
`#9 10.24 success Saved lockfile.`
`#9 10.25 success Saved 91 new dependencies.`
`#9 10.25 info Direct dependencies`
`#9 10.25 ├─ global@4.4.0`
`#9 10.25 └─ neonctl@1.20.0`
...
`#9 DONE 11.0s`

What am I missing?

Hello @Peace_Chen

Did a test on my side with a minimal version Dockerfile such as:

FROM node:20-alpine
RUN npm i -g neonctl@v1.20.0
COPY . ./
ENTRYPOINT ["/bin/sh"]
CMD ["npm", "start"]

And neonctl seems well installed:

/ # type neonctl
neonctl is /usr/local/bin/neonctl

Hello @Melvin_Zottola

Did your test have create-branch.sh to create the Neon branch? Could it be that the create script uses #!/bin/bash whereas the Dockerfile uses /bin/sh? The environment might not transfer from one to the other.

@Melvin_Zottola @rophilogene

Switching Dockerfile to /bin/bash didn’t work, nor did changing create-branch.sh to use sh instead of bash`.

What finally worked was installing neonctl in create-branch.sh, allowing it to find neonctl:

yarn add global neonctl@v1.20.0

This has been a disappointing productivity drain the last few days. I would hope that a promoted feature such as Neon would get more attention, either with more thorough testing & debugging or more direct involvement assisting here.

1 Like

Hello @Peace_Chen ,

Just tested based on the same sources from the neon repo, with a simpler create-branch.sh to see if neonctl is well found such as:

#!/bin/bash

# exit on error
set -e  

# just to see if it recognizes neonctl
yarn -s neonctl 

I get the following output, the command is well recognized:

Do you have some configuration in your repository that may interfer with yarn ?
Also, can you check if it fixes the issue to remove yarn -s prefix to call directly neonctl (without your previous fix to re-install neonctl) ? I’m pretty sure the neonctl is installed and available in your container.

Added this to create-branch.sh

yarn -s neonctl 

Same error as before

Sanitized environment name: dev
warning ../package.json: No license field
error Command "neonctl" not found.

I’m using the Docker image node:18-alpine instead of Ubuntu. It’s a slimmer image, could it be missing something from the larger images?

Actually, you can omit yarn -s, I meant instead this:

yarn -s neonctl branches create \
[...]

can you put this?

neonctl  branches create \
[...]

And you can also do the same for the other occurences in create-branch.sh:

Thanks for the clarification. Removing yarn -s and calling neonctl directly does work :+1:

Logs of the successful deployment:
https://console.qovery.com/organization/0de1b24b-a858-4281-bfd1-374550d2c97e/project/5d636cd9-5abb-4a21-9e01-3337be7d12a9/environment/c9ee9884-2dd1-47b4-b210-776d9a1e3bca/logs/60d77e9e-baa0-4256-9e4b-0bb2f7d78971/deployment-logs

At this point I’d expect to see my server startup logs. The only thing shown in the Live logs is:

Database: ps-dev
Shell script executed successfully with output values - check out your Qovery environment variables :)

Is that expected? I see the EC2 instance in AWS, but the response from that address is Nginx 404.

The deployment logs show the COPY and RUN commands in the Dockerfile, but no output for the commands after that.

# These lines are executed
COPY . ./

RUN npm install
RUN npm run build

# No indication that CMD is executed
EXPOSE 4000 80

ENTRYPOINT [ "/bin/sh" ]

# Start Server
CMD [ "npm", "start" ]

Edit: I’ll open a new ticket.