I cannot pull a docker image from my AWS ECR private registry

[danielmayor]

Hi, on a Deployment I am trying to pull a docker image from my ECR private registry but I get 401 errors:

#2 [internal] load metadata for xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/docker-image:4.0.0
#2 ERROR: unexpected status from HEAD request to https://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/v2/docker-image/manifests/1.2.3: 401 Unauthorized

I think that the setup of the registry is ok:

I tried to put
xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com

but the UI says that I need to put also https:// so I added it, therefore AWS never displays the registry URL with the protocol.
Finally I used:
https://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com

Thank you

[rophilogene (Romaric Philogene)]

Thank you @danielmayor , can you please provide the permission you set for your access key and secret access key for the credentials you supplied to the Qovery Container Registry configuration?

cc @a_carrano

[danielmayor]

It’s the same IAM policy I use for other non-qovery deployments:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecr:DescribeImageScanFindings",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:DescribeImageReplicationStatus",
                "ecr:DescribeRepositories",
                "ecr:ListTagsForResource",
                "ecr:BatchGetRepositoryScanningConfiguration",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetRepositoryPolicy",
                "ecr:GetLifecyclePolicy"
            ],
            "Resource": "arn:aws:ecr:*: xxxxxxxx:repository/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ecr:GetRegistryPolicy",
                "ecr:DescribeRegistry",
                "ecr:GetAuthorizationToken",
                "ecr:GetRegistryScanningConfiguration"
            ],
            "Resource": "*"
        }
    ]
}

[rophilogene (Romaric Philogene)]

Thank you I’m looking with the team

[Melvin_Zottola]

Hello @danielmayor,

On Qovery side, the purpose of the container registry is to build the image on your own, instead of relying on us to build from a Dockerfile located in your git repository (See Application Docs).

In your case, you are using a git repository with a reference to your private container registry, no authentication is done on our side (there is no link between a “Git Application” and a private container registry) so that explains the 401 logged.

If you really want to use an image from a container registry, you can:

• either build the complete application image on your own and re-create an application using a container registry (we cannot migrate yet to a Git Application to a Container Application)

• or as a workaround, you can push the existing image referenced in your Dockerfile to the private container registry of your cluster:

  • As we mirror the images deployed, we login to this container registry so we should be able to pull an image from this registry

[danielmayor]

Thank you @Melvin_Zottola, that was very useful.

I will try the second suggestion. How can I push an image to the private container registry of my cluster?

[rophilogene (Romaric Philogene)]

Hi @danielmayor ,

The Container Registry that Qovery creates is accessible on the same region as your AWS EKS cluster - so you need credentials with the correct permissions to push your container image on it. That’s it!

Example

For instance, in my case, my container registry created by Qovery started with 8f84becf. Cf. Screenshot below:

So, in my case, my repository on ECR might start with z8f84becf. This is where I can push my images, accessible to Qovery via the already-connected ECR.

[rophilogene (Romaric Philogene)]

@danielmayor, thanks for providing more context in our call.

To resolve this, we suggest setting up a delegated role permission from your primary ECR repository to the IAM Qovery user you’ve already configured. This approach leverages AWS’s cross-account role delegation capabilities, allowing the Qovery user to access your ECR repository with the necessary permissions.

By granting these permissions, Qovery can authenticate with your ECR registry effectively. This should resolve any issues related to the ‘FROM’ command in your Dockerfile, which points to an external ECR registry.

For a more detailed guide on how to set up these permissions, please refer to the following AWS documentation links:

1. Creating IAM Roles - This page provides step-by-step instructions on how to create an IAM role in your AWS account.
2. Tutorial: Using IAM Roles for Cross-Account Access - This tutorial walks you through the process of setting up cross-account access using IAM roles.

Let me know if that works for you

[danielmayor]

@rophilogene

delegated role permission from your primary ECR repository

I have tried to set up the suggested setup following the documentation but the deploy keeps failing.
#2 ERROR: unexpected status from HEAD request to https://ecr-account-arn.dkr.ecr.us-east-1.amazonaws.com/v2/docker-image/manifests/4.0.1: 401 Unauthorized

1. I created a role in the ECR Account with this arn "arn:aws:iam::<ecr-account-arn>:role/QoveryECRRead" and this policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecr:DescribeImageScanFindings",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:DescribeImageReplicationStatus",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:ListTagsForResource",
                "ecr:BatchGetRepositoryScanningConfiguration",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetRepositoryPolicy",
                "ecr:GetLifecyclePolicy"
            ],
            "Resource": "arn:aws:ecr:*:<ecr-account-arn>:repository/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ecr:GetRegistryPolicy",
                "ecr:DescribeRegistry",
                "ecr:GetAuthorizationToken",
                "ecr:GetRegistryScanningConfiguration"
            ],
            "Resource": "*"
        }
    ]
}

2. I added a trust relationship to this role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<qovery-account-arn>:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

3. Finally, I added this inline policy to the existing Qovery user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<ecr-account-arn:role/QoveryECRRead"
        }
    ]
}

If I try to assume the role in my local though aws cli aws sts assume-role I am able to run commands, so I assume that the auth is well configured.

Is this setup supported already by Qovery? I guess if I’m doing anything wrong. Thank you!

[Pierre_Mavro (Pierre Mavro - CTO & CO-Founder)]

We’re looking into it, we’ll get back to your shortly!

[Pierre_Mavro (Pierre Mavro - CTO & CO-Founder)]

Hi,

After tests on our side, it looks like you’re right! AWS requires STS to login with the assume role, while we do not manage it on our side at the moment.

We have to think about it on the product side (@a_carrano @Julien_Dan) and find a solution that will support any private registries during the build phase when it’s different from the target.

We’ll get back to you soon when we’ll start to work on it product side.

Thanks for your understanding

[danielmayor]

Thank you, @Pierre_Mavro

I ended up pushing the Docker image to a public ECR repository and now the app works fine!

[Julien_Dan]

Hi @danielmayor,
We plan to manage the docker build with a base image from a private registry.
Basically, you’ll have to add your container registry to Qovery and Qovery will do the rest. But we have a few questions for you:

• Is your AWS ECR in the same organization/region as your cluster deployed by Qovery?

• If so, when you will add your AWS ECR access to Qovery, we’ll need an access to be able to pull and push images from this registry. Is that ok for you?

Don’t hesitate to get in touch with us if you have any questions.
Julien

[jackm-ts (Jack Miller)]

Hello,

Our ECR is in the same organization, (different account) and same region, however the ECR is replicated to other regions.
It is okay for you guys to have access to the registry in the other AWS account.

[Erebe (Romain Gerard)]

Hello @jackm-ts

The feature is released, you should be able to build an image from a base image belonging to a private container registry.

To do so, you need to create a registry in your organization with a URL that match the one you set in your dockerfile with the FROM xxxx directive.

After you have created the registry in your orga, you should be able to deploy as usual. During your build phase, you should see that we log in into this new registry to be able to pull the image.

Let me know if you have some issue, so I can take a look at it.

[jackm-ts (Jack Miller)]

Looks great thank you!