What is the correct way to add annotations for Datadog logging?

[0xbitmonk (Pranav Prakash)]

Hi, Thanks for releasing the custom labels and annotations. I had been waiting for this feature for a long time. Today, our team decided to set this up and configure datadog logging using it. However, we are facing some issues.

Steps we did

1. Setup Datadog in our cluster as described in this doc. This works great. We have a few services that use datadog SDK to log stuff, and we can see those in the datadog console.
2. We have a few other services that don’t use datadog SDK and we want to have their logs available in the datadog console as well.
3. So we are trying adding datadog specific labels and annotations as described on their website.
4. We have the following label group and annotations group setup

Annotation Keys:

ad.datadoghq.com/eoa_fere_prod.logs

Annotation Values:

[{"source": "eoa_fere","service": "eoa_fere_prod","tags": ["env:production"]}]

Annotation Scope selected: PODS

5. We have set this to the service

6. When we try to deploy the service with these values, I keep getting the error

I have isolated and found that this happens only when the annotations are set. If I try to deploy with just the labels, it is deployed. However when I try to deploy with the annotations, that’s when I get the error.

My service console URL is https://console.qovery.com/organization/0f2a7baf-84e9-4b4e-a219-72fb44811f99/project/1e2ddd37-24f7-4e38-aaf1-250a5987410a/environment/f4e9e073-8bac-4015-af45-a5bbc5ac64ec/application/124af404-7f64-4339-a9e0-bfddf7e80064/general

[Melvin_Zottola]

Hello @0xbitmonk,

I’m going to take a look

[Melvin_Zottola]

@0xbitmonk can you try to change your annotation values by escaping the quotes ? such as:

[{\"source\": \"eoa_fere\",\"service\": \"eoa_fere_prod\",\"tags\": [\"env:production\"]}]

[0xbitmonk (Pranav Prakash)]

Yes this worked like a charm. Thanks @Melvin_Zottola . I think you should also get the documentation at Labels & Annotations | Docs | Qovery updated. Specifically the images. I saw the images and they were without quotes so I tried that first

[Melvin_Zottola]

Yes completely.
We are also goint to improve this part, to automatically escape the input

[Melvin_Zottola]

Hello @0xbitmonk,

We are going to release a fix for the initial use case.
Once the release will be done on our side, it would require on your side to remove the escaped quotes (it would become what you had at the beginning: [{"source": "eoa_fere","service": "eoa_fere_prod","tags": ["env:production"]}])
Is it ok for you if I ping you in this thread once the release is done on our side ?

[0xbitmonk (Pranav Prakash)]

Hi,

Yes that would be amazing. Thanks

[Melvin_Zottola]

Hey @0xbitmonk ,
We are going to release it in a few minutes

[Melvin_Zottola]

The release is done, I updated your annotations so the next time you deploy it will be fine

[0xbitmonk (Pranav Prakash)]

Awesome. Thanks a lot for this. Love your team.