Datadog APM Setup

Questions and Answers Deployment
1

Hi, We want to setup APM using Datadog. The datadog docs require updates to application pod template spec.
This seems like something that is not currently possible.

Is there any workaround?

2

Hey @r4881t,

That’s something we are not supporting right now, but we are working on it, more info in this thread.

I don’t have any workaround for you just now … Stay tuned for future support.

Sorry for the inconvenience,

Best

3

Actually, by reading Datadog doc, labels are necessary but this is only needed when the datadog admission controller is enabled.

It seems you should be able to activate it by settin the DD_ENV , DD_SERVICE and DD_VERSION env vars manually in qovery. From the link you shared.

Let me know if you managed to make it works.

Best

4

I did the following steps and that didn’t work. Can you provide any more info on how to do this? This is a very important for us to be able to do apm on some of our apps.

We tried with the code injection (manual) way first and found that it doesn’t work in the kubernetes deployment. That’s because it’s agent URL defaults to 127.0.0.1:8126 and in our case, we need to find this everytime it changes (due to app pod restart or datadog agent pods restart). This seemed tricky.

Hence we are using the automatic way (injecting libraries locally)

Steps for Automatic APM Setup:

  1. We are using the “datadog operator” method.
  2. The default container registry is gcr.io, which is fine
  3. Since, using Qovery, we can not annotate the pods, so we need to configure datadog to inject without checking for specific labels. To do this, we added following config to our “datadog operator” file
  features:
    admissionController:
      enabled: true
      mutateUnlabelled: true

  1. Now DD wants us to annotate the pods, but that won’t be possible with Qovery. If we could have
    we would annotate it with admission.datadoghq.com/js-lib.v4.16.0: "<CONTAINER IMAGE TAG>". Also everytime a new deployment is done, fetching the CONTAINER IMAGE TAG will be tricky. So this way clearly doesn’t work.

  2. I added the following env variables on my app using Qovery Console

    1. DD_VERSION as an alias of QOVERY_APPLICATION_ZD6368D9D_GIT_COMMIT_ID

    We have the following Datadog specific Env variables available

    Screenshot 2023-10-17 at 12.45.04 PM
    Screenshot 2023-10-17 at 12.45.04 PM702×431 30.2 KB

  3. Now, we need to configure the Admission Controller to specify communication between the application and Datadog agent. This is hostip by default, which won’t work. So we have to choose service here. This inject Datadog’s local-service DNS name in DD_AGENT_HOST environment variable. I made this change in the datadog operator yaml file, and now my file looks like below

  features:
    admissionController:
      enabled: true
      mutateUnlabelled: true
      agentCommunicationMode: service
  1. Now, I apply this config
apply -f ./helm-charts/datadog-values-aws-singapore.stag.yaml
datadogagent.datadoghq.com/datadog configured

  1. Since Admission Controller only intercepts pods created after the admission controller was deployed, we need to redeploy our services. So I did a “redeploy” of my NodeJS API -Server from the Qovery Console.

  2. After restarting, I wait for 15 minutes to see anything showing up in Datadog APM UI. Nothing happens. So I check if the pod has been set it’s env variable correctly.

    When doing the describe pod for my app, I can confirm that the env variable DD_AGENT_HOST is set. So it seems like the datadog config is working.

    Screenshot 2023-10-17 at 12.50.58 PM
    Screenshot 2023-10-17 at 12.50.58 PM842×61 7.4 KB

So at this step, I don’t know what else to do.

5

@bchastanier @Pierre_Mavro -

There may be two options here only.

  1. Annotate the pods: Is there a way to annotate pods using the Lifecycle jobs? Maybe I can write a script that will annotate the pods after they are created?

  2. How can my app pod always know the latest IP address of my datadog agent pod? This may be the case when I want to use the manual way of code injection by specifying the IP address of the DD Agent Pod.

6

Hello @0xbitmonk

Regarding your first question, we currently do not offer the option to annotate the pods.
However, we may consider implementing this feature in the future. You can vote for it in the product roadmap.

As for your second question, please note that the IP address of the agent pod can change during the application’s lifecycle due to random IP allocation.
To address this issue, you may consider placing your agent behind a service and using the service name instead of the IP address.

However, I encourage you to review the response provided by Melvin_Zottola in this discussion:

or contact Datadog support.

Regards,

7

I reached to Datadog as well, their responses have been

If you can’t annotate the pods then you can’t use the APM Library Injection method for tracing. You will need to do the manual setup following the instructions on this doc: Sending Traces to Datadog

we don’t have support for the automatic library injection without the ability to annotate to your pods.

There isn’t a workaround for this. The way to establish the connection between the app and the agent is outlined in this setup depending on your configuration: Sending Traces to Datadog

For kubernetes, the agent and the application are on the same host so it would just use a socket, or you can use TCP over port 8126 on the local host where the agent is listening on.

There is no setup if the agent is on a completely different host.

I get your solution (putting Datadog agent behind a service), but I am a developer and not a DevOps persona per-se. So I find these things extremely hard to do. If you can elaborate a bit more, it would be very helpful.

While I understand this may not be a priority for your team right now, I do expect some more example/blog around this. It’s because you mention DataDog as the recommended way for observability and monitoring and this page talks about APM without actually describing the steps for it. Similarly, the Datadog Qovery Tutorial also just mentions APM without actually mentioning how to do it.

8

Hello @0xbitmonk,

As the pod annotation is not available, you can instrument your application using the manual method. You can refer to the forum post I previously shared, where you will find an example of manual injection for a Java application. There is also a link to an application example along with its Dockerfile.

For the pod agent, as said by the Datadog support,

For kubernetes, the agent and the application are on the same host so it would just use a socket, or you can use TCP over port 8126 on the local host where the agent is listening on.

so why do you think you can’t do the manual injection?

We tried with the code injection (manual) way first and found that it doesn’t work in the kubernetes deployment. That’s because it’s agent URL defaults to 127.0.0.1:8126 and in our case, we need to find this everytime it changes (due to app pod restart or datadog agent pods restart).