App environment:
ISSUE
We are using DataDog and have many log pipelines setup in our account. By default, most of these pipelines use the SOURCE attribute of the datadog log to identify the correct log parser (for example, to associate ruby logs with the ruby parser, or nginx logs with the nginx parser).
DataDog uses pod annotations to set the value of SOURCE as shown here: Kubernetes Log collection
For example:
apiVersion: v1
kind: Pod
metadata:
name: redis
annotations:
ad.datadoghq.com/redis.logs: '[{"source": "redis","service": "redis","tags": ["env:prod"]}]'
labels:
name: redis
We are not able to set annotations on our Qovery pods though, so DataDog by default uses the image name, which is the name of our mirror cluster registry. As a result, the logs aren’t associated with the correct parser, and don’t end up being parsed at all.
Are there plans to allow custom annotations on Qovery services? It looks like we may be able to do this with Helm charts, but our preference is to use the built in Qovery services, not Helm charts.
HOW TO REPRODUCE
Prerequisites:
- A DataDog account
- A Qovery project using services, not Helm charts.
Steps:
- Install the DataDog agent into your cluster: Install the Datadog Agent on Kubernetes
- Verify logs are coming across in DataDog
- Observe the “Source” attribute in a log entry. It will be set to “qovery-mirror-cluster-*” for all log entries.
- Navigate to Logs → Configuration - Pipeline in your DataDog account. You will see most pipelines are configured using the source attribute.