How to retrieve application and infrastructure logs from Loki on my EKS cluster?

[Kmaschta (Kévin Maschtaler)]

Hey there,

I’ve installed Datadog on my cluster, it’s working really well, but we only have 2 weeks of log retention there.

How can I easily retrieve and search for my application logs that were generated before two weeks?

Thanks!

[rophilogene (Romaric Philogene)]

Hi @Kmaschta ,

Correct me if I am wrong, but what you want is getting access to the past logs from Qovery since you only keep Datadog logs for 2 weeks right?

[Kmaschta (Kévin Maschtaler)]

@rophilogene exactly!

[rophilogene (Romaric Philogene)]

Here is what you can do since Qovery stores all your logs for 12 weeks in one of your S3 buckets.

1. Follow this guide to connect to your EKS cluster with kubectl
2. Connect to your Grafana instance running on your Kubernetes cluster.

kubectl -n prometheus port-forward svc/grafana 80:8888

3. Retrieve your Grafana credentials by running

For login

kubectl -n prometheus get secrets/grafana --template='{{.data.admin-user | base64decode}}'

For password

kubectl -n prometheus get secrets/grafana --template='{{.data.admin-password | base64decode}}'

4. Then, you can connect to your Grafana interface on http://localhost:8888.
5. Go to explore and select loki filter with the pod name or namespace.

---

I hope it helps

[Kmaschta (Kévin Maschtaler)]

Ok! I’ve figured this out, thanks.

Small changes I had to made:

-kubectl -n prometheus port-forward svc/grafana 80:8888
+kubectl -n prometheus port-forward svc/grafana 8888:80

Moreover, kubectl was not happy with the dashes in admin-user and admin-password so I just ran the following command:

kubectl -n prometheus get secrets/grafana --template='{{.data}}'

And manually decoded the base64.

Finally the question, what if I want to keep the logs for more than 12 weeks?
This is needed for some regulatory / compliance reasons, and I’d love Qovery to provide an easy way to stream or archive our logs to an S3 or a CloudWatch log group.
Do you provide something like that?

[rophilogene (Romaric Philogene)]

Well done @Kmaschta

@Kmaschta

Small changes I had to made:

-kubectl -n prometheus port-forward svc/grafana 80:8888
+kubectl -n prometheus port-forward svc/grafana 8888:80

Moreover, kubectl was not happy with the dashes in admin-user and admin-password so I just ran the following command:

kubectl -n prometheus get secrets/grafana --template='{{.data}}'

And manually decoded the base64.

Thank you for sharing what you found.

@Kmaschta

Finally the question, what if I want to keep the logs for more than 12 weeks?

We can add an advanced setting at the cluster level for that. @Enzo can you do it?

@Kmaschta

I’d love Qovery to provide an easy way to stream or archive our logs to an S3 or a CloudWatch log group.

Qovery and Loki store your logs already in an S3 bucket. Is it good enough for you?

Thank you

[Kmaschta (Kévin Maschtaler)]

@rophilogene

Qovery and Loki store your logs already in an S3 bucket. Is it good enough for you?

Sure! As soon as the logs are:

1. stored on our own infra, for a period of time we are owning (1yr seems a good start)
2. discoverable at will
3. exportable in another format if needed

Seems like Loki is offering all of that features!