Going through our cluster infrastructure, I noticed our default nginx ingress controller is unable to communicate with the metrics API for autoscaling:
HorizontalPodAutoscaler / nginx-ingress-ingress-nginx-controller
failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
FailedGetResourceMetric
The metric service is running, but, looking at it, its on a different port than the APIService. Is that normal behavior? Should we be taking additional steps to configure something? Again, we are using the default Qovery-generated cluster config apart from parameters we’re configured in Qovery cluster advanced settings.
Still don’t know what the issue is but we ended up having to patch the metrics server manually. This likely caused a drift in our Qovery cluster config but it was the only option we could figure out.
We were able to resolve this and make sure the HPA is able to get metrics from the metric server. However, we are still seeing issues related to the HPA, which I’ve outlined here: VPA (Vertical Pod Autoscaler) issues in EKS Cluster
Is there a way to merge posts? Or feel free to delete, close or achive this one since my other post is the priority one we’re hoping for a response on.
As a general comment, our experience is Qovery managed cluster maintenance has been solid with the exception of HPA and nginx. Even with the cluster advanced settings this has been a bit of a black box, been the root cause of a lot of application infrastructure performance issues, required a lot of investigation and going through New Relic, and it’s not always clear why things are configured the way they are or how we can modify helm charts and other settings without creating config drift (which we ended up having to do any way to allow the HPA to connect to the metrics server). Sharing the extent this is helpful feedback or aligns with what others are experiencing.