We are preparing some monitors for our pods to be informed about high memory consumption.
One thing I noticed is Promtail agent and VPA memory consumptions are almost always high. Even when we put alerts for
97%+ VPA exceeds it often.
So I was curious to know if its expected or should we take some actions about it ? (edited)
That is expected. We have attached to it a vertical autoscaler, which is going to increase CPU and/or Memory as it see fit. So it should always be close to the limit all the time to reduce costs and avoid resource waste.
If they end-up being killed due to an OOM, the autoscaler is going to increase the resources for the pods.