Datadog Helm deploy failing

Trying to deploy Datadog monitoring via Helm using the Qovery documentation tutorial: Kubernetes observability and monitoring with Datadog | Qovery

I followed the docs with two modifications:

for the YAML values I set:

site: datadoghq.com
clusterName: {{ name of our cluster }}

Everything else is step-by-step through the docs. On deploy, getting:

ready.go:277: [debug] Deployment is not ready: zaef3c6a8-datadog/helm-zcc8ddd55-datadog-cluster-agent. 0 out of 1 expected pods are ready
ready.go:304: [debug] DaemonSet is not ready: zaef3c6a8-datadog/helm-zcc8ddd55-datadog. 0 out of 2 expected pods are ready

And the deploy ultimately times out.

Is there a resource allocation step that needs to happen? thanks

https://console.qovery.com/organization/828d9c2c-fd72-4b4e-ab58-71de40ecfdd2/project/d825456b-8b8a-4fc1-af7c-5e828fdc2707/environment/aef3c6a8-acb5-4632-a38e-48d80f1fe372/application/cc8ddd55-3d6e-4fc9-9699-7570b7615ce2/general

I see this log message API Key invalid from Datadog.

Did you check the live logs? https://console.qovery.com/organization/828d9c2c-fd72-4b4e-ab58-71de40ecfdd2/project/d825456b-8b8a-4fc1-af7c-5e828fdc2707/environment/aef3c6a8-acb5-4632-a38e-48d80f1fe372/logs/cc8ddd55-3d6e-4fc9-9699-7570b7615ce2/live-logs

Yet we’re using a correct, valid API key for qovery.env.DD_API_KEY and have that defined as a Service variable. But still the connection (and deploy) is failing. Ok will look into this further. thanks

2024-03-22 12:04:30 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:70 in Error) | check:datadog_cluster_agent | Error running check: [{"message": "HTTPConnectionPool(host='10.1.58.31', port=5000): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efdf5c5bca0>: Failed to establish a new connection: [Errno 111] Connection refused'))", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connection.py\", line 174, in _new_conn\n conn = connection.create_connection(\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/util/connection.py\", line 95, in create_connection\n raise err\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n httplib_response = self._make_request(\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 416, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connection.py\", line 244, in request\n super(HTTPConnection, self).request(method, url, body=body, headers=headers)\n File \"/opt/datadog-agent/embedded/lib/python3.9/http/client.py\", line 1285, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/opt/datadog-agent/embedded/lib/python3.9/http/client.py\", line 1331, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/opt/datadog-agent/embedded/lib/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/opt/datadog-agent/embedded/lib/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/opt/datadog-agent/embedded/lib/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connection.py\", line 205, in connect\n conn = self._new_conn()\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connection.py\", line 186, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7efdf5c5bca0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/requests/adapters.py\", line 486, in send\n resp = conn.urlopen(\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 799, in urlopen\n retries = retries.increment(\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/urllib3/util/retry.py\", line 592, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.1.58.31', port=5000): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efdf5c5bca0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/base.py\", line 1235, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/base_check.py\", line 142, in check\n self.process(scraper_config)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 581, in process\n for metric in self.scrape_metrics(scraper_config):\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 508, in scrape_metrics\n response = self.poll(scraper_config)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 845, in poll\n response = self.send_request(endpoint, scraper_config, headers)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 871, in send_request\n return http_handler.get(endpoint, stream=True, **kwargs)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/utils/http.py\", line 355, in get\n return self._request('get', url, options)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/utils/http.py\", line 419, in _request\n response = self.make_request_aia_chasing(request_method, method, url, new_options, persist)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/datadog_checks/base/utils/http.py\", line 425, in make_request_aia_chasing\n response = request_method(url, **new_options)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/requests/api.py\", line 73, in get\n return request(\"get\", url, params=params, **kwargs)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, **kwargs)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/opt/datadog-agent/embedded/lib/python3.9/site-packages/requests/adapters.py\", line 519, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='10.1.58.31', port=5000): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efdf5c5bca0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"}]

The site variable needs to be the specific Datadog hosting url for your account, including subdomain. Using datadoghq.com wouldn’t work, and I had to, in our case, switch it to us5.datadoghq.com

Worth mentioning this in the docs.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.