--- name: Datadog Log Investigation slug: datadog-log-investigation category: DevOps description: Datadog Log Investigation queries the Datadog Logs API to investigate production issues for the Nexu platform, including crash events, stderr output, gateway startup, Slack token health, and API HTTP logs. Use it when the user asks to check Datadog logs, alerts, metrics, or crash logs. github: "https://github.com/nexu-io/nexu/tree/main/skills/localdev/datadog" language: TypeScript stars: 3256 forks: 263 install: "npx degit https://github.com/nexu-io/nexu/tree/main/skills/localdev/datadog ~/.claude/skills/datadog" installs_to: ~/.claude/skills/datadog source_path: skills/localdev/datadog/SKILL.md collection_size: 23 category_size: 798 collection_url: "https://dirskills.com/collections/nexu-io/nexu" added: 2026-08-17T07:08:34.912Z last_synced: 2026-08-17T07:08:34.912Z canonical_url: "https://dirskills.com/skills/datadog-log-investigation" --- # Datadog Log Investigation Datadog Log Investigation queries the Datadog Logs API to investigate production issues for the Nexu platform, including crash events, stderr output, gateway startup, Slack token health, and API HTTP logs. Use it when the user asks to check Datadog logs, alerts, metrics, or crash logs. **Install:** ```bash npx degit https://github.com/nexu-io/nexu/tree/main/skills/localdev/datadog ~/.claude/skills/datadog ``` ## README # Datadog Log Investigation Query Datadog Logs API to investigate production issues for the Nexu platform. ## Authentication **Before making any Datadog API call, you MUST ask the user for these two keys:** - `DD_API_KEY` — Datadog API Key (Organization Settings → API Keys) - `DD_APP_KEY` — Datadog Application Key (Organization Settings → Application Keys, requires `logs_read_data` scope) Store them in shell variables for the session. Never hardcode or commit them. Site: `datadoghq.com` (US) ## API Base All requests go to `https://api.datadoghq.com/api/v2/logs/events/search`. Headers: ``` DD-API-KEY: DD-APPLICATION-KEY: Content-Type: application/json ``` ## Common Queries ### OpenClaw Crash Events ```bash curl -s "https://api.datadoghq.com/api/v2/logs/events/search" \ -H "DD-API-KEY: $DD_API_KEY" \ -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ -H "Content-Type: application/json" \ -d '{ "filter": { "query": "service:nexu-gateway @event:openclaw_crash", "from": "now-1h", "to": "now" }, "sort": "-timestamp", "page": {"limit": 20} }' ``` Key fields in results: - `attributes.attributes.exitCode` — process exit code (1 = fatal error, null = signal) - `attributes.attributes.signal` — kill signal (SIGKILL, SIGTERM, etc.) - `attributes.tags` → `pod_name`, `image_tag` — which pod and which version ### OpenClaw stderr Output (Crash Details) ```bash curl -s "https://api.datadoghq.com/api/v2/logs/events/search" \ -H "DD-API-KEY: $DD_API_KEY" \ -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ -H "Content-Type: application/json" \ -d '{ "filter": { "query": "service:nexu-gateway @stream:stderr", "from": "now-1h", "to": "now" }, "sort": "-timestamp", "page": {"limit": 50} }' ``` This shows the actual error output from the OpenClaw process (e.g., `invalid_auth`, `EADDRINUSE`, config validation failures). ### Gateway Startup / Recovery Events ```bash curl -s "https://api.datadoghq.com/api/v2/logs/events/search" \ -H "DD-API-KEY: $DD_API_KEY" \ -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ -H "Content-Type: application/json" \ -d '{ "filter": { "query": "service:nexu-gateway (\"starting gateway\" OR \"gateway is ready\" OR \"spawned openclaw\")", "from": "now-1h", "to": "now" }, "sort": "timestamp", "page": {"limit": 30} }' ``` ### Slack Token Health Check ```bash curl -s "https://api.datadoghq.com/api/v2/logs/events/search" \ -H "DD-API-KEY: $DD_API_KEY" \ -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ -H "Content-Type: application/json" \ -d '{ "filter": { "query": "service:nexu-api slack_token_health*", "from": "now-1h", "to": "now" }, "sort": "-timestamp", "page": {"limit": 20} }' ``` ### API HTTP Request Logs ```bash curl -s "https://api.datadoghq.com/api/v2/logs/events/search" \ -H "DD-API-KEY: $DD_API_KEY" \ -H "DD-APPLICATION-KEY: $DD_APP_KEY" \ -H "Content-Type: application/json" \ -d '{ "filter": { "query": "service:nexu-api http_request @attributes.status:>=500", "from": "now-1h", "to": "now" }, "sort": "-timestamp", "page": {"limit": 20} }' ``` ### Filter by Pod Add `pod_name:` to the query: ``` service:nexu-gateway pod_name:nexu-gateway-1 @event:openclaw_crash ``` ### Filter by Time Window Use ISO 8601 timestamps: ```json { "from": "2026-03-10T05:00:00Z", "to": "2026-03-10T06:00:00Z" } ``` Or relative: `"now-30m"`, `"now-1h"`, `"now-24h"`. ## Parsing Results Use python3 inline to extract key fields: ```bash curl -s ... | python3 -c " import json, sys data = json.load(sys.stdin) events = data.get('data', []) print(f'Total events: {len(events)}') for e in events: attrs = e['attributes']['attributes'] tags = e['attributes']['tags'] pod = next((t.split(':',1)[1] for t in tags if t.startswith('pod_name:')), '?') ts = attrs.get('time', '?') msg = e['attributes'].get('message', '')[:120] print(f'{ts} | pod={pod} | {msg}') " ``` ## Services and Events Reference | Service | Description | |---------|-------------| | `nexu-gateway` | Gateway sidecar (manages OpenClaw process) | | `nexu-api` | API server | | Event | Meaning | |-------|---------| | `openclaw_crash` | OpenClaw process exited unexpectedly | | `openclaw_restart_scheduled` | Sidecar scheduling a restart | | `openclaw_restart_limit` | Max restart attempts exceeded | | `openclaw_orphan_killed` | Killed zombie OpenClaw process | | `slack_token_health_check_invalidated` | Invalid Slack tokens detected and marked | ## Tag Reference | Tag | Example | |-----|---------| | `pod_name` | `nexu-gateway-1`, `nexu-gateway-2` | | `image_tag` | `sha-55f13372bb72abc7db1538cca3db2bcda0d35eba` | | `kube_stateful_set` | `nexu-gateway` | ## Investigation Playbook When investigating a crash: 1. **Check crash events** — get exit codes, signals, timestamps, affected pods 2. **Check stderr** — get the actual error message from OpenClaw 3. **Check startup events** — correlate crash with deploy times (`image_tag` changes) 4. **Check token health** — if `invalid_auth`, look for `slack_token_health_check_invalidated` 5. **Check API logs** — if API errors are contributing ## Rules 1. **Never hardcode API keys** in skill files or logs — always use variables 2. **Default time window** — start with `now-1h`, expand to `now-24h` if needed 3. **Always parse and summarize** — don't dump raw JSON to the user 4. **Correlate across services** — crashes often involve both gateway and API logs 5. **Check image_tag** to determine if crashes are related to a specific deployment