Debugging Difficult Bugs

debugging-difficult-bugs — instrument, reproduce, read, then fix

Core idea: when you can't see the failure by reading code, make the runtime tell you. Add temporary append-only JSONL logging along the real code path, reproduce the real issue once, read the log chronologically, and only then fix. Never make a second speculative fix without new runtime evidence.

Announce at start: "Using the debugging-difficult-bugs skill — instrumenting the runtime path to observe the failure."

Step 1: State the uncertainty

Write down: what you believe, what you can't verify statically, and the exact runtime path that must be observed (route → service → query, edge function, job).

Step 2: Add temporary unconditional instrumentation

Rules:

Unconditional — never gated behind an env var, debug flag, or log level. If reproduction requires remembering to set a flag, it will silently not fire.
Append-only JSONL, one JSON object per line, to a file in the process's working directory.
Log boundaries and decisions, not every line: function entry/exit, branch decisions with the data that caused them, state before/after mutation, async ordering markers, caught errors, return-value shapes.
Log shapes, not payloads: ids, keys, counts, statuses. Never log tokens, auth headers, cookies, or full user content.

import { appendFileSync } from "node:fs";
import { join } from "node:path";

function debugBug(event: string, data: Record<string, unknown> = {}) {
  appendFileSync(
    join(process.cwd(), "debug-difficult-bug.jsonl"),
    `${JSON.stringify({ ts: new Date().toISOString(), event, ...data })}\n`
  );
}

debugBug("service.beforeUpdate", { id, companyId, status: row.status });

Carbon multi-process note. The ERP/MES dev servers, edge functions (Docker edge-runtime container), and Inngest handlers run as separate processes with different working directories. Log process.cwd() + a process role once at startup, or use distinct filenames (debug-erp.jsonl, debug-edge.jsonl). For edge functions, console.error JSON lines (visible in container logs) can stand in when the container filesystem is awkward to reach.

Step 3: Reproduce the real issue once

Prefer reproducing yourself: boot the stack (crbn up if not already running), authenticate with /auth, and drive the exact failing flow with agent-browser (the /test skill documents Carbon's form gotchas — requestSubmit, react-aria blur).
If only the user can reproduce (their data, their environment), tell them exactly: "I added temporary logging. Reproduce the issue once, then point me at <cwd>/debug-difficult-bug.jsonl."

Step 4: Read the log BEFORE fixing

Read chronologically and answer, in writing:

Did the instrumented path actually run?
What was the expected sequence of events?
What was the actual sequence?
What is the first point where state/order/branch diverges from expectation?

That first divergence is the root cause candidate. Feed it back into the root-cause brief (or write one now) — then implement via /fix, whose failing regression test must assert the actual divergence you observed, not your earlier assumption.

Step 5: Clean up — mandatory

Remove every temporary log call, helper, and import.
Delete generated .jsonl files.
Check the final diff explicitly for leftovers: git diff | grep -n "debugBug\|debug-difficult\|\.jsonl" → expect no hits.

The final diff contains only the fix and its tests.

Done when

The first divergence point is identified from log evidence (quote the lines)
The fix landed via /fix with a red→green regression test asserting that behavior
Reproduction of the original flow now passes
Zero instrumentation remnants in the diff

README