---
name: Debugging Difficult Bugs
slug: debugging-difficult-bugs
category: Quality
description: "A systematic debugging approach: add temporary JSONL logging to the real code path, reproduce the bug, and analyze runtime events to find the exact point of divergence before fixing. Use when static analysis gives low confidence or the bug involves runtime state, ordering, concurrency, or manual reproduction."
github: "https://github.com/crbnos/carbon/tree/main/.ai/skills/debugging-difficult-bugs"
language: TypeScript
stars: 2272
forks: 318
install: "git clone https://github.com/crbnos/carbon"
added: 2026-07-14T01:49:54.981Z
last_synced: 2026-07-20T06:48:06.786Z
canonical_url: "https://dirskills.com/skills/debugging-difficult-bugs"
---

# Debugging Difficult Bugs

A systematic debugging approach: add temporary JSONL logging to the real code path, reproduce the bug, and analyze runtime events to find the exact point of divergence before fixing. Use when static analysis gives low confidence or the bug involves runtime state, ordering, concurrency, or manual reproduction.

**Install:** `git clone https://github.com/crbnos/carbon`

## README

# debugging-difficult-bugs — instrument, reproduce, read, then fix

Core idea: when you can't see the failure by reading code, **make the runtime
tell you**. Add temporary append-only JSONL logging along the real code path,
reproduce the real issue once, read the log chronologically, and only then fix.
Never make a second speculative fix without new runtime evidence.

**Announce at start:** "Using the debugging-difficult-bugs skill — instrumenting
the runtime path to observe the failure."

## Step 1: State the uncertainty

Write down: what you believe, what you can't verify statically, and the exact
runtime path that must be observed (route → service → query, edge function, job).

## Step 2: Add temporary unconditional instrumentation

Rules:

- **Unconditional** — never gated behind an env var, debug flag, or log level.
  If reproduction requires remembering to set a flag, it will silently not fire.
- **Append-only JSONL**, one JSON object per line, to a file in the process's
  working directory.
- Log **boundaries and decisions**, not every line: function entry/exit, branch
  decisions with the data that caused them, state before/after mutation, async
  ordering markers, caught errors, return-value shapes.
- Log **shapes, not payloads**: ids, keys, counts, statuses. Never log tokens,
  auth headers, cookies, or full user content.

```ts
import { appendFileSync } from "node:fs";
import { join } from "node:path";

function debugBug(event: string, data: Record<string, unknown> = {}) {
  appendFileSync(
    join(process.cwd(), "debug-difficult-bug.jsonl"),
    `${JSON.stringify({ ts: new Date().toISOString(), event, ...data })}\n`
  );
}

debugBug("service.beforeUpdate", { id, companyId, status: row.status });
```

**Carbon multi-process note.** The ERP/MES dev servers, edge functions (Docker
`edge-runtime` container), and Inngest handlers run as separate processes with
different working directories. Log `process.cwd()` + a process role once at
startup, or use distinct filenames (`debug-erp.jsonl`, `debug-edge.jsonl`). For
edge functions, `console.error` JSON lines (visible in container logs) can stand
in when the container filesystem is awkward to reach.

## Step 3: Reproduce the real issue once

- Prefer reproducing yourself: boot the stack (`crbn up` if not already
  running), authenticate with `/auth`, and drive the exact failing flow with
  `agent-browser` (the `/test` skill documents Carbon's form gotchas —
  `requestSubmit`, react-aria blur).
- If only the user can reproduce (their data, their environment), tell them
  exactly: "I added temporary logging. Reproduce the issue once, then point me
  at `<cwd>/debug-difficult-bug.jsonl`."

## Step 4: Read the log BEFORE fixing

Read chronologically and answer, in writing:

1. Did the instrumented path actually run?
2. What was the expected sequence of events?
3. What was the actual sequence?
4. What is the **first** point where state/order/branch diverges from expectation?

That first divergence is the root cause candidate. Feed it back into the
root-cause brief (or write one now) — then implement via `/fix`, whose failing
regression test must assert the *actual* divergence you observed, not your
earlier assumption.

## Step 5: Clean up — mandatory

- Remove every temporary log call, helper, and import.
- Delete generated `.jsonl` files.
- Check the final diff explicitly for leftovers:
  `git diff | grep -n "debugBug\|debug-difficult\|\.jsonl"` → expect no hits.

The final diff contains only the fix and its tests.

## Done when

- [ ] The first divergence point is identified from log evidence (quote the lines)
- [ ] The fix landed via `/fix` with a red→green regression test asserting that behavior
- [ ] Reproduction of the original flow now passes
- [ ] Zero instrumentation remnants in the diff