---
name: Debugging Difficult Bugs
slug: debugging-difficult-bugs-2
category: Quality
description: Add temporary unconditional JSONL logging to runtime code paths to observe elusive bugs, then reproduce the issue and identify the root cause. Ideal when static analysis is insufficient, especially for concurrency, state management, or UI reproduction.
github: "https://github.com/crbnos/carbon/tree/main/.claude/skills/debugging-difficult-bugs"
language: TypeScript
stars: 2281
forks: 325
install: "git clone https://github.com/crbnos/carbon"
added: 2026-07-21T06:31:21.243Z
last_synced: 2026-07-28T07:00:24.168Z
canonical_url: "https://dirskills.com/skills/debugging-difficult-bugs-2"
---

# Debugging Difficult Bugs

Add temporary unconditional JSONL logging to runtime code paths to observe elusive bugs, then reproduce the issue and identify the root cause. Ideal when static analysis is insufficient, especially for concurrency, state management, or UI reproduction.

**Install:** `git clone https://github.com/crbnos/carbon`

## README

# debugging-difficult-bugs — instrument, reproduce, read, then fix

Core idea: when you can't see the failure by reading code, **make the runtime
tell you**. Add temporary append-only JSONL logging along the real code path,
reproduce the real issue once, read the log chronologically, and only then fix.
Never make a second speculative fix without new runtime evidence.

**Announce at start:** "Using the debugging-difficult-bugs skill — instrumenting
the runtime path to observe the failure."

## Step 1: State the uncertainty

Write down: what you believe, what you can't verify statically, and the exact
runtime path that must be observed (route → service → query, edge function, job).

## Step 2: Add temporary unconditional instrumentation

Rules:

- **Unconditional** — never gated behind an env var, debug flag, or log level.
  If reproduction requires remembering to set a flag, it will silently not fire.
- **Append-only JSONL**, one JSON object per line, to a file in the process's
  working directory.
- Log **boundaries and decisions**, not every line: function entry/exit, branch
  decisions with the data that caused them, state before/after mutation, async
  ordering markers, caught errors, return-value shapes.
- Log **shapes, not payloads**: ids, keys, counts, statuses. Never log tokens,
  auth headers, cookies, or full user content.

```ts
import { appendFileSync } from "node:fs";
import { join } from "node:path";

function debugBug(event: string, data: Record<string, unknown> = {}) {
  appendFileSync(
    join(process.cwd(), "debug-difficult-bug.jsonl"),
    `${JSON.stringify({ ts: new Date().toISOString(), event, ...data })}\n`
  );
}

debugBug("service.beforeUpdate", { id, companyId, status: row.status });
```

**Carbon multi-process note.** The ERP/MES dev servers, edge functions (Docker
`edge-runtime` container), and Inngest handlers run as separate processes with
different working directories. Log `process.cwd()` + a process role once at
startup, or use distinct filenames (`debug-erp.jsonl`, `debug-edge.jsonl`). For
edge functions, `console.error` JSON lines (visible in container logs) can stand
in when the container filesystem is awkward to reach.

## Step 3: Reproduce the real issue once

- Prefer reproducing yourself: boot the stack (`crbn up` if not already
  running), authenticate with `/auth`, and drive the exact failing flow with
  `agent-browser` (the `/test` skill documents Carbon's form gotchas —
  `requestSubmit`, react-aria blur).
- If only the user can reproduce (their data, their environment), tell them
  exactly: "I added temporary logging. Reproduce the issue once, then point me
  at `<cwd>/debug-difficult-bug.jsonl`."

## Step 4: Read the log BEFORE fixing

Read chronologically and answer, in writing:

1. Did the instrumented path actually run?
2. What was the expected sequence of events?
3. What was the actual sequence?
4. What is the **first** point where state/order/branch diverges from expectation?

That first divergence is the root cause candidate. Feed it back into the
root-cause brief (or write one now) — then implement via `/fix`, whose failing
regression test must assert the *actual* divergence you observed, not your
earlier assumption.

## Step 5: Clean up — mandatory

- Remove every temporary log call, helper, and import.
- Delete generated `.jsonl` files.
- Check the final diff explicitly for leftovers:
  `git diff | grep -n "debugBug\|debug-difficult\|\.jsonl"` → expect no hits.

The final diff contains only the fix and its tests.

## Done when

- [ ] The first divergence point is identified from log evidence (quote the lines)
- [ ] The fix landed via `/fix` with a red→green regression test asserting that behavior
- [ ] Reproduction of the original flow now passes
- [ ] Zero instrumentation remnants in the diff