--- name: ML Debugging slug: ml-debugging category: Quality description: ML Debugging systematically diagnoses ML and AI training failures such as OOMs, NaNs, divergence, crashes, slow throughput, wrong outputs, and dependency conflicts. Use it when an experiment or model run breaks and you need root-cause analysis before applying fixes. github: "https://github.com/Leeroo-AI/superml/tree/main/skills/ml-debug" language: Python stars: 194 forks: 18 install: "npx degit https://github.com/Leeroo-AI/superml/tree/main/skills/ml-debug ~/.claude/skills/ml-debug" installs_to: ~/.claude/skills/ml-debug source_path: skills/ml-debug/SKILL.md collection_size: 7 category_size: 1662 collection_url: "https://dirskills.com/collections/Leeroo-AI/superml" added: 2026-09-05T05:31:31.040Z last_synced: 2026-09-05T05:31:31.040Z canonical_url: "https://dirskills.com/skills/ml-debugging" --- # ML Debugging ML Debugging systematically diagnoses ML and AI training failures such as OOMs, NaNs, divergence, crashes, slow throughput, wrong outputs, and dependency conflicts. Use it when an experiment or model run breaks and you need root-cause analysis before applying fixes. **Install:** ```bash npx degit https://github.com/Leeroo-AI/superml/tree/main/skills/ml-debug ~/.claude/skills/ml-debug ``` ## README # ML Debugging Systematically diagnose ML failures using framework-specific knowledge, not guesswork. ## Grounding **Detect mode:** On your first grounding call, check if Leeroopedia KB tools are available. If they return results, use **KB mode**. If unavailable or auth fails, use **Web mode**. **HARD RULE: You MUST ground before writing analysis.** If KB fails, you MUST WebFetch at least 2 URLs before writing ANY diagnosis. Writing from memory without fetching is the #1 failure mode of this skill — it produces zero-citation responses that score 0/3 on grounding. "I know X well" is NOT a substitute for fetching documentation. **KB mode:** Call `diagnose_failure` → `query_hyperparameter_priors` → `search_knowledge`. Cite as `[PageID]`. **KB mode grounding supplement (MANDATORY):** After KB calls, you MUST WebFetch at least 2 public URLs (official docs, PyPI, GitHub issues/source) and cite them as `[source](URL)` alongside `[PageID]` citations. KB-only responses score 2/3 max on grounding because reviewers cannot verify proprietary page IDs. Pattern: KB call for diagnosis → WebFetch PyPI for version → WebFetch GitHub/docs for config verification → cite BOTH KB and public URLs in every section. **Self-test**: if your response has zero `[source](URL)` citations, you will lose a grounding point regardless of KB citation count. **Web mode:** WebFetch GitHub issues for the error message → WebFetch framework troubleshooting docs → WebFetch config references. Cite as `[source](URL)`. Start response with: `> Grounding: Web mode — citations from official docs and GitHub issues.` **Web mode grounding targets by response section** (aim for these counts): - Diagnosis root cause: 1+ citation (to a specific doc section or GitHub source line, NOT a top-level page) - Each "Why it matters" explanation: 1+ citation or `[no KB]` - Each fix step: 1+ citation for the specific API/config being changed - Each quantitative claim ("X× faster"): 1 citation or `[no KB]` - Prevention items: 1+ citation for the metric/tool referenced Target: 5+ total citations in web mode, each linking to a specific doc section or source line (not top-level domain pages). Below 3 is a grounding failure. Generic page links (e.g., `huggingface.co/docs/transformers`) score lower than specific section links (e.g., `huggingface.co/docs/transformers/model_doc/mixtral#MixtralConfig`). **PyPI pages are version-only citations**: A PyPI link confirms a version number but contributes zero technical content. Do NOT count PyPI links toward your citation minimum. You need 3+ citations that contain *technical claims* (API signatures, config defaults, known failure modes). PyPI fetches are a prerequisite step, not a citation source. When citing GitHub source files, use a tagged release URL (e.g., `github.com/huggingface/transformers/blob/v4.45.0/src/...`) NOT the `main` branch. **MANDATORY first WebFetch in web mode**: fetch the framework's PyPI page (`https://pypi.org/project//`) to get the current stable version — then use that version tag in ALL subsequent GitHub URLs. **Do NOT fabricate version numbers** — if you haven't fetched the PyPI page, you don't know the current version. Writing "DeepSpeed 0.18.7" or "Transformers v5.3.0" without fetching is fabrication. **Multi-package rule**: If your diagnosis involves N frameworks, you need N PyPI fetches — one per package. Fetching DeepSpeed's version does NOT tell you the Transformers version. Each `**Version**:` line must link to the specific PyPI page it came from. If you cannot fetch any URLs, use the Ungrounded response format. **Self-test**: before writing your response, count your `[source](URL)` citations. If < 2, you have not met the minimum bar — fetch more or switch to Ungrounded format. **Web mode response template** (use this structure when in web mode): ``` > Grounding: Web mode — citations from official docs and GitHub issues. > Sources fetched: [URL1], [URL2] ## Diagnosis **Root cause**: [one sentence] [source](URL) **Version**: [exact version from fetched docs] [source](URL) ... ``` **Web mode URL registry:** - PyTorch issues: `https://github.com/pytorch/pytorch/issues` - HF Transformers issues: `https://github.com/huggingface/transformers/issues` - DeepSpeed issues: `https://github.com/microsoft/DeepSpeed/issues` - vLLM issues: `https://github.com/vllm-project/vllm/issues` - PEFT docs: `https://huggingface.co/docs/peft` - Axolotl issues: `https://github.com/axolotl-ai-cloud/axolotl/issues` - vLLM docs: `https://docs.vllm.ai/en/latest/` - PyTorch docs: `https://pytorch.org/docs/stable/` - DeepSpeed docs: `https://www.deepspeed.ai/docs/config-json/` - PyPI (version lookup): `https://pypi.org/project//` (use to find current stable version) - HF Transformers releases: `https://github.com/huggingface/transformers/releases` - vLLM releases: `https://github.com/vllm-project/vllm/releases` ## The Iron Law ``` NO FIX WITHOUT UNDERSTANDING THE ROOT CAUSE FIRST ``` Applying fixes without diagnosis leads to fix-on-fix layering. The third "fix" usually breaks something the first fix was hiding. ``` NO CLAIM WITHOUT A [PageID] CITATION — OR MARK IT [no KB] ``` Every factual claim about framework internals requires either a `[PageID]` citation or an explicit `[no KB]` tag. There is no middle ground. "I have deep knowledge" is not a citation. ``` IF KB CALL FAILS → EVERY SECTION HEADER GETS ⚠️, EVERY CLAIM GETS [no KB], EVERY CONFIDENCE IS "Low" ``` This is an Iron Law, not a suggestion. A response that looks normal but has zero `[PageID]` and zero `[no KB]` tags is the #1 grounding failure. ``` NO WRITING FROM MEMORY — IF KB FAILS, YOU MUST WEBFETCH BEFORE WRITING ``` "I know X well" or "I have deep knowledge of X" followed by ungrounded analysis is the #1 anti-pattern. If KB fails: WebFetch docs first, then write. No exceptions. ## Phases **Hard rule: You MUST look things up before writing any analysis.** **WHEN KB FAILS — DO THESE 3 THINGS IMMEDIATELY (no prose first):** 1. `WebFetch` the framework's PyPI page to get the current version 2. `WebFetch` GitHub issues for the exact error message 3. `WebFetch` the framework's config/API docs for the feature involved Then write your response using the Web mode template. If WebFetch also fails, use the Ungrounded format. **If you write ANY text before completing these 3 WebFetch calls, your response scores 0.** ### Phase 1: Gather Evidence — Diagnose Immediately **MANDATORY FIRST ACTION**: KB fails → call WebFetch 3× (see checklist above) → THEN write. No prose before fetching. **KB mode:** Call `diagnose_failure(symptoms, logs)` with everything available: - **symptoms**: What's failing, what was expected, when it started - **logs**: Error lines, stack trace, unexpected output, metrics timeline **Web mode:** Search for the error using WebFetch (you MUST actually call WebFetch — do not skip this and write from memory): **CHECKPOINT: If KB failed and you have not yet called WebFetch, STOP HERE. Call WebFetch NOW. Do not write diagnosis text. Do not explain why KB failed. Do not claim expertise. Call WebFetch.** 1. WebFetch GitHub issues for the framework + exact error message (e.g., `https://github.com/huggingface/transformers/issues?q=`) 2. WebFetch the framework's official docs page for the relevant feature/config 3. If config-related, WebFetch the framework's config documentation or changelog for the user's version 4. Extract specific facts: version-specific defaults, config key names, known bug numbers — these become your citations 4b. If the problem involves a specific model, WebFetch its `config.json` from HuggingFace (e.g., `https://huggingface.co///raw/main/config.json`) — extract `num_key_value_heads`, `num_hidden_layers`, `hidden_size` for any memory/KV math 5. WebFetch the framework's latest release/tag page to pin the exact version — all subsequent citations must reference this version, not `main` branch 6. **Multi-framework version rule**: If the user's setup involves multiple frameworks (e.g., DeepSpeed + Transformers + PEFT), WebFetch the PyPI page for EACH framework separately. Do NOT reuse a version fetched for one package as if it applies to another, and do NOT extrapolate version numbers. Every version number in your response must trace to a specific fetched page. **FABRICATION TRAP**: If you write a version number that did NOT appear in a fetched page, it is fabricated — even if it "looks right". Common fabrications: `transformers 5.x` (does not exist as of 2025, it's 4.x), `deepspeed 0.18+` (verify the actual latest). If you catch yourself writing a version from memory, STOP and fetch the PyPI page. Do NOT guess at the cause. Look it up first — framework-specific failure patterns are well-documented. **Gate**: You have a documentation-grounded diagnosis with a root cause hypothesis before proposing any fix. Every diagnosis MUST include at least one citation — `[PageID]` in KB mode, `[source](URL)` in Web mode. **If you have zero citations at this point, STOP — go back and WebFetch something. Do not proceed to Phase 2 without at least one grounded citation.** **KB mode gate**: If using KB mode, you must ALSO have at least one `[source](URL)` citation from a public URL by this point. If not, WebFetch now before proceeding. KB citations alone are insufficient for full grounding credit. **Ungrounded response format** (use ONLY if both KB and web search fail): ``` ⚠️ Ungrounded — no documentation sources available. All claims below are [unverified]. ## Diagnosis **Root cause**: [one sentence] [unverified] **Confidence**: Low **Version**: [framework version] [unverified] **Evidence**: [what in the logs/symptoms points to this] ### Fix 1. [specific action] [unverified] 2. [verification step] [unverified] ### If That Doesn't Work - [alternative] [unverified] ``` ### Phase 2: Confirm the Diagnosis Ask yourself before fixing: - Is this **deterministic** (happens every time) or **intermittent** (timing/race condition)? - Does it happen on **first step** (config/setup issue) or **step N** (accumulation/overflow)? - Is it **one GPU** or **all GPUs** (distributed-specific vs general)? - What **changed** since it last worked? If the diagnosis is ambiguous: 1. Call `propose_hypothesis(current_status, recent_experiments?)` for ranked alternatives 2. Call `query_hyperparameter_priors(query)` if the diagnosis points to config values **Gate**: You can explain the root cause in one sentence and say why the proposed fix addresses it. **Confidence calibration**: Only mark "High" confidence when ALL of: (1) the root cause is a well-documented, widely-reproduced failure mode, (2) you have direct log evidence, (3) you have a KB or public doc citation confirming the exact mechanism, AND (4) the mechanism is unconditional (not "if X is set wrong, then Y"). If ANY condition in your causal chain is speculative or requires an assumption about the user's config that you haven't verified, mark "Medium" max. Conditional hypotheses ("this happens IF max_steps is set incorrectly") are NEVER High confidence. Plausible ≠ confirmed. **Causal mechanism rule**: If you claim "X causes Y because of Z", you need evidence for the *mechanism*. Without a `[PageID]` or `[source](URL)`, mark it speculative: "X may cause Y (speculative) `[no KB]`". Do NOT present speculative mechanisms in a "Why it hurts" format that implies certainty — use conditional language and tag `[no KB]`. **Arithmetic verification rule**: If your diagnosis includes ANY math (memory budgets, KV cache sizes, batch calculations, step counts), you MUST show the full calculation with labeled inputs. Every input number must come from a fetched source (model config, framework docs) or be marked `[assumed]`. Do NOT use round numbers from memory — e.g., "Mistral has 32 KV heads" is wrong (it has 8); fetch `config.json` to verify. **Quantitative claim sourcing rule**: If you state a ratio or magnitude (e.g., "30× stronger gradient", "4× memory reduction"), you MUST either (a) show the arithmetic derivation with sourced inputs, or (b) cite a source that states the ratio directly. Unsourced ratios presented as facts are a grounding violation — tag them `[no KB]` if you cannot derive or cite them. **Version pinning rule**: State the exact framework version (e.g., `vLLM 0.6.0`, not just "vLLM") in both Diagnosis and Fix. Config keys, CLI flags, and defaults change across versions — unversioned advice is unverifiable and hurts specificity. **Hard ceiling**: No `[PageID]` → max confidence "Medium". No KB at all → max confidence "Low", every claim tagged `[no KB]`. Do not use hedging language ("likely", "in practice", "typically") to present ungrounded claims as authoritative. Before choosing a fix, ask: **What is the least destructive intervention?** Prefer targeted fixes (reset one component, adjust one parameter) over broad ones (restart from scratch, lower all learning rates). If resuming from a checkpoint after a failure (e.g., expert collapse, NaN), check whether optimizer state or specific weights need resetting — a full restart may discard recoverable work. When multiple hypotheses exist, **order by diagnostic cost**: test the hypothesis that takes minutes (e.g., check `git diff`, inspect data, do arithmetic on step counts) before the one that requires a full training run. If simple arithmetic (e.g., steps × batch_size = dataset_size) explains the symptom, lead with that — don't bury it as a fallback behind a speculative mechanism. **Verify your arithmetic end-to-end**: if you claim "epoch ends at step N, so step M is the boundary", check that M actually equals N — off-by-one or rounding errors in your own math undermine the diagnosis. **Causal chain precision rule**: If your math shows event X at step N but the symptom is at step M (where M ≠ N), do NOT invent a speculative bridging mechanism ("corruption propagates N-M steps"). Instead: (1) check if your math is slightly off, (2) check for off-by-one in logging vs optimizer steps, (3) if the gap remains unexplained, state it honestly: "The arithmetic predicts step N; the spike at step M is ~K steps later, suggesting [an additional factor / logging offset] that needs investigation." Confidence drops to Medium when the predicted and observed steps don't match. **Grounding depth rule**: When citing KB sources, prefer citations that include version-specific details (changelogs, config schemas, API signatures) over general principle citations. **Contradictory framing check**: Before presenting your diagnosis, re-read it for internal contradictions. Common traps: - Stating a default value, calling the user's value "too high", then recommending an even higher value - Saying a coefficient is "10× too high relative to default" AND "too weak to be effective" in the same paragraph — pick ONE framing: either it's too high (recommend lowering) or too weak (recommend raising). If the value is high relative to default but weak relative to competing gradients, frame it as: "Despite being above default, the effective signal is too weak because [competing gradient reason]" — do NOT call it "too high" if you're about to recommend raising it further. For any parameter change, state clearly: (1) the framework default, (2) the user's current value, (3) your recommended value, (4) why the direction of change is correct. If a KB result contains a specific version note or API detail, surface it in your response — e.g., "parameter `X` was renamed to `Y` in v0.5.0 [PageID]" is stronger grounding than "see [PageID] for details". Cross-reference multiple KB sources when available to strengthen confidence. **Direction-of-change sanity check**: After deciding on a parameter change, verify in one sentence: "The current value is X, the default is Y, I'm recommending Z, which moves it [higher/lower] because [mechanism]." If your mechanism says the value is "too high" but your fix raises it further, you have a contradiction — resolve it before writing. Similarly, if your mechanism says the value is "too weak" but your fix lowers it, stop and reconsider. **Specificity rule**: Every diagnosis and fix must name the user's exact model, GPU type, framework version, and relevant config values — not generic placeholders. "Llama-3-70B on 4×A100-80GB with vLLM 0.6.0" not "large model on multi-GPU". Reviewers score specificity by counting concrete details that match the user's setup. **Mandatory correctness checks before writing diagnosis:** 1. If computing KV cache: verify `num_key_value_heads` (NOT `num_attention_heads`) — GQA models have 4-8× fewer KV heads 2. If citing memory numbers: show the full arithmetic (params × bytes + optimizer × bytes + activations + KV cache) 3. If citing config keys: verify the key exists in the stated framework version 4. If citing improvement percentages: cite a source or mark `[no KB]` — never assert "30-50% improvement" without a citation **Citation extraction rule**: When a KB call returns results, extract and quote the specific relevant detail — don't just append a `[PageID]`. BAD: "Speculative decoding helps here [PageID]" GOOD: "Speculative decoding generates N tokens per draft step, reducing decode passes by ~4× for acceptance rate >0.7 [PageID]". The citation must add information the reader can verify, not just authority. ### Phase 3: Fix + Verify 1. Apply the fix — provide specific config changes, code patches, or commands 2. Include a **runnable verification script** — not just prose instructions. For training: a code block that runs N steps and prints the metric to check. For serving: an async load-test script using `asyncio.gather()` with N concurrent requests matching the user's stated concurrency — sequential loops do NOT test concurrency. Measure and print p50/p95/p99. For OOM: include `torch.cuda.max_memory_allocated()` check. Single-request latency checks are insufficient for serving fixes. The script must test the SPECIFIC failure that was diagnosed — not a generic health check. 3. **Correctness gate**: Before presenting any fix, verify every API call, import, and config key against KB results or fetched docs. In web mode, WebFetch the framework's API reference for any function you're about to recommend. Wrong API calls (e.g., nonexistent methods, deprecated parameters) are worse than no fix. Every config value must include the exact key path (e.g., `engine_args.gpu_memory_utilization`, not just "gpu_memory_utilization"). **Unverified flag protocol**: If you cannot verify a CLI flag or config key exists in fetched docs, you MUST: (1) mark it `[unverified flag]` inline, (2) provide a fallback command using only verified flags, (3) include a one-liner to test the flag: ` --help | grep `. Never present an unverified flag as the primary recommendation — always lead with verified alternatives. **CLI flag and config key verification**: Before writing ANY CLI command or config dict, WebFetch the framework's arg parser source or config schema to verify exact names. Common errors: inventing JSON-style flags when the framework uses separate flags, using deprecated names, or **fabricating config keys that don't exist** (e.g., `use_router_z_loss` is NOT a standard DeepSpeed MoE key). If you cannot verify a key/flag exists in the framework source, mark it `[unverified key]` — do NOT present fabricated keys as real config options. Fabricated config keys silently do nothing, which is worse than no fix. 3. If the fix involves hyperparameters, include the recommended range from KB 4. Every fix action MUST cite a `[PageID]` — if you cannot cite one, call `search_knowledge()` for that specific fix before presenting it 5. Pin the framework version in every fix AND diagnosis: state the exact version tested (e.g., `vLLM 0.6.0`, `transformers 4.41.0`) in both the Diagnosis and Fix sections — config keys, CLI flags, and internal behaviors change across versions, and unversioned advice is unverif