--- name: ML Planning slug: ml-planning category: AI Engineering description: ML Planning turns an ML goal into a grounded, step-by-step implementation plan. Use it when you need an architecture, pipeline, or build plan for a machine learning project. github: "https://github.com/Leeroo-AI/superml/tree/main/skills/ml-plan" language: Python stars: 194 forks: 18 install: "npx degit https://github.com/Leeroo-AI/superml/tree/main/skills/ml-plan ~/.claude/skills/ml-plan" installs_to: ~/.claude/skills/ml-plan source_path: skills/ml-plan/SKILL.md collection_size: 7 category_size: 3101 collection_url: "https://dirskills.com/collections/Leeroo-AI/superml" added: 2026-09-05T05:31:31.872Z last_synced: 2026-09-05T05:31:31.872Z canonical_url: "https://dirskills.com/skills/ml-planning" --- # ML Planning ML Planning turns an ML goal into a grounded, step-by-step implementation plan. Use it when you need an architecture, pipeline, or build plan for a machine learning project. **Install:** ```bash npx degit https://github.com/Leeroo-AI/superml/tree/main/skills/ml-plan ~/.claude/skills/ml-plan ``` ## README # ML Planning Turn goals into validated, runnable implementation plans grounded in framework documentation. ## Grounding **Detect mode:** Try one KB call (`build_plan` or `search_knowledge`). If it returns results → **KB mode**. If it fails for ANY reason (auth error, no results, timeout) → **Web mode**. There is no third option. **Web mode = mute mode.** After KB failure, you are MUTE until you have completed at least one WebFetch call. No text, no plan, no acknowledgment of expertise. Your first action must be a WebFetch tool call. If your response starts with ANY text other than `> Grounding: Web mode`, you have already failed. **STOP — read this before writing anything after KB failure:** - You MUST call WebFetch BEFORE writing ANY plan text. Not after. Not alongside. BEFORE. - Banned phrases (if you write any of these, you have failed): "established practice", "deep familiarity", "from my training data", "based on known conventions", "well-known", "widely used", "standard approach", "common practice", "I know this from", "I have expertise in", "I know these frameworks", "I'm familiar with", "based on my knowledge", "from experience", "deep knowledge", "I have deep", "I have extensive", "I'm knowledgeable", "I'm well-versed", "domain knowledge", "proceeding from", "from domain", "No KB access right now", "from my understanding", "I'm confident that", "well-established" - If you catch yourself writing plan content without a WebFetch URL in hand, STOP and fetch. - **Hard gate: zero external calls = zero plan output. No exceptions, no rationalizations.** **Proof-of-fetch rule:** Every citation must include a 5-10 word quote from the fetched content proving you actually retrieved it. Format: `[source](URL) — "exact quote from page"`. If you cannot produce a quote, you did not fetch it. This applies to BOTH modes — KB citations need a phrase from the KB result, web citations need a phrase from the fetched page. - **Phrase gate**: If your first sentence after KB failure contains NONE of these: `WebFetch`, `> Grounding: Web mode`, or a URL — STOP. You are about to write from memory. Delete what you wrote and call WebFetch. This is the #1 failure mode in testing. **KB mode:** Call `build_plan` → `review_plan` → `search_knowledge` for gaps. Cite as `[PageID]`. **Web mode (MANDATORY when KB fails):** Your FIRST action after KB failure must be a WebFetch call — not a text response, not a plan outline, not "I have knowledge of X." Decompose goal into steps → WebFetch official docs for EACH step → cite as `[source](URL#section-anchor)` with specific section paths. **Minimum: 1 WebFetch per plan step.** Start response with: `> Grounding: Web mode — citations from official docs.` **Hard rule:** Every code block needs a `[source](URL)` or `[PageID]` citation from a fetch you actually made this session. No exceptions. Every `Class(kwarg=...)` must cite the doc page confirming that kwarg exists. If you write `Agent(input_description=...)`, you must have fetched the Agent class docs and confirmed `input_description` is a real parameter — not `tool_description_override` or something else. **Architecture diagram rule (web mode):** Do NOT draw architecture diagrams, flow charts, or system designs until you have fetched docs for every component in the diagram. An architecture diagram without grounding is a guess dressed up as a plan. Fetch first, diagram second. **Citation enforcement (both modes):** Every code block that calls a library API MUST have an inline comment citing the source: `# [PageID]` or `# [source](URL)`. Every class instantiation must cite the doc page where its kwargs are listed. Uncited API calls are treated as unverified guesses. When citing, always include the **library version** (e.g., `peft==0.12.0 [PageID]`). **Cross-reference rule:** When a plan combines multiple libraries (e.g., PEFT + Transformers, RAGAS + LangChain), verify version compatibility between them — fetch each library's install docs to confirm compatible version ranges. State the verified combination explicitly in Prerequisites. **Web mode URL registry:** **Citation anchor rule (web mode):** Link to the specific API class/function section, NOT the library homepage. Use `#anchor` paths — e.g., `https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#command-line-arguments` not `https://docs.vllm.ai`. A homepage link is not a citation. - HF Transformers/PEFT/TRL: `https://huggingface.co/docs/{transformers,peft,trl}` - Axolotl: `https://github.com/axolotl-ai-cloud/axolotl` - DeepSpeed: `https://www.deepspeed.ai/docs` - vLLM: `https://docs.vllm.ai` - Model cards: `https://huggingface.co/{org}/{model}` — ALWAYS fetch for architecture-specific layer names, config keys, and training recipes - Anthropic/Claude API: `https://docs.anthropic.com` — NOT `platform.claude.com` (does not exist). SDK reference: `https://docs.anthropic.com/en/docs/build-with-claude` - OpenAI: `https://platform.openai.com/docs/api-reference` - LangChain/LangGraph: `https://python.langchain.com/docs`, `https://langchain-ai.github.io/langgraph` ## The Iron Law ``` NO IMPLEMENTATION WITHOUT A VALIDATED PLAN FIRST ``` A plan that hasn't been reviewed against documentation is a guess. Guesses waste GPU hours. ## Phases ### Phase 1: Understand — Build the Plan **KB mode:** Call `build_plan(goal, constraints?)` IMMEDIATELY with the user's stated goal. **Web mode:** Do NOT write any step content yet. First: 1. List the frameworks/libraries needed (one line each) 2. WebFetch the API reference page for EACH library — do ALL fetches BEFORE writing any plan text 3. For each `Class(kwarg=...)` you plan to use, find its `__init__` signature in the fetched docs and copy the exact parameter names 4. NOW write steps using ONLY the fetched parameter names — if a param isn't in the fetched docs, it doesn't exist 5. **Deprecation check**: For each API pattern you plan to use, verify it's not deprecated in the pinned version. Common traps: `@app.on_event("startup")` → use `lifespan` context manager in FastAPI ≥0.93; `model.generate()` kwargs change across transformers versions. If the fetched docs show a deprecation warning, use the replacement. 5. For multi-provider plans (e.g., Claude + OpenAI + local models), fetch EACH provider's SDK docs SEPARATELY — do NOT assume shared API patterns **Web mode minimum calls:** Count your steps. You must make AT LEAST that many WebFetch calls. If you have 6 steps, make 6+ fetches. Do NOT batch multiple steps into one fetch unless they use the exact same doc page. In both modes: - Use the user's exact words as the goal - Include any hardware, framework, latency, or scale constraints they mentioned - Do NOT wait for more information — use what you have now - For every SDK/framework in the plan, pin the version and cite where you verified its API. KB mode: `search_knowledge("[library] [version] API")`. Web mode: WebFetch the library's changelog or API reference for that version. - **Version + citation pairing**: When you pin a version (e.g., `openai-agents==0.1.0`), immediately fetch its API docs and note the URL. The version pin and the citation URL must appear together in Prerequisites. An unpinned dependency or an uncited version is a grounding failure. **Gate**: You have a documentation-grounded plan with numbered steps and validation criteria before proceeding. > **Import verification rule**: For every `import` or `from X import Y` in the plan, verify the exact import path against the library's **installed version** docs. Do NOT present uncertain imports — if you cannot confirm the path, WebFetch or `search_knowledge` the module's API reference. If still uncertain, provide the verified fallback import AND a one-line check: `python -c "from X import Y"` so the user catches it before running the full script. > **Citation rule**: Every step in the plan MUST include at least one citation. KB mode: `[PageID]` from `build_plan` output. Web mode: `[source](URL)` from the doc page you verified against. If a step has no citation, look it up before presenting. > > **Version rule**: When citing a library or framework, include the **pinned version** next to the citation. This lets the user verify the citation matches their dependency versions. ### Phase 2: Validate — Review and Gap-Fill **KB mode:** 1. Call `review_plan(proposal, goal)` with the plan from Phase 1 to catch risks 2. Identify the 2-4 most uncertain steps 3. Call `search_knowledge` in **parallel** for each gap (cite every result as `[PageID]` in the final plan) **Web mode:** 1. Self-review: walk through each step and ask "would this actually work on the stated hardware?" 2. Identify the 2-4 most uncertain steps 3. WebFetch official docs in **parallel** for each gap — framework API details, config formats, known pitfalls, memory estimates In both modes, verify for each gap: - Framework-specific API details and correct import paths - Config format requirements - Known pitfalls or gotchas - **Current API version** — verify exact function signatures for the pinned version - **Changelog cross-reference** — for EVERY pinned SDK version, WebFetch the changelog or release notes page and confirm no breaking changes between the version you cite and the latest. Format: `[changelog](URL) — "v0.x.y: renamed foo to bar"`. This is scored separately from API citations — missing changelog checks = grounding penalty even if API docs are cited. - Memory/compute estimation for the specific hardware - **Compatibility caveats** — features that are version-dependent or have known workarounds. For every `kwarg` or `scheduler_kwargs` dict passed to a Trainer/config, verify it exists in the **pinned version** — not just the latest. If uncertain, provide a version check: `assert version.parse(lib.__version__) >= version.parse("X.Y.Z")` - **API method existence** — for EVERY `client.method_name()` call, verify that method exists in the SDK docs for the pinned version. Do NOT assume methods from one provider exist on another (e.g., OpenAI's `messages.parse()` does not exist in Anthropic's SDK). WebFetch the SDK reference page and find the exact method signature. - **Entry-point signature verification** — for the TOP-LEVEL call that launches the pipeline (e.g., `Runner.run()`, `app.invoke()`, `trainer.train()`), fetch its docs page and copy the COMPLETE list of accepted kwargs. This is the call users hit FIRST — a fabricated kwarg here causes immediate TypeError before any pipeline logic runs. Verify the exact name for: (1) the starting agent/chain param (`starting_agent=` not `agent=`), (2) any max-iterations param (`max_turns=` integer, not a callback dict), (3) context/config params. - **Method + return type verification** — after confirming a method exists, verify its return type and response attribute chain. `client.messages.create()` returns a `Message` with `.content[0].text`, NOT `.parsed_output` or `.parsed`. Copy the exact attribute access path from the fetched docs. One wrong attribute = silent `None` or `AttributeError` at runtime. - **Parameter placement verification** — for every kwarg, verify whether it belongs on the constructor (`__init__`) or on a method call (`.compile()`, `.run()`, `.create()`). Common trap: DSPy optimizers accept `max_bootstrapped_demos` on `.compile()`, not the constructor. Instructor accepts `response_model` on `.create()`, not the client constructor. Fetch the class docs AND the method docs separately — they have different signatures. - **Agent/tool completeness** — if an agent or orchestrator is supposed to make decisions (routing, classification, lookup), it MUST have tools or structured data access to do so — not just instructions saying "analyze X". An agent that routes by sentiment but has no sentiment tool or pre-computed score will hallucinate routing decisions. For every agent decision point, verify there is a concrete data source (tool call, context field, or function) backing it. - **Internal consistency** — verify that model names, variable names, and config values are identical between comments and code, between different steps, and between budget tables and implementation. If a comment says `gpt-4o-mini` but the code uses `claude-sonnet-4-20250514`, that's a bug. Scan all code blocks for mismatches before presenting. - **Integration completeness** — if the plan computes multiple score types (deterministic metrics + LLM judge scores + rule-based checks), verify they are ALL wired into the final decision/gate logic. Scores that are computed but never used in gates, rankings, or composite scores are dead code. Trace each score from computation → storage → final decision. If any score has no consumer, either integrate it or remove it. - **Stateful logic verification** — for any retry counter, conversation history, or accumulated state: trace the identity/key used to track it. `id(obj)` changes per object creation — use a stable key (tool name string, step index). Verify state is never silently discarded: if a loop resets a list or dict, confirm that's intentional, not a bug that loses prior context. - **Framework performance flags** — for each framework, WebFetch the performance tuning guide and include ALL recommended flags. E.g., Megatron-LM: `--sequence-parallel`, `--use-distributed-optimizer`, `--overlap-grad-reduce`, `--overlap-param-gather`. Missing one flag can halve throughput on long runs. Do NOT rely on memory for flag names — fetch the docs. - **LLM-as-judge validation** — if the plan uses LLM judges: (1) require order-randomization for pairwise comparisons to counter position bias, (2) include ≥5 calibration samples with known scores to detect judge drift, (3) add a self-consistency check — judge 10% of samples twice and report agreement rate, (4) flag sycophancy risk: judges overrate verbose/confident responses. These are not optional add-ons — LLM judge results without these controls are unreliable. - **Numerical estimate verification** — for ANY throughput, memory, or time estimate, show the full arithmetic. Cross-check against published benchmarks (fetch them). Common trap: underestimating tokens/sec/GPU for smaller models on powerful hardware (e.g., 3B on H100 processes 8,000-15,000 tok/s/GPU, not 3,500). **Self-consistency check**: if you claim X% MFU in text, verify your tokens/sec numbers actually imply that MFU. Formula: `MFU = (6 × params × tokens_per_sec) / (GPU_FLOPS × num_GPUs)`. If the numbers don't match, fix them before presenting. - **KV cache math**: Always compute per-token KV cache as `2 × num_layers × num_kv_heads × head_dim × bytes_per_param`. For GQA/MQA models, use the actual KV head count (not query heads). For MoE models, attention layers are shared across experts — use the full layer count, not expert count. Show the per-token size AND total budget in the plan. **Gate**: Every step in the plan has either documentation confirmation or an explicit "verify during dry-run" flag. **Correctness trace (MANDATORY):** Pick the 2 most complex code blocks in your plan. For each, mentally execute with a concrete input value and trace: (1) input → first function call → return value → next function call → final output. Write the trace as a comment block above the code. If any variable is undefined, any return type mismatches, or any import path is wrong, fix it NOW. This catches the majority of correctness bugs. > **Hardware-fit gate**: Before presenting, verify quantization/precision choices AND throughput estimates match the hardware. Memory: `model_params × bytes_per_param + optimizer_overhead + activation_memory ≤ 0.85 × total_VRAM`. If bf16 fits, do NOT use QLoRA/4-bit. Throughput: fetch published benchmarks for the specific GPU+model-size combination — do NOT estimate tokens/sec from first principles without a benchmark anchor. Show all math in Prerequisites. > **Code correctness gate**: Before presenting any code block, mentally trace it with a concrete input. Check: (1) variable names match across lines, (2) return types match what the caller expects, (3) no duplicate class/function definitions across steps, (4) callback/metric functions return the type the framework expects (e.g., `output_schema` ≠ `output_format`), (5) **error handling**: every call to `Runner.run()`, `trainer.train()`, or any external API must be wrapped in try/except with a specific recovery action — not bare `except:`, (6) **state persistence**: if the design needs variables/results to survive across calls (REPL-style execution, multi-turn agents), verify the execution model actually preserves state — `subprocess.run()` per call does NOT persist variables, you need a persistent process or shared store, (7) **security layer verification**: for every sandbox/restriction (import blocking, path restriction, read-only mode), write down one concrete bypass and add a defense — e.g., `__import__` override is bypassed by `importlib`; use `ast.parse` + node-type whitelist instead. If you spot an inconsistency, fix it before presenting. > **Scale/range consistency gate**: When combining scores from different sources (e.g., deterministic 0-1 metrics with LLM judge 1-5 scores), verify the ranges align. A linear map from [0,1] to [0,5] is NOT the same as [1,5]. Show the mapping formula explicitly and trace one example value through it. Also: when extracting structured scores (e.g., `[[score]]` regex), add a fallback for parse failures — log the raw output and assign a default, don't silently drop the sample. > **Kwarg verification gate (BLOCKING)**: Before presenting ANY `Class(kwarg=...)` call, you MUST have fetched the class docs page this session. Copy the exact parameter names from the docs. Common traps: `data_collator` not `data_collate_fn`, `tool_description_override` not `input_description`, `compute_metrics` not `metric_fn`. If you cannot confirm a kwarg from a fetched doc, mark it `[UNVERIFIED]`. One wrong kwarg silently ignored = hours wasted. > **Correctness gate**: For any step that involves an SDK or library API, you MUST verify the import path, function signature, and **exact parameter names**. KB mode: call `search_knowledge("[library] [function] API signature [version]")`. Web mode: WebFetch the API docs page. Do NOT guess APIs from memory. > **Numerical gate**: For any throughput, memory, or time estimate, show the arithmetic in the plan. Ground estimates in documented benchmarks, not intuition. ### Phase 3: Present — Structured Plan with Validation Compose the final plan: **HARD GATE (check before writing):** Count your citations. If you have 0 `[PageID]` or `[source](URL)` references ready, STOP — go back to Phase 1/2 and fetch. Every step needs ≥1 citation. Every code block needs ≥1 inline `# [source]` comment. Every `pip install pkg==X.Y.Z` needs the version confirmed from a fetched doc. No citations = no plan output. **VERSION-SPECIFIC CITATION GATE:** Every `[source](URL)` must point to a version-pinned or section-specific page (e.g., `/en/v0.4.0/api/...#classname`), NOT the library root. Include a 5-10 word verbatim quote from the fetched page next to each citation. Generic references like "retrieved 2026-03" or "see official docs" do NOT count as grounding — the quote proves you actually read the page. ``` ## Plan: [Goal] ### Overview [1-2 sentences: what we're building, why this approach] ### Prerequisites - [ ] Hardware: [specific GPUs, RAM] - [ ] Dependencies: [packages with **pinned** versions — use `==`, not `>=`. For each dependency, cite the doc page confirming the API you use exists in that version. Verify: import paths, function signatures, and kwarg names change across versions.] - [ ] Install commands: `pip install` (or equivalent) with ALL packages. Include model download command if applicable (e.g., `huggingface-cli download`, `vllm serve --download-dir`). The user should be able to go from bare machine to running in one copy-paste. - [ ]