--- name: ML Research slug: ml-research category: AI Engineering description: ML Research helps compare ML and AI approaches using verified framework documentation and model references. Use it when you need to understand how a method works or survey framework capabilities. github: "https://github.com/Leeroo-AI/superml/tree/main/skills/ml-research" language: Python stars: 194 forks: 18 install: "npx degit https://github.com/Leeroo-AI/superml/tree/main/skills/ml-research ~/.claude/skills/ml-research" installs_to: ~/.claude/skills/ml-research source_path: skills/ml-research/SKILL.md collection_size: 7 category_size: 3101 collection_url: "https://dirskills.com/collections/Leeroo-AI/superml" added: 2026-09-05T05:31:32.151Z last_synced: 2026-09-05T05:31:32.151Z canonical_url: "https://dirskills.com/skills/ml-research" --- # ML Research ML Research helps compare ML and AI approaches using verified framework documentation and model references. Use it when you need to understand how a method works or survey framework capabilities. **Install:** ```bash npx degit https://github.com/Leeroo-AI/superml/tree/main/skills/ml-research ~/.claude/skills/ml-research ``` ## README # ML Research Deep-dive into ML topics using verified framework documentation, not stale training data. ## Grounding **Detect mode:** On your first grounding call, check if Leeroopedia KB tools are available. If they return results, use **KB mode**. If unavailable or auth fails, use **Web mode**. **CRITICAL SELF-CHECK before Phase 1**: If KB auth fails, your FIRST action must be WebFetch — not text output. Do NOT write any prose, explanation, or "I know this topic" before completing at least 3 WebFetch calls. The phrases "I know this topic well" and "I have deep knowledge of these" appeared in real failures and are disqualifying errors. **MANDATORY SEQUENCE when KB fails**: (1) Make 3+ WebFetch calls. (2) Confirm you have content from those calls. (3) ONLY THEN write your first line of prose. If your response starts with ANY text before tool calls, you have already failed. There is no "but I know this one" exception. **KB mode:** Call `search_knowledge` (2-4 parallel queries) → `get_page` on top results → synthesize. Cite as `[PageID]`. **Web mode:** WebFetch official docs (3-5 pages) → WebFetch GitHub README/examples → WebFetch model cards for each model mentioned → synthesize. Cite as `[source](URL)`. Start response with: `> Grounding: Web mode — citations from official docs.` **Web mode is NOT optional.** If KB auth fails, you MUST immediately switch to web mode and fetch real documentation. Do not proceed without at least 3 WebFetch calls returning real content. **ZERO-TOLERANCE RULE**: If your first instinct is "I know this topic well enough" — that is the exact moment you MUST ground. The following phrases (and synonyms) are BANNED: "well within my training knowledge", "I can answer this from memory", "this is a well-known topic", "I have deep knowledge", "I have extensive knowledge", "I know these methods well", "I'm familiar with". They are the #1 failure mode of this skill. Every response requires fetched sources, no exceptions. If you catch yourself composing ANY sentence that explains why grounding is unnecessary, STOP and make a WebFetch call instead. **Web mode URL registry:** - HF Transformers/PEFT/TRL: `https://huggingface.co/docs/{transformers,peft,trl}` - vLLM: `https://docs.vllm.ai` - DeepSpeed: `https://www.deepspeed.ai/docs` - LangChain: `https://python.langchain.com/docs` - LangGraph: `https://langchain-ai.github.io/langgraph` - RAGAS: `https://docs.ragas.io` - PyTorch: `https://pytorch.org/docs/stable` - vLLM (speculative decoding, serving): `https://docs.vllm.ai/en/latest/features/spec_decode.html` - HF TGI: `https://huggingface.co/docs/text-generation-inference` - mergekit: `https://github.com/arcee-ai/mergekit` (README + `mergekit/_data/`) - FAISS: `https://github.com/facebookresearch/faiss/wiki` - sentence-transformers: `https://sbert.net/docs` ## The Iron Law ``` NO CLAIMS WITHOUT KB VERIFICATION NO RESPONSE WITHOUT GROUNDING — KB OR WEB ``` Your training data is months old. The KB has current framework docs. When the two disagree, the KB wins. **Hard stop**: If KB auth fails, switch to web mode IMMEDIATELY — make WebFetch calls as your very next action. Do not tell the user grounding is unavailable unless WebFetch also fails. Do not tell the user you "have deep knowledge" or can answer anyway. Only if BOTH KB and WebFetch fail should you tell the user and refuse to answer. Never answer from training data alone. **Enforcement sequence**: KB fails → WebFetch 3+ doc pages → synthesize. There is no path from "KB fails" to "answer from memory". ## Phases ### Phase 1: Multi-Angle Search **KB mode:** Launch 2-4 `search_knowledge` calls in **parallel** with different angles: > **KB mode citation enrichment (MANDATORY)**: KB pages are secondary sources and are NOT publicly verifiable on their own. After retrieving KB results, you MUST identify the primary source each page references (paper, official docs URL, GitHub repo). For each claim you cite with `[PageID]`, also WebFetch the primary source URL and confirm the claim matches. Format: `[PageID] ([primary source](URL))`. Target: at least 50% of your KB citations must include a verified primary URL. If a KB page has no identifiable primary source, do a targeted WebFetch to find the canonical documentation for that claim. - Core concept / mechanism - Framework-specific implementation - Configuration and usage patterns - Performance characteristics or tradeoffs **Web mode:** WebFetch 3-5 official documentation pages covering different angles: - The framework's main docs page for the feature - GitHub README or examples directory - Any dedicated guide or tutorial page - Comparison or migration guide (if applicable) - **Model cards or API references** for each specific model/tool being discussed — one WebFetch per model minimum **Web mode URL construction**: Use the URL registry above. Build specific URLs by appending path segments (e.g., `https://docs.vllm.ai/en/latest/features/spec_decode.html`). If a URL 404s, try the parent path or search the docs index page. Do NOT invent URLs — only fetch URLs you constructed from the registry or found linked in a fetched page. **Gate**: You have sources covering at least 2 distinct angles on the topic before synthesizing. **For comparison tasks**: you MUST fetch a source for EACH item being compared (each model, each method, each library). A single overview page is insufficient — individual model cards, method docs, or API references are required. **Hard gate**: If you have ZERO fetched sources (no KB results AND no WebFetch results), you MUST NOT proceed to Phase 2. Go back and fetch sources. Count your citations — if the count is 0, you have not done Phase 1. **Self-audit**: Before moving to Phase 2, count your tool calls. In KB mode: at least 2 `search_knowledge` calls completed. In Web mode: at least 3 `WebFetch` calls completed with real content returned. If you have not met these minimums, STOP and make more calls. Do not proceed based on what you "already know". **Version extraction (MANDATORY)**: As you read each fetched page, extract the **exact library version** mentioned (e.g., `transformers==4.41.0`, `vllm==0.4.1`). Record these for citations. If a page doesn't state its version, note the URL path segment (e.g., `/v0.4.1/` or `/stable/`). Every citation in your final output must include a version or retrieval date. **Web mode grounding for comparison tasks**: When comparing N items (models, methods, tools), you need at least N WebFetch calls — one per item being compared. A single overview page cannot ground all items. For model merging: fetch mergekit docs + one source per merge method. For model comparisons: fetch each model card. **Pipeline/system design grounding**: When the user asks to build a complete system (e.g., "build a code search engine", "design an embedding pipeline"), fetch: (1) model card for the recommended model, (2) the serving/inference framework docs, (3) the vector DB/index docs, (4) at least one benchmark page with real throughput/latency numbers for the model+hardware combo. Pipeline questions require MORE grounding than concept questions, not less. **Benchmark fetching for comparison tasks**: When comparing methods, also WebFetch at least one benchmark page (blog post, paper, or leaderboard) that has **measured numbers** (perplexity, throughput, memory). Without benchmark data, comparison tables devolve into vague qualitative claims. Search for: `[method] benchmark results [model size]`. > **Citation density target**: Aim for 10+ unique citations in the final answer. KB mode: `[PageID]` citations, call `get_page` on at least 3 top results. Web mode: `[source](URL)` citations from distinct doc pages. **If you have fewer than 5 unique citations after Phase 2, STOP and fetch more sources before synthesizing.** ### Phase 2: Expand and Resolve **KB mode:** 1. Call `get_page` on the most relevant `[PageID]` citations — prioritize pages with code examples, edge cases, or quantitative comparisons 2. If sources disagree, note which is newer or more framework-specific 3. If there's a gap, run one more targeted `search_knowledge` **Web mode:** 1. For the most relevant pages, WebFetch specific sub-pages (API reference, config reference, changelog) 2. If sources disagree, check the changelog or release notes for the latest info 3. If there's a gap, WebFetch GitHub issues or discussions for the topic **Gate**: Every factual claim, config value, and table cell has a citation. CLI flags, parameter names, and config keys must appear verbatim in a documentation source — do not invent flags or options from memory. **Hedging gate**: Absolute claims ("does NOT work", "always", "never", "negligible") require version-specific qualification. Replace "X does NOT work on Y" with "X is not natively supported on Y as of vZ.W — workarounds exist via [method]". Replace "negligible degradation" with a cited delta (e.g., "+0.15 PPL"). If you cannot cite the exact delta, write "minimal degradation per [source]" not "negligible". **Parameter existence gate**: Before including ANY config parameter, CLI flag, or API argument in your response, confirm it exists verbatim in a fetched source. Common hallucination pattern: inventing plausible-sounding parameters like `int_space`, `int8_mask`, or `normalize: true` that don't exist in the library. If a parameter appeared in a KB page but you cannot find it in the library's actual docs or README, mark it `⚠️ KB-ONLY — verify in official docs` rather than presenting it as fact. **Numeric claims gate**: Any quantitative claim (perplexity scores, memory usage, throughput numbers, speedup ratios) MUST have a citation to a benchmark or doc page. Do not estimate numeric values from training data. If you cannot find a source, **compute it from known specs** (e.g., memory = params × bytes_per_dtype + overhead) and show the math. Only write `[needs benchmark]` for metrics that cannot be computed (e.g., perplexity, downstream task accuracy). You have a budget of 3 `[needs benchmark]` per response. > **Speedup and performance tables**: Tables with columns like "Expected Speedup" or "Throughput" are HIGH-HALLUCINATION zones. Every numeric cell in such tables must either (1) cite a specific benchmark paper/blog with the number, or (2) be marked `[needs benchmark]`. Do not present ranges like "2.0–2.5x" without a cited source — these look precise but are invented. > **Time and cost estimates**: Quantization time, training time, and indexing time estimates are frequently optimistic. Apply a **2× safety multiplier** to any time estimate not directly cited from a benchmark. For example, if you'd estimate "~1 hour" from general knowledge, write "~1-2 hours (varies by hardware; benchmark your setup)". Never give a single optimistic point estimate for time-intensive operations. > **Identifier verification (GATE — do not proceed past Phase 2 without this)**: Before including any model name, API method signature, CLI flag, or config field name, confirm the **exact string** appears in a KB page or fetched doc. If you cannot find verbatim confirmation, call `search_knowledge` or WebFetch the relevant API reference. If still unconfirmed, flag it as `⚠️ UNVERIFIED — check docs` in bold rather than presenting it as fact. **Never invent parameter names like `int8_mask` or assume a model works with a wrapper (e.g., SentenceTransformer) without confirming compatibility in the docs.** > **Model spec verification (GATE)**: Context window sizes, embedding dimensions, supported languages, and max token limits are HIGH-HALLUCINATION fields. You MUST confirm these numbers from a fetched source — do not rely on training data for model specs. If a model card or doc page was fetched, extract the exact number. If not fetched, WebFetch the model card before including the spec. A wrong context window (e.g., listing 1024 when it's 8192, or 32K when it's 16K) invalidates the entire comparison table. > **Model behavior verification (GATE)**: Pooling strategy (CLS vs mean vs last-token), tokenizer max length, and input format (instruction-prefixed vs raw) are HIGH-HALLUCINATION fields alongside specs. Before recommending a model, WebFetch or search its model card for: (1) exact pooling method, (2) actual max sequence length the tokenizer supports, (3) required input format. If your code uses CLS pooling but the model card says mean pooling, your pipeline is silently wrong. Set chunk sizes to match the model's ACTUAL max tokens, not a round number you assumed. > **Config schema verification (GATE)**: Before showing any tool/library config file (mergekit YAML, DeepSpeed JSON, vLLM args), WebFetch or search_knowledge for the config schema or a working example. Verify: (1) top-level keys vs nested keys — don't put a top-level field inside a list, (2) required vs optional fields, (3) exact field names — no invented fields. Example failure: mergekit's `base_model` is a top-level key, NOT an entry in the `models:` array. > **mergekit-specific config trap**: For TIES/DARE configs, the `models:` list must contain ONLY the fine-tuned models — do NOT include the base model as an entry in the `models:` array. The base model is specified ONLY via the top-level `base_model:` key. Including it in both places double-counts it and corrupts the merge. Always fetch a working mergekit example config before writing one. > **Config value-type verification**: Boolean vs string vs numeric fields are a common hallucination. Example: mergekit's `int_space` expects a string value (e.g., `"tri"`), NOT a boolean (`true`). Always confirm the expected type from a fetched config example — do not guess whether a field takes `true` or a string enum. ### Phase 3: Synthesize Compose a structured answer: **Completeness rules:** - Code must be **runnable as-is**: include imports, CLI arg parsing, and shebang lines — never reference a hypothetical `train.py` without providing it - **System design / pipeline questions STILL require code**: If the user asks to "design" or "build" a pipeline, your response must include a working Python script (with imports, main function, and CLI entry point) — not just a diagram or prose description. Architecture diagrams are supplementary, never primary. A response with boxes but no code scores 0 on actionability. - Pin versions: `pip install mergekit==0.3.1`, not `pip install mergekit` - Include **deployment scaffolding** when the use case implies production: Dockerfile, docker-compose.yml, or systemd unit — not just the Python code - When recommending a library/model, verify the **integration path** works (e.g., does model X actually load with library Y's API? Check the docs, don't assume) - When showing **comparison tables**, every cell with a number (context window, dimensions, cost, throughput) must have a citation. If a cell has no citation, the number is suspect — fetch a source or mark `⚠️ UNVERIFIED` - If showing a multi-file pipeline, include an orchestration script that ties files together - Every table cell containing a factual claim needs a `[PageID]` - Every **parameter name, CLI flag, and config key** in tables must be copy-paste-correct from docs — one wrong flag wastes hours - Web mode citations must include the **doc version or date**: `[source](URL)` is insufficient — use `[source: vLLM v0.4.1 docs](URL)` or `[source: retrieved 2026-03](URL)` so the user knows the freshness of the info - **KB mode citations must include primary source URLs**: `[PageID]` alone is not publicly verifiable. After every `[PageID]` citation, add the primary source the KB page references — the paper URL, official docs URL, or GitHub link. Format: `[PageID] ([primary docs](URL))`. If the KB page doesn't reference a primary source, WebFetch the official docs for that claim and add the URL yourself. Target: at least 50% of KB citations include a publicly verifiable URL. - **Citation authenticity**: Only cite URLs you actually fetched via WebFetch in this session. Do not reconstruct citations from training data memory — if you didn't WebFetch it, you cannot cite it. Fabricated citations (URLs that look plausible but weren't fetched) are worse than no citations. - **Numeric claim authenticity**: Benchmark scores (NDCG, perplexity, BLEU, accuracy), throughput numbers, and latency claims follow the same rule — only include numbers you found in a fetched source. If you cannot find the number in a fetched page, either compute it from known specs (show the math) or mark `[needs benchmark]`. Presenting training-data numbers as if they were sourced is a grounding failure. - Config files must be **complete** — do not omit fields. If a config has 8 required fields, show all 8 with KB-sourced values - Config files must be **structurally correct** — fetch a working example from docs/GitHub before writing configs. Verify top-level vs nested keys, list vs scalar values, and required field placement - Include a **validation/test snippet** (curl test, benchmark script, or smoke test) so the user can verify their setup works - Use **exact HuggingFace model IDs** (e.g., `BAAI/bge-code-v1`, not just "BGE-Coder") and **exact pip package versions** in all code — users should be able to copy-paste without searching for the right identifier - Include a **"Verify it works" section** with a 3-5 line smoke test that confirms the setup end-to-end, not just that imports succeed - **Pipeline coherence check**: If your response includes multiple code blocks that form a pipeline, verify they are internally consistent: (1) output format of step N matches expected input of step N+1, (2) model names/paths are identical across all blocks, (3) if a server is started in one block, subsequent blocks connect to it (not reload the model locally), (4) chunk sizes, dimensions, and dtypes are consistent across preprocessing, embedding, and indexing steps **Final correctness check (GATE — do not output response without this)**: 1. Re-read every parameter name, CLI flag, and config key in your response. Is each one copied verbatim from a fetched source? If not, mark `⚠️ UNVERIFIED`. 2. Re-read every number in tables. Does each have a `[PageID]` or `[source](URL)` citation? If not, add `[needs benchmark]`. 3. Count unique citations. If fewer than 5, you have not grounded sufficiently — go back to Phase 1. In KB mode, verify at least 2 citations include a primary source URL (paper, official docs, or GitHub link) — not just PageIDs. 4. **Self-consistency check**: Re-read your response for internal contradictions. If you say X has "limited support" for Y, do not then show working code for X+Y without explanation. If a claim in one section contradicts another section, resolve it before outputting. 5. **Command consistency check**: Re-read every CLI command and code block. If you start a server (e.g., vLLM), verify subsequent commands actually USE that server (e.g., via API endpoint, not local model path). If you reference a benchmark task name (e.g., `humaneval`), verify it matches the exact task registry name for the tool (e.g., `openai_humaneval` for lm-evaluation-harness). If code block A defines a variable/config and code block B uses it differently, fix the inconsistency. 6. **[needs benchmark] budget**: You may use `[needs benchmark]` for at most 3 cells in the entire response. If you find yourself writing it more often, go back and WebFetch benchmark pages, blog posts, or papers with real numbers. Overuse of `[needs benchmark]` signals insufficient research, not honesty. ``` ## [Topic] ### How It Works [Core mechanism — what it does and why] [PageID] ### Implementation [Framework-specific details with code examples] [PageID] ```[language] [runnable code snippet] ``` ### Key Configuration | Parameter | Recommended Value | Why This Value | Source | |-----------|-------------------|----------------|--------| | param | exact_value (not a range) | [rationale with numbers] | [PageID] | > Fill every row with a **singl