--- name: ML Autoresearch Loop slug: ml-autoresearch-loop category: AI Engineering description: ML Autoresearch Loop runs an autonomous machine learning research cycle where each experiment is analyzed before the next change is made. Use it for open-ended model improvement with one change per run, optionally grounded in literature. github: "https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/ml-autoresearch" language: Python stars: 166 forks: 19 install: "npx degit https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/ml-autoresearch ~/.claude/skills/ml-autoresearch" installs_to: ~/.claude/skills/ml-autoresearch source_path: loops/ml-autoresearch/SKILL.md collection_size: 25 category_size: 3670 collection_url: "https://dirskills.com/collections/gaasher/Agent-Loop-Skills" added: 2026-09-08T05:34:43.304Z last_synced: 2026-09-08T05:34:43.304Z canonical_url: "https://dirskills.com/skills/ml-autoresearch-loop" --- # ML Autoresearch Loop ML Autoresearch Loop runs an autonomous machine learning research cycle where each experiment is analyzed before the next change is made. Use it for open-ended model improvement with one change per run, optionally grounded in literature. **Install:** ```bash npx degit https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/ml-autoresearch ~/.claude/skills/ml-autoresearch ``` ## README # ML Autoresearch Loop This loop is **analysis-first**: every experiment is followed by a diagnostic pass that examines what happened inside the model, and the next change is a hypothesis grounded in that evidence — not a guess. The feedback signal is `` read from the run log; the analysis is the spine that decides what to change. A `` dial (`on`/`off`) optionally grounds each change in prior work via the sibling `literature-search` skill. One change per iteration, so each metric move is attributable. You are the researcher. Do not pause to ask for permission once the loop is running. ## When to use Use for an open-ended, autonomous ML research campaign where you want each change motivated by analysis of the model's actual behaviour. Set ` = off` for a self-contained analysis-and-score loop; set ` = on` to additionally ground changes in the scientific literature (paper search, evidence grading, a reusable findings backlog). Not for a single training run, a fixed sweep, or tasks with no measurable scalar metric. Default to `off` unless the user wants literature grounding or the problem is a known, well-published one where prior recipes will pay off. ## Setup **Resolve bindings interactively.** If `loop.run.yaml` exists in the working dir, load it and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is available — record `` = `claude-code`) infer a likely value for each binding from the project and present it as the recommended option; on other hosts (`` = `other`) ask each as a quoted plain-text prompt. Then write `loop.run.yaml` (format: `examples/run.example.yaml`) and **confirm every value with the user before creating any other files.** For `branches` strategy, create `git checkout -b autoresearch/` (tag from today's date; branch must not exist). For `time` gating, write `/run_with_timeout.sh` (`timeout $(( * 60 )) "$@"`) and use it as the run command, hard-killing at `2 × ` min; for `epochs`, patch the epoch cap in an `` file. | binding | meaning | default | how to infer | |---|---|---|---| | `` / `` | scalar to optimize + `minimize`/`maximize` | — | scan editable files + README for metric names | | `` / `` | command that runs one experiment end to end | — | `pyproject.toml` / `.venv` / README | | `` | files fair game to edit (never the eval harness) | — | model / config / train scripts; exclude data, logs, env, harness | | `` | where snapshots + ledgers live | `./sandbox` | next to the editable files | | `` | `snapshots` or `branches` | `snapshots` | is the working dir a clean git repo? | | `` / `` | `time` (min) or `epochs`, and the limit | — | existing time/epoch settings in config | | `` | `on` = literature-grounded; `off` = analysis-only | `off` | does the user want paper grounding? | | `` *(on only)* | depth dial `low`/`medium`/`high`/`x-high` | `medium` | see roles/research-subagent.md | | `` *(on only)* | one-phrase problem domain; seeds query phrasing only, never filters | — | infer from data/model/task | | `` *(on only)* | install dir of the `literature-search` skill | sibling of this loop | `~/.claude/skills/literature-search/` (adjust per host) | | `` *(on only)* | Python ≥3.9 interpreter for the lit helper (stdlib-only) | `python3` | independent of `` | **FILE EDIT GUARD**: before touching any file at any point — setup or loop — confirm it is in ``, because everything else is read-only ground truth (the eval harness defines ``). No exceptions. ### Literature toolchain (only when ` = on`) Paper search goes through the sibling **`literature-search` skill** (stdlib-only, no installs): ` = /tools/lit_search.py` (note the `tools/` segment). Reuse one cache by appending `--cache-dir /literature/.cache` after the subcommand. Subcommands print JSON; on failure they print `{"error","fallback"}` and exit non-zero — then **degrade to the host's WebSearch/WebFetch** (never fabricate citations). Smoke-test ` --help` at setup; if the skill is absent, tell the user and offer to install it (`cp -r /loops/literature-search ~/.claude/skills/`) or proceed degraded. For onboarding and API keys (all optional; a free `S2_API_KEY` is recommended), run ` keys --init` — it manages the shared gitignored `keys.env` at the project root and reports presence as booleans (secrets never enter chat). Persist the live tiers to `loop.run.yaml`. ## Initialise the sandbox Create the layout (extra `literature/` tree only when ` = on`) and write the ledger headers: ``` / ├── loop.run.yaml ← resolved bindings (written now) ├── results.tsv ← experiment ledger, header only (written now) ├── literature/ ← (on only) │ ├── corpus.tsv ← findings ledger, header only (written now) │ ├── .cache/ ← lit_search on-disk cache │ ├── pdfs/ ← fallback PDF reads │ └── text/ ← extracted LaTeX section text └── iter1/ ← created at loop start ``` `results.tsv` header (tab-separated; the `literature_basis` column exists only when ` = on`): ``` iter status analysis_summary [literature_basis ]description ``` ## The loop (LOOP FOREVER — until interrupted) Iteration 1 is always the **unmodified baseline**: skip change-planning and research (no diagnostics yet to ground a change), but still write a baseline `plan.md` and run the **mandatory analysis** — it produces the first empirical anchor that iteration 2 builds on. Everything in `` is fair game (architecture, optimizer, hyperparameters, data pipeline, loss); the only constraints are that the code runs without crashing and finishes within ``. **Simplicity criterion**: all else equal, simpler is better — a 0.001 gain that adds 20 lines of hacky code is not worth it; a 0.001 gain (or an equal metric) from *deleting* code is a `keep`. Copy this checklist each iteration and tick items off: - [ ] **1. Look at the state.** *branches*: `git log --oneline -5`. *snapshots*: confirm `iter/` doesn't exist. Read iter N-1's analysis summary; (on) skim `corpus.tsv` for unimplemented keepers. - [ ] **2. Plan one change** (iteration 1: SKIP — run baseline unmodified). Grounded in iter N-1's analysis. See **Planning a change** below; (on) it also runs the literature step. - [ ] **3a. Snapshot / commit, then apply the one change.** *snapshots*: create `iter/{code_snapshot,analysis,results}/`, copy every `` into `code_snapshot/`, copy `loop.run.yaml` to `iter/`, then apply. *branches*: apply, then `git commit -am ""`. When implementing a published/library technique, ground it in a real current example or the actual library in the repo (read it first) — never write the API from memory. - [ ] **3b. Write the analysis plan BEFORE the run** + add any instrumentation it needs. See **The analysis plan** below. - [ ] **4. Run the experiment**, redirecting everything (never `tee`): ` > /iter/ 2>&1` (or `run_with_timeout.sh` when time-gated). If it overruns, kill it and treat as a crash. - [ ] **5. Read the metric**: `grep '^:' /iter/`. If empty, `tail -n 50 `, read the trace, attempt one trivial fix (typo/import); if fundamentally broken, log `crash` and continue. - [ ] **6. Analyse the results** — MANDATORY, produces real artifact files. See **Analysing** below. - [ ] **7. Log to the ledger(s)** (untracked — never commit). See **Ledger**. - [ ] **8. Keep or revert** (the change ran this iteration). Improved per `` → `keep`, update current-best. Equal/worse/crash → `discard`/`crash`; *branches* `git reset --hard HEAD~1`, *snapshots* restore `` from `iter/code_snapshot/`. Apply the simplicity criterion before logging `discard`. On a crash/OOM, fix with the *minimal* change that preserves the intent (OOM → smaller batch + grad-accum to hold effective batch) — never mutate the experiment into something the plan didn't call for. - [ ] **9. Go to step 1** — the analysis from step 6 is the primary input to the next hypothesis. ### Planning a change (step 2 — iterations 2+) The latest analysis sets the direction; it is the master input every iteration. Decide exactly **one** lever, grounded in iter N-1's analysis. Other vetted ideas are queued, not bundled into one run. **State explicitly** before applying: - the one change and which `` it touches; - the **empirical anchor** — a specific file + value/pattern from iter N-1's `results/` that motivates it. Every non-simplification change must cite an anchor; theoretical reasoning alone is insufficient. A *swing* to a different architecture is anchored too (a ceiling/structural finding, e.g. "the family plateaus at X with headroom" or "it fails exactly on cases needing Y"), not a local pathology. - what you predict will happen and why the finding supports it. **When ` = off`:** that anchor is the whole basis — pick the change directly from the analysis. Before writing the plan, scan prior analyses (`ls /iter*/analysis/*.py`) so you don't repeat a diagnostic without a comparison reason. **When ` = on`:** after fixing the anchor, *ground the change in the literature* — - **2a. Retire drift, then consult the backlog as a cache.** If the last kept change altered the architecture *family* (e.g. CNN→transformer), set `result=stale` for every unimplemented keeper whose `scope` is a non-matching architecture tag; `scope=agnostic` keepers (schedules, weight decay, augmentation, init *philosophies*) survive. Then check `corpus.tsv` for an unimplemented `keep` targeting the analysis's direction — reuse it **only if it still passes the gate (2c) against the CURRENT architecture** (re-validate now; a finding that no longer applies is retired, not forced in). - **2b. Research the direction** (the default unless 2a yielded a still-valid lever). Turn the analysis's limitations into questions (tie limitations to questions), record them in `iter/questions.md`, then dispatch **research subagents** — see `roles/research-subagent.md` (spawn-or-degrade: real isolated subagents on Claude Code, otherwise run the research inline in this context) at the dial's depth/effort (see that file's dial table for ``). Level 1 = high-level (architecture fit, prior approaches, does the literature show success); research L1 first — if it surfaces a compelling new direction that becomes the lever. Level 2 = specific micro-opts (init, weight-decay dynamics, attention/cache for the sequence length, norm placement, schedule). *Anti-rut: a keeper passed over for ~3 iterations, or no longer on any live direction, is retired (`result=stale`) so it stops resurfacing.* - **2c. Evidence gate** (re-validated; consults history). Keep/reuse a finding only if **all** hold: `evidence.strength` is `moderate`/`strong` (not `weak`); `applicability_to_our_setup` transfers to the CURRENT data/model/budget; every `files_to_edit` is within `` and fits ``; and it is **not materially equivalent to a change already logged `no-effect`/`hurt`/`crashed`** in `corpus.tsv` (unless the finding explains why it differs now). Tag each kept finding's **`scope`** (`agnostic` or the current architecture tag) so drift retirement is mechanical. Record every finding in `corpus.tsv` (keep or reject); kept-but-unchosen findings stay queued (`implemented=n`, `result=pending`). - **2d. Choose the single lever** from the keepers (backlog + new) — highest evidence × applicability × gain-per-complexity. Add to the explicit statement above: the **literature basis** (kept finding + citation(s)), or `none` for a pure analysis-driven simplification/ablation; and the **verify-todo** (the metric/log/analysis from the finding's `how_to_verify` that will tell whether it helped). The literature basis is *additive* grounding — it never replaces the empirical anchor. ### The analysis plan (step 3b — written BEFORE the run) Write `/iter/analysis/plan.md` as a deliverables **table**, not prose — each row is a hard commitment: a script you will write and run, the exact output file it produces, the question it answers, and what each outcome would look like: | script | output file | question it answers | expected if change helped | expected if not | |--------|-------------|---------------------|---------------------------|-----------------| The plan **must** include at least one diagnostic dimension **not measured in any prior iteration** (the history scan enforces this); the "expected if not" column is mandatory (it forces you to define falsification before seeing the result). When ` = on`, also include the **verify-todo** from 2d, and periodically (and whenever a swing is on the table) a **ceiling/headroom probe** — is the current architecture near its limit? (train/val plateau, representation collapse, structural failure). **Instrumentation (mandatory when needed).** If a planned measurement needs a metric/log not currently produced (e.g. per-layer grad norms, per-class accuracy, `val_loss` alongside `val_acc`) and the producing script is in ``, **add that logging NOW, before the run.** Writing the plan first is what forces this — don't discover at analysis time that the number you need wasn't logged. ### Analysing (step 6 — MANDATORY; produces real artifacts) This is the spine of the loop. **Do not proceed to step 7 until every row in `plan.md` has a corresponding non-empty file in `iter/results/`.** Analysis that didn't write a file did not happen. - **6a. Execute the plan.** For each row, write the script in `iter/analysis/` and run it, redirecting to `iter/results/`. Diagnostic dimensions to draw from (choose what the hypothesis and results call for, not a fixed rotation): gradient norms/flow, activation stats/saturation, embeddings (PCA/CKA/collapse), error & confusion analysis, loss dynamics & **headroom**, weight/parameter stats, data profiling (auditing the data itself is one of the highest-yield diagnostics), compute profiling. - **6b. Checklist.** `ls iter/results/` against `plan.md`; write and run any missing deliverable now. - **6c. Interpret against the plan.** For each row, did it match "expected if helped" or "if not"? A mismatch is information, not a failure. - **6d.** (on) **Check the verify-todo** from 2d — did the change do what the literature predicted? This sets the finding's `result` in step 7. - **6e. Opportunistic follow-up.** If something unexpected appears (odd loss shape, a consistently-wrong class, a layer behaving differently), write one more script to chase it — the most novel findings come from here. Write a concise **analysis summary** (3–8 bullets): what you examined, the single most important finding (expected or not), (on) whether the architecture is near its ceiling, and the specific **empirical anchor** (file + value/pattern) that will motivate the next plan. **Empirical-justification gate.** Before step 7, you must be able to complete this sentence with a real file reference and a concrete value/pattern — not a theoretical argument: > *"The next change will be X because the analysis showed Y (from `results/`, value/pattern Z)."* If you cannot, go back to 6a and run more analysis until you can. Proceeding without an empirical anchor is the exact failure this loop is built to prevent. **Forward-looking instrumentation.** After analysing, ask *"what would I wish I had logged?"* — if the producing script is in ``, add it now (best-epoch checkpoint, extra per-epoch diagnostics) so future analyses are richer. Adding instrumentation is a valid iteration on its own. ## Ledger `/results.tsv`, tab-separated, never commas in free text. The `literature_basis` column is present only when ` = on`. Status ∈ {`keep`, `discard`, `crash`}; use `0.000000` for the metric on crashes. For `branches`, the first column is the 7-char commit instead of the iter number. ``` iter status analysis_summary literature_basis description 1 0.6320 keep grad norms clean; no pathologies none (baseline) baseline 2 0.7050 keep overfit on small data per val/train gap Dosovitskiy 2020 ViT — aug+warmup add RandAug + 3-ep warmup 3 0.7010 discard warmup ok but LR too high; unstable Loshchilov 2017 — decoupled wd switch to AdamW wd=0.05 ``` **`/literature/corpus.tsv`** (only when ` = on`; tab-separated): a backlog of vetted findings reused across iterations instead of re-researched. The orchestrator (not the subagent) writes it and applies the final evidence gate, so verdicts are consistent across questions. ``` iter level paper_id title scope relevance verdict implemented result 2 649def… 1 An Image is Worth 16x16 Words vit high keep y helped 3 a1c2… 2 Decoupled Weight Decay Regularization agnostic high keep y no-effect 4 7d9e… 1 ConvNeXt blocks cnn low keep n stale ``` Columns: `level` ∈ {1,2}; `scope` = `agnostic` or an architecture tag (drives drift retirement); `relevance` ∈ {high,med,low}; `verdict` ∈ {keep,reject}; `implemented` ∈ {y,n}; `result` ∈ {helped,no-effect,hurt,pending,stale} (`stale` = retired on drift/expiry). Last row: a CNN keeper retired after the architecture moved to ViT; the `agnostic` weight-decay finding survives drift. Report the **best** iteration (not necessarily the last) when summarising progress. Do not commit `results.tsv` or `literature/` to git — leave them untracked. ## Constraints - **Only edit files in ``** — confirm before every edit, because everything else is read-only ground truth and the eval harness defines ``. - **One change per iteration**, so each metric delta is attributable to one lever. - **Analysis is mandatory every iteration** and must produce real files in `results/` (each `plan.md` row → a non-empty output file). A change with no analysis behind it is not allowed — it degrades the loop into blind iteration (or, when on, recipe-following that plateaus blind). - **Every non-simplification change must cite an empirical anchor** (file + value/pattern) from that analysis; when on, the literature basis is additive, never a substitute. Record both in `results.tsv`. - Do not install packages or add dependencies the project lacks; helper scripts (and the lit helper) stay stdlib-only. - Always redirect training output to ``; never use `tee`. The sandbox is self-contained — no `../` escapes. - (on) Never print or commit API keys — `keys` reports presence only; `keys.env` lives at the project root and must stay gitignored. When a lit tool returns `{"error","fallback"}`, degrade to WebSearch/WebFetch; never fabricate citations — every cited paper must come from a real tool result. ## Stops This loop runs **forever until the human interrupts it** — do not pause to ask "should I continue?" or "is this a good stopping point?". The human may be away and expects autonomous work indefinitely. A working result is the start of the next iteration, not the end. If you run out of ideas: re-read the in-scope files for missed angles, go deeper on the analysis (gradients/activations/embeddings/errors always surface something), combine previous near-misses from `results.tsv`, try more radical changes — and when on, **go back to the literature**: crawl citation graphs deeper, read methodology of recent work cit