--- name: Exploratory Autoresearch slug: exploratory-autoresearch category: AI Engineering description: Exploratory Autoresearch runs an autonomous ML research loop that alternates analysis with scheduled swings, merges, and exploits. It is used for open-ended experiments where you want broad exploration first and a stagnation guard to prevent getting stuck in small-step tuning. github: "https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/exploratory-autoresearch" language: Python stars: 166 forks: 19 install: "npx degit https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/exploratory-autoresearch ~/.claude/skills/exploratory-autoresearch" installs_to: ~/.claude/skills/exploratory-autoresearch source_path: loops/exploratory-autoresearch/SKILL.md collection_size: 25 category_size: 3670 collection_url: "https://dirskills.com/collections/gaasher/Agent-Loop-Skills" added: 2026-09-08T05:34:41.575Z last_synced: 2026-09-08T05:34:41.575Z canonical_url: "https://dirskills.com/skills/exploratory-autoresearch" --- # Exploratory Autoresearch Exploratory Autoresearch runs an autonomous ML research loop that alternates analysis with scheduled swings, merges, and exploits. It is used for open-ended experiments where you want broad exploration first and a stagnation guard to prevent getting stuck in small-step tuning. **Install:** ```bash npx degit https://github.com/gaasher/Agent-Loop-Skills/tree/main/loops/exploratory-autoresearch ~/.claude/skills/exploratory-autoresearch ``` ## README # Exploratory Autoresearch Loop This loop runs hot. Like the standard `ml-autoresearch`, every experiment is followed by a diagnostic analysis pass. **Unlike** it, the *type* of change at each iteration is set by a **temperature scheduler**, not the agent's intuition: it forces wide, diverse swings early (full rewrites, fundamentally different architectures and training regimes), then drops into an adaptive phase that chooses between **swing** (a fresh wild approach), **merge** (combine two registered approaches), or **exploit** (a focused tweak of the best). A **stagnation guard** bans exploit once it has run `` times in a row, forcing a pivot back to swing or merge so the loop never gets stuck hill-climbing. The feedback signal is `` read from the run log; an `approaches.md` registry and a `move_type` per iteration are what make the scheduler work. You are the researcher. Do not pause to ask for permission once the loop is running. ## When to use Use for an open-ended ML campaign where you want forced breadth before refinement — the scheduler guarantees you sample several distinct families before converging, and the stagnation guard prevents endless small steps. Default to ` = 3` and ` = 3`; raise `` for wider initial exploration. Not for the standard analysis-first `ml-autoresearch` (use that when you want the analysis alone to drive each change, with no forced-swing scheduler), not for a single training run or a fixed sweep, and not for tasks with no measurable scalar metric. ## Setup **Resolve bindings interactively.** If `loop.run.yaml` exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is available — record `` = `claude-code`) infer a likely value for each binding from the project and present it as the recommended option; on other hosts (`` = `other`) ask each as a quoted plain-text prompt. Then write `loop.run.yaml` (format: `examples/run.example.yaml`) and **confirm every value with the user before creating any other files.** For `branches` strategy, create `git checkout -b autoresearch/` (tag from today's date; branch must not exist). For `time` gating, write `/run_with_timeout.sh` (`timeout $(( * 60 )) "$@"`) and use it as the run command, hard-killing at `2 × ` min; for `epochs`, patch the epoch cap in an `` file. | binding | meaning | default | how to infer | |---|---|---|---| | `` / `` | scalar to optimize + `minimize`/`maximize` | — | scan editable files + README for metric names | | `` / `` | command that runs one experiment end to end | — | `pyproject.toml` / `.venv` / README | | `` | files fair game to edit (never the eval harness) | — | model / config / train scripts; exclude data, logs, env, harness | | `` | where snapshots + ledgers live | `./sandbox` | next to the editable files | | `` | `snapshots` or `branches` | `snapshots` | is the working dir a clean git repo? | | `` / `` | `time` (min) or `epochs`, and the limit | — | existing time/epoch settings in config | | `` | forced wild swings before adaptive mode | 3 (3–5) | wider = more initial breadth | | `` | max consecutive exploits before a forced pivot | 3 | — | **FILE EDIT GUARD**: before touching any file at any point — setup or loop — confirm it is in ``, because everything else is read-only ground truth (the eval harness defines ``). No exceptions. ### Initialise the sandbox Create the layout and write the ledger headers: ``` / ├── loop.run.yaml ← resolved bindings (written now) ├── results.tsv ← experiment ledger, header only (written now) ├── approaches.md ← registry of every distinct approach (header only, written now) └── iter1/ ← created at loop start ``` `results.tsv` header (tab-separated; `move_type` ∈ {`swing`, `merge`, `exploit`}): ``` iter status move_type analysis_summary description ``` `approaches.md` header: `# Approach Registry` plus a one-line note that the merge step consults it to find complementary approaches to combine. ## The loop (LOOP FOREVER — until interrupted) Iteration 1 is always the **unmodified baseline** (it does not count as a swing): skip move-selection and change-planning, but still run the **mandatory analysis** — it is the first empirical anchor iteration 2 builds on. Everything in `` is fair game (architecture, optimizer, hyperparameters, data pipeline, loss, init, eval); on swings especially, full rewrites are encouraged. The only constraints are that the code runs and finishes within ``. **Epoch efficiency is part of the objective** — a change that reaches the same score in fewer effective steps is a real win. **Simplicity criterion**: all else equal, simpler is better — a 0.001 gain that adds 20 lines of hacky code is not worth it; a 0.001 gain (or an equal metric) from *deleting* code is a `keep`. The scheduler keeps two counters in memory across iterations: **`swings_taken`** (total swing iterations, excludes the baseline) and **`consecutive_exploit`** (exploits since the last swing/merge; resets to 0 on any swing or merge). Copy this checklist each iteration and tick items off: - [ ] **1. Look at the state.** *branches*: `git log --oneline -5`. *snapshots*: confirm `iter/` doesn't exist. Read iter N-1's analysis summary and the two counters. - [ ] **2. Pick `move_type`** (iteration 1: SKIP — baseline). Apply the scheduler below, then record the move before touching any file. - [ ] **3. Form the hypothesis** (iteration 1: SKIP). State the move and why (cite the rule or the analysis), what you will do, and which `` it touches. See **The three moves**. - [ ] **4. Snapshot / commit, then apply the change.** *snapshots*: create `iter/{code_snapshot,analysis,results}/`, copy every `` into `code_snapshot/`, copy `loop.run.yaml` to `iter/`, then apply. *branches*: apply, then `git commit -am ": "`. - [ ] **5. Run the experiment**, redirecting everything (never `tee`): ` > /iter/ 2>&1` (or `run_with_timeout.sh` when time-gated). If it overruns, kill it and treat as a crash. - [ ] **6. Read the metric**: `grep '^:' /iter/`. If empty, `tail -n 50 `, read the trace, attempt one trivial fix (typo/import); if fundamentally broken, log `crash` and continue. - [ ] **7. Analyse the results** — MANDATORY, produces real artifact files. See **Analysing**. - [ ] **8. Update `approaches.md`** (swing and merge moves only). See **The registry**. - [ ] **9. Log to `results.tsv`** (untracked — never commit). See **Ledger**. - [ ] **10. Keep or revert** (the change ran this iteration). Improved per `` → `keep`, update current-best. Equal/worse/crash → `discard`/`crash`; *branches* `git reset --hard HEAD~1`, *snapshots* restore `` from `iter/code_snapshot/`. Apply the simplicity criterion before logging `discard`. On a crash/OOM, fix with the *minimal* change that preserves the intent (OOM → smaller batch + grad-accum to hold effective batch) — never mutate the experiment. - [ ] **11. Update counters** (below) and go to step 1. ### The scheduler (step 2 — this is the loop's identity) Follow the rules **exactly, in order** — they are hard constraints, not suggestions: ``` IF iter == 1 → baseline (run unmodified; no move) ELIF swings_taken < → swing (forced exploration) ELIF consecutive_exploit >= → swing OR merge (forced pivot — exploit BANNED) ELSE → agent chooses: swing / merge / exploit ``` On the free `ELSE` branch, let iter N-1's analysis decide: - **swing** if the current family has a fundamental ceiling — e.g. all top results share a failure mode. - **merge** if two+ `approaches.md` entries have distinct, non-overlapping strengths (prefer parents that changed *different axes* — they combine additively rather than interfere). - **exploit** if the current best has obvious analysis-suggested headroom not needing a new architecture. ### Counter update (step 11) ``` if move_type in {swing, merge}: swings_taken += 1 (swing only); consecutive_exploit = 0 elif move_type == exploit: consecutive_exploit += 1 ``` ### The three moves (step 3) - **Swing** — *fundamentally* different from every previous swing (not a tweak; the diff should look obviously different from the current best). Most people swing on architecture by reflex — fight that. These axes are equally valid and underexplored: **architectural family** (how information flows, depth vs width, skip connections, local vs global); **initialization** (magnitude-based, structure-preserving, input-statistics-driven, sparse — different early dynamics); **data pipeline** (ordering, sampling, determinism, coverage of the view space — not just augmentation flavours); **per-component LR decoupling** (early/late layers, norms, biases, heads each have their own optimal step); **evaluation** (single pass, multi-view, checkpoint averaging, calibration); **objective** (loss shape, target sharpness, auxiliary/consistency signals). A genuine swing explores one of these in a way not yet tried. - **Merge** — select two+ entries from `approaches.md` and name what is taken from each; the result is a new approach that is not a minor variant of either parent. Prefer components from *different axes*. - **Exploit** — a targeted, focused change to the current best, grounded in a specific analysis finding. One or two things at a time; **decouple the axes** (test a new optimizer and a new LR as separate iterations so you know which caused the result). Never a different architecture. ### Analysing (step 7 — MANDATORY; produces real artifacts) This is the spine that feeds the next move. Run whatever analysis most increases your understanding of *why* this result happened. Every analysis script goes in `iter/analysis/`; every output (plots, CSVs, text) goes in `iter/results/`, redirecting stdout there. Do not proceed until the results exist — analysis that wrote no file did not happen. Dimensions to draw from (choose what fits): gradient norms/flow, activation stats/saturation, embeddings (PCA/CKA/collapse), error & confusion analysis, loss dynamics & **headroom** (was it still improving at cutoff?), weight/parameter stats, data profiling (often the highest-yield), compute profiling. Write a concise **analysis summary** (3–8 bullets): what you examined, the single most important finding, and what it implies for the next move (whether it favours swing / merge / exploit). **Ablation discipline** (after any `keep` that touched more than one axis): you don't yet know which part caused the gain — flag it and consider an ablation exploit next, reverting one component at a time. Building on an unablated multi-axis change is building on an unknown. **Forward-looking instrumentation.** After analysing, ask *"what would I wish I had logged?"* — if the producing script is in ``, add it now (best-epoch checkpoint, per-layer grad norms, per-class accuracy). Richer logs improve every future analysis; adding instrumentation is a valid iteration on its own (log it as `move_type=exploit`). ### The registry (step 8 — `approaches.md`, swing and merge only) Append an entry; the `axes changed` field is what the merge step uses to find complementary parents (two that changed *different* axes beat two that both changed the architecture): ```markdown ## Approach : - **move_type**: swing | merge - **iter**: - ****: (status: keep | discard | crash) - **axes changed**: architecture | initialization | data pipeline | optimizer | lr schedule | evaluation | objective | other - **key ideas**: - **strengths** / **weaknesses** (from analysis): - **ablated?**: yes (which components isolated, what was found) / no - **parents** (merge only): Approach X + Approach Y — axes taken from each ``` ## Ledger `/results.tsv`, tab-separated, never commas in free text. Status ∈ {`keep`, `discard`, `crash`}; use `0.000000` for the metric on crashes. For `branches`, the first column is the 7-char commit instead of the iter number. ``` iter status move_type analysis_summary description 1 0.6320 keep swing baseline; grad norms clean, no pathologies baseline 2 0.5910 discard swing ResNet blocks; gradient flow good but overfit on 5 epochs ResNet-style residual blocks 3 0.6890 keep swing MLP-Mixer; feature mixing effective, less overfit MLP-Mixer token+channel mix 4 0.7120 keep merge Mixer channel-mix + ResNet skips; best of both merge: Mixer + ResNet skips 5 0.7250 keep exploit train/val gap small; warmup helped stability add 2-epoch LR warmup 6 0.7240 discard exploit no gain from dropout; val acc unchanged add dropout 0.1 7 0.6800 discard swing forced pivot after stagnation guard; ViT too data-hungry ViT patch-16 from scratch ``` `approaches.md` registry — one entry per swing/merge, format above. Example: ```markdown # Approach Registry The merge step consults this file to find complementary approaches to combine. ## Approach 3: MLP-Mixer - **move_type**: swing - **iter**: 3 - **val_acc**: 0.6890 (status: keep) - **axes changed**: architecture - **key ideas**: token mixing + channel mixing, no convolutions - **strengths**: less overfit than ResNet, fast per-epoch - **weaknesses**: spatial locality not exploited; edges/textures underused - **ablated?**: no ``` Report the **best** iteration (not necessarily the last) when summarising. Do not commit `results.tsv` or `approaches.md` — leave them untracked. ## Constraints - **Only edit files in ``** — confirm before every edit, because everything else is read-only ground truth and the eval harness defines ``. - **One change per iteration**, so each metric delta is attributable to one move. - **The scheduler rules are hard constraints, not suggestions.** When `consecutive_exploit >= ` you MUST choose swing or merge — exploit is unavailable regardless of what the analysis suggests; this guard is what prevents endless hill-climbing. - **Analysis is mandatory every iteration** and must produce real files in `iter/results/`; a move with no analysis behind it degrades the loop into blind iteration and starves the next decision. - A swing must be *fundamentally* different from every prior swing; a merge must name what it takes from each parent — otherwise the registry and the breadth guarantee mean nothing. - Do not install packages or add dependencies the project lacks; helper scripts stay stdlib-only. - Always redirect training output to ``; never use `tee`. The sandbox is self-contained — no `../` escapes. ## Stops This loop runs **forever until the human interrupts it** — do not pause to ask "should I continue?" or "is this a good stopping point?". The human may be away and expects autonomous work indefinitely. A working result is the start of the next iteration, not the end. If ideas run dry: re-read the in-scope files for missed angles, go deeper on the analysis (gradients/activations/embeddings/errors always surface something), combine previous near-misses from `approaches.md`, or take a more radical swing.