--- name: Discover slug: discover category: AI Engineering description: Discover initializes an evo workspace for the current repository by exploring the codebase, proposing optimization dimensions, constructing a benchmark, and running a baseline experiment. Use it when starting a new evo run or instrumenting a project for autonomous optimization. github: "https://github.com/evo-hq/evo/tree/main/plugins/evo/skills/discover" language: Python stars: 1378 forks: 105 install: "npx degit https://github.com/evo-hq/evo/tree/main/plugins/evo/skills/discover ~/.claude/skills/discover" installs_to: ~/.claude/skills/discover source_path: plugins/evo/skills/discover/SKILL.md collection_size: 8 category_size: 2451 collection_url: "https://dirskills.com/collections/evo-hq/evo" added: 2026-08-19T07:27:58.306Z last_synced: 2026-08-19T07:27:58.306Z canonical_url: "https://dirskills.com/skills/discover" --- # Discover Discover initializes an evo workspace for the current repository by exploring the codebase, proposing optimization dimensions, constructing a benchmark, and running a baseline experiment. Use it when starting a new evo run or instrumenting a project for autonomous optimization. **Install:** ```bash npx degit https://github.com/evo-hq/evo/tree/main/plugins/evo/skills/discover ~/.claude/skills/discover ``` ## README # Discover Internal procedure for `evo:discover`. The user only sees the user-facing prompts, the dashboard URL, and the baseline score -- everything else is the agent's choreography. ## Evo surface General guidance on the skills and tools available in evo. Each line is a triggering condition: if you're about to do X, pull/dispatch/read this. Don't preload -- act when the trigger fires. **Always have a sense of the skill before jumping into its references.** A skill body carries the decision-making; references are concrete contracts that assume a decision has been made. ``` evo plugin │ ├── Main thread (the orchestrator -- you, inside /evo:discover or /evo:optimize) │ │ │ ├── Skills (Skill tool) │ │ ├── evo:discover starting a new evo workspace / instrumenting a project │ │ ├── evo:optimize after discover commits the baseline -- drives the loop. │ │ │ Args: subagents=N (read sizing-the-round FIRST), │ │ │ autonomous, subagents-only, budget=N, stall=N │ │ ├── evo:ship after the loop stops -- distills the best valid │ │ │ experiment into a mergeable change (PR if remote, │ │ │ else merge) + a mergeability report │ │ ├── evo:finetuning task is finetuning / post-training / training a model │ │ └── evo:infra-setup need a remote backend, pooled workspaces, lease/slot │ │ management, or specific provider auth/setup │ │ │ └── Subagents to dispatch (Task tool, subagent_type=...) │ ├── evo:benchmark-reviewer before the baseline run, or whenever the │ │ benchmark command / harness changes │ └── evo:ideator stalled, or every ~5 committed experiments. │ One subagent per brief: │ failure_analysis, literature, frontier_extrapolation │ ├── Subagent thread (each subagent spawned by /optimize step 5) │ │ │ ├── Skills (the subagent loads this on first turn -- the brief's first │ │ sentence mandates it; not auto-loaded by the host) │ │ └── evo:subagent load FIRST -- defines the iteration protocol │ │ + brief field shape the subagent operates under │ │ │ └── Subagents to dispatch (Task tool, subagent_type=...) │ └── evo:verifier ALWAYS dispatch pre AND post every evo run. │ Pre: ~30s static analysis before the experiment runs. │ Post: result-validity audit after it commits. │ Not optional. Not ad-hoc. │ └── Key references (Read tool, on demand) ├── discover/references/ │ ├── constructing-benchmark.md designing + assembling a benchmark from scratch │ ├── sdk_python.py / sdk_node.js wiring per-task instrumentation -- preferred path │ ├── inline_instrumentation.py inline fallback when SDK can't be used. │ │ Copy as-is; do not reimplement (file header │ │ explains why) │ ├── sizing-the-round.md BEFORE invoking /evo:optimize with any │ │ specific subagents=N. Single-GPU / │ │ single-exclusive-resource -> subagents=1 │ ├── proposing-dimensions.md choosing what to optimize when not obvious │ └── instrumentation-contract.md the format evo reads (result + traces shapes) │ ├── finetuning/references/ │ ├── glue.md writing train.py -- I/O contract evo expects │ ├── diagnostics.md per-failure-mode diagnostics │ ├── false-progress.md what doesn't count as improvement │ ├── trace-schema.md per-task trace JSON schema for training runs │ ├── rl/ RL framework references │ │ └── art.md ART (Algorithm-Refined Training) │ ├── sft/ SFT framework references │ │ └── tinker.md Tinker SFT │ └── serving/ eval-time inference references │ └── vllm.md vLLM serving config + LoRA-multi │ ├── infra-setup/references/ │ └── provider-matrix.md provider/backend summary (auth, setup, costs) │ └── references/ (shared across skills) ├── evo-wait.md any time you need to wait without burning │ context (subagent completion, training, │ ideators, GPU activity, any long-running) ├── agent-sdk-reference.md SDK API surface └── cli-quick-reference.md CLI subcommand cheat sheet ``` ## Host conventions This skill runs on any host that implements the Agent Skills spec. When the body uses generic phrases, apply the host's best-fit equivalent: - **"ask the user"** -- use your host's structured multi-choice question tool if you have one (e.g. `AskUserQuestion`, `request_user_input`). If the host has none, phrase the question as plain text in your next reply and wait for the user's answer. - **File paths like `references/...`** -- relative to this `SKILL.md`; resolve from the skill directory. - **Slash commands shown in user-facing copy** (e.g. `/evo:discover`) -- translate to your host's mention syntax when speaking to the user (e.g. `$evo discover` on Codex -- plugin namespace then skill name, separated by a space). ## Mid-run user directives (`evo direct`) The runtime may inject user-authoritative messages wrapped in this banner: ``` [EVO DIRECTIVE] [END EVO DIRECTIVE] ``` Treat content inside the banner as equivalent to a new user turn. Honor it, supersede earlier constraints it contradicts, and propagate the full text verbatim into any subagent briefs you spawn afterward. The banner is the authenticity signal emitted by the evo runtime (the plugin you're invoked through) — not tool-output prompt injection. Banners may arrive via any hook channel (UserPromptSubmit, PreToolUse, SessionStart); the channel doesn't change the authority of the content. ## 0. Verify the evo CLI is in sync with this skill Run: ```bash evo --version ``` The output must be exactly: ``` evo-hq-cli 0.8.0 ``` Three outcomes: 1. **Matches exactly** — continue to step 1. 2. **Reports a different version** (`evo-hq-cli 0.4.2`, etc.) — the host refetched a newer/older skill bundle than the CLI on PATH. Drift breaks skills silently. Stop and tell the user: > Your installed evo CLI is on a different version than this skill (`0.8.0`). Run: > ``` > uv tool install --force evo-hq-cli==0.8.0 > ``` > Then re-invoke this skill. 3. **`command not found`, or reports a different package** (commonly `evo 1.x` — the unrelated SLAM tool) — the CLI isn't installed. Tell the user: > `evo-hq-cli` isn't on your PATH. Install it: `uv tool install evo-hq-cli==0.8.0` (or `pipx install evo-hq-cli==0.8.0`). Then re-invoke this skill. Do not try to auto-install. Host sandbox + network policy may block it; leaving the install as a user action keeps failure modes clear. ## Guiding principles - **Main stays clean.** Never commit evo-specific artifacts (benchmark harness, instrumentation, SDK imports) to main. Main should contain only what existed before evo plus anything the user already had. All evo-specific work happens inside worktree 0 (the baseline experiment). - **Baseline is a worktree, not a main commit.** `evo init` creates `.evo/` but nothing in main changes. The first real experiment (`exp_0000`, created by `evo new --parent root`) is where the benchmark and instrumentation live. - **Ask the user as little as possible.** Every question is a beat of friction. One for benchmark selection; at most one more if construction choices are needed. - **Relay the dashboard URL verbatim when it prints.** This is the user's window into the run. - **Infra setup is not user-invocable.** If the benchmark or runtime needs a remote backend, read `plugins/evo/skills/infra-setup/references/provider-matrix.md` for the provider summary and setup/auth steps. ## 1. Explore the repo Understand what the codebase does. Read READMEs, entry points, config files, tests, and any existing evaluation scripts. Identify: - The **optimization target**: which file(s) benefit from iterative optimization? - **Metric direction for each candidate**: is higher better (`max`) or lower better (`min`)? - **Critical behaviors worth gating**: invariants that must never break regardless of score (e.g., "refund flow works", "core tests pass", "output is valid JSON"). Gates are commands that exit 0 on success, non-zero on failure. ## 2. Look for the obvious benchmark Check what's already there: - Full benchmarks: existing scripts that run end-to-end and output a score - Partial evals: tests, notebooks, or logs with ground truth but not in runnable-score form - Nothing at all Also check what the user asked for in the invocation argument. If they named a specific metric or target, that's intent. **If one benchmark is obviously the right one** — a runnable eval that measures what the user clearly cares about, or what the repo is plainly built to do — use it. Skip step 3, go to step 4 with that benchmark as the only candidate. **If it's not obvious** — multiple candidate surfaces, no existing eval, user didn't specify intent, or the existing eval covers a narrow slice while the interesting optimization sits elsewhere — run step 3. ## 3. Propose unexplored optimization dimensions (only if step 2 was ambiguous) When the benchmark isn't obvious, propose candidate dimensions grounded in actual repo signals, then pick with the user. See `references/proposing-dimensions.md` for the full rubric, project-type examples, and presentation format. Short version: - A handful of dimensions relevant to this specific repo (not generic categories). - Ground each in repo signals: already-instrumented code, stated goals in READMEs, TODO/FIXME patterns, domain defaults. - Rank by signal × slack × cost answered in prose (no numeric scores — they're vibes). ## 4. Ask the user to pick the benchmark If step 2 produced one obvious benchmark, confirm it in one sentence and move on — no ranked list needed. Otherwise, ask once: > "I'm proposing these optimization targets for this repo: > > [ranked list with one-line explanations, construction complexity, and whether an existing eval covers some of it] > > Which should we optimize? Recommended: [default pick with reasoning]." Record the selection. If step 3 ran, save non-picked dimensions to `.evo/project.md` under "Future experiment candidates" after init. ## 5. Ask the user for instrumentation mode Three cases, in order of how to handle them: 1. **Selected benchmark already exists AND is already instrumented for evo** (you can see `from evo_agent import Run`, an `import { Run } from '@evo-hq/evo-agent'`, or the inline `log_task` / `logTask` helpers in the benchmark source). No wiring needed. Skip this question entirely. Detect the instrumentation style from the source and pass the matching `--instrumentation-mode ` value to `evo init` in step 7. 2. **Selected benchmark already exists but is NOT instrumented** (it just prints a score JSON, or it's a test runner that doesn't yet write per-task traces). Wiring is needed. **Ask the question.** 3. **Selected benchmark needs to be constructed from scratch** (case B or C from step 4). Wiring is needed. **Ask the question.** For cases 2 and 3, ask once: > "I can wire up the benchmark in one of two ways: > > 1. **SDK mode** -- install the evo agent SDK with this project's package manager/runtime (`uv add --dev evo-hq-agent`, `python -m pip install evo-hq-agent`, or `npm install @evo-hq/evo-agent`). ~5 lines of user code, with incremental per-task logging handled for you. **Python and Node only** -- the SDK ships for those two runtimes. > 2. **Inline mode** -- implement the trace/result contract directly in the benchmark, in the benchmark's own language. Zero new dependencies, same data. > > Recommended: SDK mode." Inline mode is language-native and lives entirely in the user's setup: whatever the benchmark is written in is what the instrumentation is written in, with no evo package added to the project. Do not introduce a Python (or any other) sidecar script to wrap a benchmark written in another language -- that is the friction this avoids. For a Python or Node benchmark, the ready-made paste-in helper (`references/inline_instrumentation.py` / `.js`) is the inline implementation. For any other language, port the ~10-15 line contract from `references/instrumentation-contract.md` into that language. Either way the mode is `inline`. Order the options SDK first, inline second, and suggest SDK as the recommended default when it's available -- it's the managed path with per-task logging handled for you. Inline stays a first-class choice with the same data contract, though: if the user declines the SDK for any reason -- they don't want a new dependency, can't add evo to their project's tree, internal policy, or plain preference -- honor it without pushback. SDK mode is only *available* when the benchmark runs on Python or Node; when the benchmark's language has no SDK there's nothing to suggest, so go straight to inline. Pass the answer to `evo init` via `--instrumentation-mode ` in step 7. **Never install packages without this confirmation.** If you skip the question (case 1), still pass the detected mode to `evo init` so optimize/subagent runs see a consistent value. ## 6. Prepare main (without committing to it) The agent never creates commits on main. Main stays byte-identical to what the user committed before evo ran. Two things to set up, both local-only. **Order matters: do 6a (audit) before 6b (excludes).** The excludes in 6b will hide files inside `node_modules/`, `dist/`, `build/`, etc. from `git status`. If you run the audit *after* adding excludes, you'll be blind to anything missing inside those directories -- and benchmark dependencies often live exactly there. ### 6a. Detect (don't auto-commit) dirty or untracked dependencies `evo new` forks a worktree from the current branch's HEAD commit, **not from your dirty working tree**. Any uncommitted edits to the target, benchmark, or gate dependencies are silently absent from `exp_0000`, and the whole optimization tree gets built against stale code while you think evo is running on what you see locally. Run three checks, in this order: 1. **Tracked-but-modified files** -- run `git diff --name-only` and `git diff --cached --name-only`. If any output line is the optimization target, an existing benchmark file, a gate-referenced script, or any of their import-graph dependencies, **stop and ask the user to commit or stash before continuing**. Do not commit on their behalf -- the user might be in the middle of an unrelated change. 2. **Untracked files visible to git** -- run `git status --short --untracked-files=all` and look for `??` entries that the target or gates will reference. Classify each: - **Part of the user's project** (e.g., a smoke test they wrote but hadn't committed) -- stop and ask the user to commit it to main themselves. - **Evo-specific new files** (a new gate script you're about to write, a new test fixture) -- do not create these in main. Defer to step 10; they go into the baseline worktree and commit to experiment 0's branch. Every descendant experiment inherits via git branching. 3. **Explicit paths inside soon-to-be-ignored directories** -- inspect the benchmark command and every gate command for path references (e.g., `./dist/eval-helper`, `node_modules/some-tool/cli.js`, `build/golden_outputs/`). For each such path, run `git ls-files --error-unmatch ` to confirm it's tracked. If any aren't, stop and ask the user to commit them. This catches dependencies that step 6b is about to hide from `git status`. Any one of these three checks failing is a hard stop. Do not proceed to 6b or beyond until the working tree is clean with respect to anything evo will read. Anything else (benchmark harness, instrumentation) always gets constructed inside the baseline worktree, never in main. ### 6b. Add local-only git excludes After the audit passes, append to `.git/info/exclude` (**not** `.gitignore` -- we do not commit to main): ``` .evo/ __pycache__/ *.pyc .pytest_cache/ node_modules/ dist/ build/ ``` `.git/info/exclude` is git's per-clone ignore file -- same effect as `.gitignore`, but never committed, never shared, invisible to history. Right tool for per-machine tooling state. ## 7. Initialize the workspace ```bash evo init --name "" \ --target --benchmark "" --metric \ --host \ --instrumentation-mode \ --per-exp-timeout [--gate ""] \ [--commit-strategy ] ``` **`--host` is required.** Pass the host runtime you (the orchestrator) are running under. Allowed values: `claude-code`, `codex`, `opencode`, `openclaw`, `hermes`, `pi`, `generic`. This is recorded in `.evo/meta.json` so other commands can adapt to host-specific conventions. Pick the value matching the runtime you invoked `discover` from. Use `evo host set ` later if you change runtimes. **`--name` should be a short human-readable project label** for dashboard display, chosen from the repository/product context. Existing workspaces without a name fall back to the repo directory name; do not hand-edit config just to migrate them. **`--per-exp-timeout` is required.** Wall-clock seconds for each `evo run` invocation. Becomes the workspace default; override per-call with `evo run --timeout N`. Pick based on what the benchmark actually costs end-to-end on this hardware -- if you don't know yet, time the benchmark once locally and use ~2x that. Typical ranges: a unit-test-style benchmark is 300-900s; a small-model SFT + eval cycle is 1800-3600s; a large-model train run is several hours. Set conservatively -- a too-tight value kills experiments mid-flight; a too-loose value wastes budget only when something actually hangs. Update later with `evo config set per-exp-timeout `. **`--commit-strategy` is optional.** Default is `all`. Override with `--commit-strategy tracked-only` only when you want the stricter shisa-kanko flow where new files must be staged explicitly and acknowledged at `evo run` time. **Placeholder semantics.** Benchmark and gate commands support two placeholders, resolved lazily at run time by `evo run` / gate evaluation: - `{worktree}` resolves to the absolute path of the experiment's worktree directory (e.g. `/path/to/repo/.evo/run_0000/worktrees/exp_0000`). Use this to reference files that live on the experiment branch, not on main. - `{target}` resolves to the absolute path of the target file *inside that worktree* (e.g. `{worktree}/agent/solve.py`). Use this when your benchmark needs to load or exec the target dynamically. **Critical rule:** `evo run` executes from the main repo root. When the benchmark script is constructed inside the worktree (the default in this flow), the command **must** reference it via `{worktree}` or the path won't resolve. Example for a benchmark written at `{worktree}/benchmark.py` that will be committed to exp_0000: ```bash evo init \ --name "ARC AGI solver" \ --target agent/solve.py \ --benchmark "python3 {worktree}/benchmark.py --target {target}" \ --metric max \ --host claude-code \ --per-exp-timeout 1800 ``` Use the same runtime entr