--- name: FrontierHarness Eval slug: frontierharness-eval-2 category: Quality description: FrontierHarness Eval benchmarks a third-party coding-agent harness against FrontierHarness baselines using identical runs, saved trajectories, and a comparison report. Use it when evaluating a harness, comparing results, or checking FrontierHarness, DeepSWE, or Terminal-Bench setups. github: "https://github.com/frontier-harness-eval/eval/tree/main/skills/frontierharness-eval" language: JavaScript stars: 162 forks: 9 install: "npx degit https://github.com/frontier-harness-eval/eval/tree/main/skills/frontierharness-eval ~/.claude/skills/frontierharness-eval" installs_to: ~/.claude/skills/frontierharness-eval source_path: skills/frontierharness-eval/SKILL.md collection_size: 1 category_size: 1897 added: 2026-09-08T05:35:49.322Z last_synced: 2026-09-08T05:35:49.322Z canonical_url: "https://dirskills.com/skills/frontierharness-eval-2" --- # FrontierHarness Eval FrontierHarness Eval benchmarks a third-party coding-agent harness against FrontierHarness baselines using identical runs, saved trajectories, and a comparison report. Use it when evaluating a harness, comparing results, or checking FrontierHarness, DeepSWE, or Terminal-Bench setups. **Install:** ```bash npx degit https://github.com/frontier-harness-eval/eval/tree/main/skills/frontierharness-eval ~/.claude/skills/frontierharness-eval ``` ## README # FrontierHarness Eval for a Third-Party Harness Score a harness that is not in the published FrontierHarness v1.0 set, on the same tasks, runtime, and cost accounting, so the result can be placed next to the twelve baseline configurations in `results/eval-data.json`. ## Prerequisites Confirm all of these before touching a runtime: ```bash runta --version # brew install runta-dev/tap/runta (or npm i -g @runta/runta-cli) runta checkpoint ls # any API call proves the CLI is authenticated jq --version && node --version # jq for trial parsing, node >= 18 for the report scripts ``` Run every command below from the repository root — or from the workspace that `npx @frontierharness/eval` creates, which has the same layout — so `results/eval-data.json`, `benchmark.json`, and `tasks/` resolve. This repo has its own `scripts/` directory, so address the skill's scripts through an explicit variable rather than a bare `scripts/`: ```bash FH=skills/frontierharness-eval/scripts ``` Collect from the user before starting: harness name and version, the GitHub repo and commit for the harness under evaluation, and the task subset. **The model is not a variable; the provider is.** FrontierHarness holds the model constant at **Kimi K3** so the harness is the only thing that differs. Which provider serves it is up to the user, selected with `--provider`. The scripts warn if the model is not Kimi K3, and refuse an unknown provider name. | `--provider` | Model route | Key to collect | | --- | --- | --- | | `fireworks` (default, used by the published baselines) | `fireworks_ai/accounts/fireworks/models/kimi-k3` | `FIREWORKS_API_KEY` | | `moonshot` | `moonshot/kimi-k3` | `MOONSHOT_API_KEY` | | `openrouter` | `openrouter/moonshotai/kimi-k3` | `OPENROUTER_API_KEY` | | `together` | `together_ai/moonshotai/Kimi-K3` | `TOGETHER_API_KEY` | | `custom` | supply `--model` | supply `--secret-name` | Ask which provider the user has a key for, and use `--provider fireworks` if they have no preference. A different provider keeps the pass rate comparable, since the model is identical; it only puts the cost column at risk, so check that the provider's input, cached-input, and output prices match the ones in `reference.md`. The report raises this caveat automatically. Only change the *model* if the user explicitly wants a non-comparable run, and say so in the report. ## Workflow Copy this checklist into your working notes and keep it updated: ``` - [ ] 1. Clean runtime created - [ ] 2. Harness repo cloned at a pinned commit - [ ] 3. Benchmark stack installed and frozen as a golden checkpoint - [ ] 4. Trials run from fresh restores, trajectories saved - [ ] 5. Comparison diagram generated - [ ] 6. Report built and shared ``` Steps 1 through 3 are one command (`provision-golden-checkpoint.sh`), but read the per-step notes below because the fidelity rules live there. ### 1-3. Clean runtime, repo, and golden checkpoint Authenticate the CLI with `runta login`, or set `RUNTA_TOKEN` if you prefer an explicit token. The provider key only has to be exported the first time: it is stored as a tenant secret, and the API never hands the value back, so a later re-cut of the checkpoint reuses the stored secret instead of demanding the plaintext again. ```bash export FIREWORKS_API_KEY=... # or the key for whichever --provider you pick $FH/provision-golden-checkpoint.sh \ --runtime fh-build \ --checkpoint fh-golden-myharness-v1 \ --harness my-harness \ --provider fireworks \ --repo https://github.com/acme/my-harness \ --commit 9f2c1ab \ --cpus 4 --memory 8192 --disk-size-gib 100 \ --prepull-tasks tasks \ --install-script ./install-my-harness.sh ``` What the script does, and why each part matters: - **Clean runtime.** `runta run` with no `--agent` preset, so no vendor harness is pre-installed and nothing competes with the harness under test. Disk defaults to 100 GiB, which is what the eval environment needs: building a harness from source plus the pre-pulled task images overflows the 16 GiB Runtime Image default. Keep it at 100 GiB so every trial restores with the same capacity as the baselines. - **Repo pinned by commit.** The harness is cloned to `/work/harness` at `--commit`. A branch name is not reproducible; always pin a SHA. - **Benchmark stack.** Installs `uv`, Harbor `0.22.0` for Terminal-Bench, `datacurve-pier==0.3.1` plus the `deep-swe` corpus at **v1.1**, and `runta-sdk[harbor]`. - **Credential as a secret stub.** The provider key named by `--secret-name` (defaulted from `--provider`) is stored with `runta secret set` and injected by the egress proxy, so the real key never lands inside the runtime or inside a checkpoint. The script also allowlists the provider host (and `astral.sh` for verifier uv downloads). Verify with `runta exec fh-build -- sh -lc 'test "$FIREWORKS_API_KEY" = runta-secret-stub'`. - **Cache warming on sample tasks only.** Docker images for the formal tasks are pre-pulled, but the only task ever *executed* before the checkpoint is `terminal-bench-sample@2.0` with Harbor's `oracle` agent. Never execute a formal task before the checkpoint — that is warm-cache bias and it invalidates the comparison. - **Manifest.** `/work/manifest.json` records tool versions, the harness commit, the model, and image digests. It is captured inside the checkpoint and copied out, which is what makes a later run verifiable. - **Golden checkpoint.** `runta checkpoint create` freezes filesystem *and* process state. Every trial restores from it, so all trials share one identical cold start. Before moving on, confirm the checkpoint is ready: ```bash runta checkpoint ls ``` ### 4. Run trials and save trajectories Each task gets its own fresh restore, then the runtime is deleted. Never reuse a runtime across tasks. ```bash $FH/run-trials.sh \ --checkpoint fh-golden-myharness-v1 \ --harness my-harness \ --provider fireworks \ --run-id 2026-09-02-myharness \ --out runs ``` Pass the same `--provider` here as at provisioning time. The checkpoint has that provider's key name baked in as a stub, so a mismatch leaves the harness without a credential. With no `--tasks`, the script runs every task defined in this repo's `tasks/` directory, reading the suite-prefixed id out of each `tasks//task.toml`. That is the published 30-task set, and it is the same directory `--prepull-tasks` warmed at provisioning time, so the set that runs is exactly the set that was prepared — there is no second list to drift out of sync. To run a subset, point `--tasks` at a file holding one suite-prefixed id per line: ``` terminal-bench/regex-log terminal-bench/build-cython-ext datacurve/anko-typed-variable-bindings ``` A subset is not comparable to the published leaderboard; say so in the report. Per task the script restores the checkpoint, runs the harness through Harbor (`terminal-bench/*`) or Pier (`datacurve/*`), copies `/work/jobs/` out, writes a normalized `trial.json`, and removes the runtime. Evidence lands in `runs//trials//` and includes the agent trajectory, verifier logs, the `model.patch` artifact, and the raw runner stdout. Keep it — the report links to it and it is the only proof a score is real. If a trial dies on infrastructure rather than the task, mark it and rerun it rather than scoring it as a failure: ```bash # Re-running an existing --run-id only replaces the tasks listed, leaving the rest. echo "terminal-bench/" > retry.txt $FH/run-trials.sh --checkpoint fh-golden-myharness-v1 --harness my-harness \ --run-id 2026-09-02-myharness --tasks retry.txt --out runs # If it fails on infrastructure again, mark it so it is excluded rather than scored. trial=runs/2026-09-02-myharness/trials/terminal-bench-/trial.json jq '.status = "infra_invalid" | .success = false' "$trial" > "$trial.tmp" && mv "$trial.tmp" "$trial" ``` ### 5. Generate the diagram ```bash node $FH/normalize-results.mjs --run runs/2026-09-02-myharness --label "My Harness" node $FH/generate-chart.mjs --run runs/2026-09-02-myharness ``` `normalize-results.mjs` folds the trials into `candidate.json` using the same field names and definitions as `results/eval-data.json`, so the candidate slots directly into the baseline set. `generate-chart.mjs` writes `runs//report/chart.svg`: a pass-rate versus cost scatter with the twelve baselines muted and the candidate highlighted, plus a pass-rate ranking panel. ### 6. Build and share the report ```bash node $FH/build-report.mjs --run runs/2026-09-02-myharness ``` This writes `runs//report/REPORT.md` and a self-contained `runs//report/index.html` with the chart inlined, so a single file can be attached or opened anywhere. Both end with a link back to the source evaluation at . Share it with whichever path fits: ```bash # Public link, no repo access needed gh gist create runs//report/REPORT.md runs//report/chart.svg --public \ --desc "FrontierHarness Eval: My Harness" # Single portable file open runs//report/index.html # Commit alongside the published results git add runs/ && git commit -m "add My Harness evaluation" ``` ## Reproducibility rules A result is only comparable to the published leaderboard if all of these hold. State explicitly in the report which ones were relaxed. | Rule | Why | | --- | --- | | Kimi K3, the same model as every published configuration, from any provider serving it | Harness effects and model effects are otherwise inseparable | | Provider token prices matching the baselines, or a stated caveat | Pass rate survives a provider swap; the cost column does not | | One golden checkpoint per task set, every trial a fresh restore | Identical cold start, identical disk and memory state | | Identical vCPU, memory, and disk (100 GiB) across all restores | Compute differences show up as time and pass-rate differences | | No formal task executed before the checkpoint | Prevents warm-cache bias | | Canonical result is the first valid attempt | Matches `benchmark.json` `canonical_selection` | | Infra failures marked `infra_invalid`, not `failure` | A crashed runtime or failed restore is not a harness failure. A harness process crash is a failure and stays in the denominator | | One shared golden checkpoint for third-party runs | The published 360 cells used a per-task checkpoint. The skill freezes one checkpoint with every image pre-pulled; say so in the report | Cost comparability has one caveat worth repeating in every report: baseline costs in `results/eval-data.json` reprice first-turn cache reads consistently across harnesses. `effective_cost_per_pass` (total cost over all tasks divided by passes) is reproducible from raw per-task cost and is the safe field to compare. The `*_normalized` fields are not reproducible from public data — the scripts leave them null rather than inventing values. ## Metric definitions `normalize-results.mjs` computes these from `trial.json` files, matching the baseline: | Field | Definition | | --- | --- | | `pass_rate` | passes / tasks attempted (infra_invalid excluded) | | `expected` | published task count from `benchmark.json` (30) | | `completed` | scoreable trials | | `comparable` | `completed === expected`; the report ranks only then | | `effective_cost_per_pass` | total cost across *all* tasks / passes | | `median_cost_per_success` | median per-task cost over successful tasks only | | `median_duration_seconds` | median wall-clock over successful tasks only | | `cache_hit_rate_typical` | median runner-reported cache hit rate over successful tasks; not the baseline's repriced series | | `mean_turns` | mean agent turns over successful tasks only | ## Additional resources - Command reference, runner templates, and troubleshooting: [reference.md](reference.md) - Published results and task definitions: `results/eval-data.json`, `tasks//task.toml` - Source evaluation: