--- name: Skill Forge Eval slug: skill-forge-eval category: Quality description: Skill Forge Eval runs evaluation pipelines for Claude Code skills to check triggering accuracy, workflow correctness, and output quality. It creates eval metadata, runs executor and grading agents, and produces benchmark and feedback reports. github: "https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval" language: Python stars: 169 forks: 44 install: "npx degit https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval ~/.claude/skills/skill-forge-eval" installs_to: ~/.claude/skills/skill-forge-eval source_path: skills/skill-forge-eval/SKILL.md collection_size: 10 category_size: 1897 collection_url: "https://dirskills.com/collections/AgriciDaniel/skill-forge" added: 2026-09-08T05:33:55.938Z last_synced: 2026-09-08T05:33:55.938Z canonical_url: "https://dirskills.com/skills/skill-forge-eval" --- # Skill Forge Eval Skill Forge Eval runs evaluation pipelines for Claude Code skills to check triggering accuracy, workflow correctness, and output quality. It creates eval metadata, runs executor and grading agents, and produces benchmark and feedback reports. **Install:** ```bash npx degit https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval ~/.claude/skills/skill-forge-eval ``` ## README # Skill Evaluation Pipeline Run structured evaluations against Claude Code skills to verify triggering, correctness, and quality using a multi-agent pipeline. ## Process ### Step 1: Define Eval Set Accept eval definitions from: - **Path to eval set JSON**: `evals/evals.json` or user-specified file - **Inline prompts**: User provides eval queries directly - **Auto-generated**: Generate from skill description (see Step 1b) **Eval set JSON schema:** ```json { "skill_name": "my-skill", "skill_path": "./my-skill", "evals": [ { "eval_id": 0, "eval_name": "descriptive-name", "prompt": "The user's task prompt", "input_files": [], "assertions": [ { "name": "output-has-score", "check": "Output contains a numeric score between 0-100", "weight": 1.0 } ], "should_trigger": true } ] } ``` #### Step 1b: Auto-Generate Eval Set If no eval set exists, generate one: 1. Read the skill's SKILL.md description and instructions 2. Run `python scripts/generate_eval_set.py ` to produce a starter set 3. Present the generated set to the user for review and editing 4. User approves or modifies before proceeding ### Step 2: Set Up Workspace Create the eval workspace **outside** the skill directory to avoid confusing eval artifacts with skill files. Use a sibling directory or a dedicated location: ``` eval-workspace/ iteration-1/ eval-0/ eval_metadata.json # Assertions and config for this eval with_skill/ outputs/ # Skill execution outputs timing.json # Token count + duration grading.json # Assertion results + evidence baseline/ outputs/ timing.json grading.json eval-1/ eval_metadata.json with_skill/ outputs/ timing.json grading.json baseline/ outputs/ timing.json grading.json benchmark.json # Aggregated metrics benchmark.md # Human-readable report ``` For each eval directory, create `eval_metadata.json` from the eval set entry: ```json { "eval_id": 0, "eval_name": "descriptive-name", "prompt": "The user's task prompt", "assertions": [...], "should_trigger": true } ``` ### Step 3: Execute Eval Runs For each eval in the set, spawn two parallel runs: **With-skill run** (delegate to `agents/skill-forge-executor.md`): ``` Execute this task: - Skill path: - Task: - Input files: - Save outputs to: /iteration-/eval-/with_skill/outputs/ - Outputs to save: ``` **Baseline run** (delegate to `agents/skill-forge-executor.md`): - For new skills: run without the skill loaded - For improved skills: run with the previous version (snapshot it first) Save timing data to `timing.json` in each run directory: ```json { "total_tokens": 84852, "duration_ms": 23332, "total_duration_seconds": 23.3 } ``` ### Step 4: Grade Results Delegate to `agents/skill-forge-grader.md` for each completed run: 1. Grade against assertions defined in eval_metadata.json 2. Save results to `grading.json` per run: ```json { "eval_id": 0, "run_type": "with_skill", "assertions": [ { "name": "output-has-score", "passed": true, "evidence": "Found score: 87/100 on line 14" } ], "pass_rate": 1.0 } ``` ### Step 5: Aggregate and Analyze 1. Run `python scripts/aggregate_benchmark.py /iteration- --skill-name ` 2. This produces `benchmark.json` and `benchmark.md` with: - Pass rate per eval (with_skill vs baseline) - Average time and token usage - Improvement ratio (with_skill / baseline) 3. Delegate to `agents/skill-forge-analyzer.md` to: - Surface patterns that aggregate stats might hide - Identify consistently failing assertion types - Flag regressions from previous iterations ### Step 6: Present Results Generate a summary report: ```markdown # Eval Report: [skill-name] — Iteration [N] ## Overall | Metric | With Skill | Baseline | Delta | |--------|-----------|----------|-------| | Pass Rate | X% | Y% | +Z% | | Avg Time | Xs | Ys | -Zs | | Avg Tokens | X | Y | -Z | ## Per-Eval Results | Eval | With Skill | Baseline | Status | |------|-----------|----------|--------| | eval-0 | PASS | FAIL | Improved | | eval-1 | PASS | PASS | Maintained | ## Patterns & Insights [From analyzer agent] ## Recommendations [Specific improvements based on failures] ``` ### Step 7: Collect Feedback Save user feedback to `feedback.json`: ```json { "reviews": [ { "run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "2026-03-06T12:00:00Z" } ], "status": "complete" } ``` Pass feedback to `/skill-forge evolve` for the next iteration. ## Advanced: Blind Comparison For rigorous A/B testing between skill versions: 1. Delegate to `agents/skill-forge-comparator.md` 2. Pass two directories: `eval-/with_skill/outputs/` and `eval-/baseline/outputs/` 3. Comparator assigns random labels (Version A / Version B) so it cannot know which is new 4. Rates each output on assertion criteria from `eval_metadata.json` 5. Returns preference scores without knowing which is "new" vs "old" ## Error Handling - **Executor timeout**: If a run exceeds 5 minutes, terminate and mark as `"timed_out": true` in timing.json - **Executor failure**: If a run crashes, save the error to `error.txt` in the run directory and continue with remaining evals - **Grading failure**: If grading cannot determine pass/fail, mark assertion as `"passed": null` with evidence explaining why - **Missing files**: If timing.json or grading.json is missing after a run, flag the eval as incomplete in the report - **Partial completion**: Always aggregate and report whatever results are available — do not block on one failed eval ## Quality Gates Before marking an eval run as complete: - [ ] All evals executed (with_skill + baseline) - [ ] Timing data captured for every run - [ ] All assertions graded with evidence - [ ] Benchmark aggregated with pass rate, time, tokens - [ ] Analyzer patterns documented - [ ] Results presented to user