Documentation
README
Skill Benchmarking & Performance Tracking
Measure and compare skill performance across iterations with statistical rigor using multiple trials, variance analysis, and trend tracking.
Process
Step 1: Define Benchmark Configuration
Accept configuration as:
- Existing eval set: Path to
evals/evals.json(from/skill-forge eval) - Benchmark config: Custom config with trial count and thresholds
Benchmark config schema:
{
"skill_name": "my-skill",
"skill_path": "./my-skill",
"eval_set_path": "./evals/evals.json",
"trials_per_eval": 3,
"baseline_type": "no_skill",
"previous_benchmark": null,
"thresholds": {
"min_pass_rate": 0.8,
"max_avg_tokens": 100000,
"max_avg_duration_seconds": 120,
"min_improvement_ratio": 1.0
}
}
This is the opening of the README. Read the full README on GitHub.