--- name: Agent Judge slug: agent-judge category: AI Engineering description: Agent Judge adds judge-level skill installation for agent-based evaluation runs. Use it when the evaluator needs reusable rubrics or domain guidance separate from the skill under test. github: "https://github.com/alibaba/skill-up/tree/main/proposals/0002-agent-judge-specific-skill.md" language: Go stars: 680 forks: 45 install: "npx degit https://github.com/alibaba/skill-up/tree/main/proposals ~/.claude/skills/proposals" installs_to: ~/.claude/skills/proposals source_path: proposals/0002-agent-judge-specific-skill.md collection_size: 23 category_size: 2451 collection_url: "https://dirskills.com/collections/alibaba/skill-up" added: 2026-08-24T05:16:08.832Z last_synced: 2026-08-24T05:16:08.832Z canonical_url: "https://dirskills.com/skills/agent-judge" --- # Agent Judge Agent Judge adds judge-level skill installation for agent-based evaluation runs. Use it when the evaluator needs reusable rubrics or domain guidance separate from the skill under test. **Install:** ```bash npx degit https://github.com/alibaba/skill-up/tree/main/proposals ~/.claude/skills/proposals ``` ## README # SUP-0002: agent_judge Judge-Specific Skill Support Language: English | [中文](zh/0002-agent-judge-specific-skill.md) - [Summary](#summary) - [Motivation](#motivation) - [Goals](#goals) - [Non-Goals](#non-goals) - [Requirements](#requirements) - [Proposal](#proposal) - [User Scenario Quick Reference](#user-scenario-quick-reference) - [Notes, Constraints, and Caveats](#notes-constraints-and-caveats) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details](#design-details) - [Configuration Schema](#configuration-schema) - [Configuration Merge Semantics](#configuration-merge-semantics) - [Path Resolution and Install Target](#path-resolution-and-install-target) - [Agent Installation Adaptation and Progressive Loading](#agent-installation-adaptation-and-progressive-loading) - [Mandatory Use Semantics](#mandatory-use-semantics) - [Evaluator Execution Flow](#evaluator-execution-flow) - [Isolation Semantics](#isolation-semantics) - [Report Metadata](#report-metadata) - [Documentation and Template Updates](#documentation-and-template-updates) - [Test Plan](#test-plan) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) - [Infrastructure Needed](#infrastructure-needed) - [Upgrade & Migration Strategy](#upgrade--migration-strategy) ## Summary `agent_judge` can currently describe grading standards only through `judge.criteria`; it cannot install a Skill dedicated to the judge agent. For evaluations that need domain knowledge, reusable rubrics, strict output formats, or long-lived grading guidance, this forces authors to place large amounts of judging logic into every case YAML file. It also bypasses each Agent Engine's native Skill progressive-loading mechanism, increasing the risk of filling the context window with large rubric documents. This proposal, based on [GitHub issue #134](https://github.com/alibaba/skill-up/issues/134), adds a `judge.skills` configuration so `agent_judge` can install one or more judge-specific Skills through the selected Agent adapter's native Skill installation path. The feature keeps judge Skills isolated from the Skill under test, preserves existing `judge.criteria`, benchmark `with_skill` / `without_skill`, and Agent execution semantics, and records the configured judge Skill information in reports. ## Motivation `agent_judge` is useful when semantic understanding is required, but today's configuration surface is limited to natural-language criteria, model selection, pass threshold, and timeout. Eval authors commonly run into these problems: 1. **Rubrics are too long for YAML**: domain rules, style guides, scoring details, and negative examples are awkward to maintain inline. 2. **Judging logic needs reuse**: when many evals or cases share the same rubric, duplicated criteria drift over time. 3. **The judge agent needs different context**: the run agent should install the Skill under test, while the judge agent may need a separate Skill that teaches it how to grade. 4. **Judge prompts should evolve independently**: a judge Skill can version and refine its rubric without editing every case file. 5. **Isolation matters**: judge helper Skills must not leak into the run agent, especially in benchmark mode where they would pollute the measured capability. 6. **Progressive loading is the point**: the goal is not to concatenate `SKILL.md`, `references/`, and `assets/` into the judge prompt. The goal is to rely on the Agent's own Skill discovery, selection, and on-demand loading behavior so long rubrics enter context only when needed. 7. **Reports need auditability**: results should say which judge Skills were configured, otherwise reviewers cannot understand the grading basis or reproduce the judge environment. The current code path exposes the gap: - `config.JudgeConfig` contains fields such as `type`, `model`, `criteria`, `pass_threshold`, and `timeout_seconds`, but no judge-level Skill reference. - `defaultEvaluator.setupCaseEnvironment()` installs `evalCfg.Skills` only for the main run agent. - `agent_judge` later creates or resolves a judge agent in `resolveJudgeAgent()`, wraps it through `judge.NewJudge()`, and `AgentJudge.Evaluate()` calls `judgeAgent.Run()`. - There is no judge-phase step that calls `judgeAgent.InstallSkill(...)`. ### Goals 1. **Support judge-level Skill configuration**: add a `skills` field to `JudgeConfig`, reusing existing `SkillRef` semantics. 2. **Apply only to `agent_judge`**: `judge.skills` is meaningful only when `judge.type: agent_judge`. 3. **Install into the judge agent**: install `judge.skills` into the judge agent runtime before `AgentJudge.Evaluate()` calls `Run()`. 4. **Keep run and judge agents isolated**: `judge.skills` is not installed into the main run agent, and `eval.skills` is not automatically installed into the judge agent. 5. **Preserve benchmark semantics**: `judge.skills` is installed for both `with_skill` and `without_skill` because it is grading tooling, not the Skill under test. 6. **Remain backward compatible**: existing `agent_judge` configurations behave the same when `judge.skills` is absent. 7. **Preserve native Agent Skill mechanisms**: different Agent Engines may have different Skill directories, manifests, indexes, and discovery mechanisms. skill-up should trigger installation through the Agent adapter's `InstallSkill` abstraction, not inject Skill documents into prompts. 8. **Require judge Skill usage**: when users configure `judge.skills` for `agent_judge`, the judge prompt must explicitly instruct the judge agent to use the installed Skills as authoritative grading guidance. Installation is necessary but not sufficient. 9. **Make reports auditable**: report the judge Skill metadata used for each judged result. 10. **Cover tests and docs**: add coverage for config loading, validation, merging, installation isolation, mandatory usage prompt behavior, report metadata, and documentation examples. ### Non-Goals 1. **No new Skill package manager**: this proposal reuses `SkillRef` / `runtime.SkillConfig` and does not design registry download, version locking, or dependency resolution. 2. **No change to main run Skill installation**: `eval.skills` continues to mean Skills needed by the run agent or the Skill under test. 3. **No rewrite of the `AgentJudge` scoring protocol**: this phase continues to use criteria-driven JSON result parsing. 4. **Judge Skills do not define the criteria list by themselves**: Skills may provide detailed rubrics and constraints, but structured scoring dimensions still come from `judge.criteria`. 5. **No Skill installation for non-`agent_judge` judges**: `rule_based` and `script` judges do not read `judge.skills`. 6. **No prompt-concatenation fallback**: if an Agent adapter cannot install Skills, this proposal does not allow reading Skill files and appending their contents to the judge prompt as a substitute. ## Requirements ### Must Have | ID | Requirement | Acceptance Criteria | | --- | --- | --- | | R1 | `JudgeConfig` supports `skills` | YAML can declare a `skills` array under `judge:` and load it into config | | R2 | Only `agent_judge` can use it | `rule_based` / `script` with `judge.skills` fails validation with a clear error | | R3 | Install before judging | Configured judge Skills are installed before `AgentJudge.Evaluate()` invokes the judge agent | | R4 | Installation isolation | The run agent receives only `eval.skills`; the judge agent receives `judge.skills` | | R5 | Benchmark does not suppress judge Skills | `without_skill` skips `eval.skills` but still installs `judge.skills` | | R6 | Consistent path resolution | Local judge Skill paths resolve relative to the Skill root, consistent with `eval.skills` | | R7 | Backward compatibility | Existing configs require no changes and existing `judge.criteria` behavior remains | | R8 | Native Skill progressive loading | Implementation calls the judge Agent adapter's `InstallSkill`; it must not read Skill docs and concatenate them into the judge prompt | | R9 | Report judge Skills | JSON/HTML reports show the judge Skill list; JUnit exposes it at least through properties | | R10 | Mandatory judge Skill use | When `judge.skills` is non-empty, the prompt sent by `AgentJudge` must include a mandatory-use instruction and Skill identifiers | ### Should Have | ID | Requirement | Acceptance Criteria | | --- | --- | --- | | S1 | Multiple judge Skills | `judge.skills` can declare multiple Skills, installed in configuration order | | S2 | Case-level override | Case-level `judge.skills` can override eval-level judge configuration | | S3 | Clear diagnostics | Installation failures include the judge Skill path and judge-phase context | | S4 | Documentation updates | English/Chinese writing-evals docs and skill-upper references include examples | | S5 | Document Agent differences | Docs explain that judge Skill installation depends on each adapter's Skill support and does not guarantee identical behavior across Agents | ### Nice to Have | ID | Requirement | Acceptance Criteria | | --- | --- | --- | | N1 | Richer report metadata | Reports may include judge Skill digest, target, and install status without exposing sensitive absolute paths | | N2 | Future skill-only judge compatibility | Future work can allow a judge Skill to provide default criteria without breaking this design | ## Proposal Add `judge.skills`, using the same plural form as top-level `skills`: ```yaml judge: type: agent_judge model: anthropic/claude-sonnet-4-6 skills: - source: local_path path: evals/fixtures/judge-skill criteria: - "The answer is correct according to the rubric in the installed judge Skill" ``` High-level flow: 1. Parse `judge.skills` from `eval.yaml` and case YAML. 2. Validate that `judge.skills` is used only with `agent_judge`. 3. Execute the case as usual: prepare runtime, install the run agent, install MCP, and install `eval.skills` for the run agent. 4. Run the main agent and collect `SessionResult`, workspace diff, transcript, and other judge inputs. 5. Enter the judge phase and resolve or create the judge agent. 6. Install merged `judge.skills` into the judge agent through that Agent adapter's `InstallSkill`. 7. When building the judge prompt, if `judge.skills` is non-empty, add an instruction that requires the judge agent to use the installed judge Skills. Do not grade as ordinary criteria-only `agent_judge`. 8. Run the judge agent and parse the structured JSON grading result. 9. Write judge Skill metadata into reports. ``` Case Runtime setupCaseEnvironment - install run agent - install MCP - install eval.skills --------------+ | Main Run Agent runs Skill under test | transcript/diff/output | judge phase | - resolve judge agent | - install judge.skills -----+ | - AgentJudge.Evaluate | | v v Judge Agent grades with installed judge Skill + criteria ``` ### User Scenario Quick Reference #### Scenario 1: Reusable Domain Rubric ```yaml schema_version: v1alpha1 skills: - source: local_path path: . judge: type: agent_judge model: anthropic/claude-sonnet-4-6 skills: - source: local_path path: evals/fixtures/sql-judge-skill criteria: - "The SQL change satisfies the safety and compatibility rules defined by the judge Skill" - "The grading decision cites concrete evidence rather than generic opinions" ``` The run agent installs the Skill under test. The judge agent installs `sql-judge-skill`, which contains database review rules, counterexamples, and output requirements. #### Scenario 2: Case-Level Judge Skill Override ```yaml # evals/eval.yaml judge: type: agent_judge model: anthropic/claude-sonnet-4-6 skills: - source: local_path path: evals/fixtures/default-judge-skill criteria: - "Grade output quality according to the default judge Skill" ``` ```yaml # evals/cases/security-review.yaml judge: type: agent_judge skills: - source: local_path path: evals/fixtures/security-judge-skill criteria: - "Grade whether the answer identifies high-risk issues according to the security judge Skill" ``` When a case declares its own `judge.type`, the case-level judge config is treated as a complete judge strategy: `skills` and `criteria` come from the case, while `model`, `pass_threshold`, and `timeout_seconds` can still inherit from global defaults. #### Scenario 3: Stable Grading Tooling in Benchmark Mode ```yaml benchmark: enabled: true skills: - source: local_path path: . judge: type: agent_judge model: anthropic/claude-sonnet-4-6 skills: - source: local_path path: evals/fixtures/judge-rubric criteria: - "Grade whether the output satisfies the acceptance criteria according to judge-rubric" ``` Benchmark execution: - `with_skill`: the run agent installs `eval.skills`, and the judge agent installs `judge.skills`. - `without_skill`: the run agent skips `eval.skills`, and the judge agent still installs `judge.skills`. This compares the effect of the Skill under test, not whether the grading tool exists. ### Notes, Constraints, and Caveats 1. **`criteria` remains required**: in this phase, `judge.criteria` still defines structured scoring dimensions. A judge Skill may contain long rubrics, but YAML keeps at least one criterion so result count and report structure remain deterministic. 2. **Reuse `SkillRef`**: `judge.skills` uses `source`, `path`, and `target`; there is no parallel singular `judge.skill` syntax. 3. **Must rely on Agent Skill mechanisms**: judge Skills are valuable because Agents can discover and load them on demand. Implementation must not read an entire Skill directory into `criteria` or the judge prompt. 4. **Agent installation methods may differ**: Claude Code, Codex, Qoder CLI, and custom Agents may use different Skill locations or indexes. skill-up hands the same `runtime.SkillConfig` to the relevant adapter. 5. **Local paths first**: phase one uses the existing local Skill installation capability. If top-level `skills` later supports registries, `judge.skills` can reuse that path. 6. **Installation failure is ERROR**: failed judge Skill installation means the judge environment is not ready; mark the case as ERROR, not FAIL. 7. **No silent fallback**: if `judge.skills` is configured but cannot be installed, do not continue with an unskilled judge agent. ### Risks and Mitigations | Risk | Impact | Probability | Mitigation | | --- | --- | --- | --- | | Judge Skill accidentally installs into the run agent | Benchmark results are polluted | Medium | Installation code receives only `judgeAgent`; tests record run/judge installs separately | | `without_skill` skips judge Skills | Baseline cannot be graded by the same rubric | Medium | Judge Skill installation does not depend on `configName` | | Case/global merge semantics are unclear | Authors cannot predict which judge Skill is used | Medium | Reuse existing full override semantics for case-level `judge.type` and document them | | `criteria` conflicts with judge Skill rubric | Grading becomes unstable | Medium | Docs recommend stable criteria dimensions and detailed rubrics in Skills | | Skill path escapes the Skill root | Unexpected local files may be read | Low | Reuse or strengthen path validation so resolved relative paths remain under the Skill root | | Prompt-concatenation fallback for compatibility | Loses progressive loading and can exhaust context | Medium | Explicitly forbid prompt-concatenation fallback; unsupported adapters return ERROR | | Skill discovery differs by Agent | Same judge Skill may behave differently across engines | Medium | Keep adapter-level tests and record engine plus judge Skill metadata in reports | | Reports omit judge Skill info | Grading basis is not auditable | Medium | Add judge Skill metadata to EvalResult/report generation | | Judge Skill is installed but not used | Grading still follows ordinary criteria, ignoring the user-defined rubric | Medium | `AgentJudge` prompt must require use of installed judge Skills; unit tests assert the prompt and fixtures verify behavior | ## Design Details ### Configuration Schema Extend `JudgeConfig` in `internal/config/schema.go`: ```go // JudgeConfig describes the evaluation strategy. type JudgeConfig struct { Type string `json:"type" yaml:"type"` ScriptPath string `json:"script_path,omitempty" yaml:"script_path,omitempty"` Model string `json:"model,omitempty" yaml:"model,omitempty"` Criteria []string `json:"criteria,omitempty" yaml:"criteria,omitempty"` Skills []SkillRef `json:"skills,omitempty" yaml:"skills,omitempty"` PassThreshold *float64 `json:"pass_threshold,omitempty" yaml:"pass_threshold,omitempty"` TimeoutSeconds *int `json:"timeout_seconds,omitempty" yaml:"timeout_seconds,omitempty"` Success []Rule `json:"success,omitempty" yaml:"success,omitempty"` Failure []Rule `json:"failure,omitempty" yaml:"failure,omitempty"` } ``` Validation rules: 1. If `judge.skills` is non-empty, `judge.type` must be `agent_judge`. 2. Each `judge.skills[*].source` should currently be `local_path` or another source already supported by top-level `skills`. 3. For `source: local_path`, `path` is required and must not be blank. 4. `target` is optional and follows top-level `skills[*].target` semantics. 5. `agent_judge` still requires `model` and at least one `criteria` entry unless a later proposal changes the `AgentJudge` protocol. 6. Checks that depend on inherited judge defaults, especially the required `agent_judge` `model`, must run against the effective judge config after `judge.MergeJudgeConfig(global, caseLevel)`. Raw case validation may continue to check case-local constraints, but it must not reject a case-level `agent_judge` only because `model` is inherited from the global judge config. ### Configuration Merge Semantics Reuse the current `judge.MergeJudgeConfig(global, caseLevel)` behavior: - If a case does not declare `judge.type`, use the global judge config, including global `judge.skills`. - If a case declares `judge.type`, treat the case judge config as a full override. `skills`, `criteria`, `success`, `failure`, and `script_path` come from the case. - `model`, `pass_threshold`, and `timeout_seconds` may continue inheriting from global config because current logic already does this. No implicit append behavior is added for the new field. A judge Skill usually represents a complete grading context; silently merging global and case-level Skills can create rubric conflicts. If an author needs multiple Skills, the case-level `judge.skills` should explicitly list all of them. ### Path Resolution and Install Target Local judge Skill path resolution matches top-level `skills`: ```go skillSourceDir := e.loader.SkillDir() skillSource := filepath.Join(skillSourceDir, judgeSkillRef.Path) skillCfg := runtime.SkillConfig{ Source: skillSource, Target: judgeSkillRef.Target, } ``` Implementation should extract a small shared helper, for example: ```go func resolveSkillConfig(skillDir string, ref config.SkillRef) runtime.SkillConfig { return runtime.SkillConfig{ Source: filepath.Join(skillDir, ref.Path), Target: ref.Target, } } ``` Notes: - Relative paths are resolved from the Skill root, not the current working directory or `eval.yaml` directory. - Installation order follows the `judge.skills` array order. - Error messages should include `judge.skills[i].path`. ### Agent