AI Evaluation Engineering

by devcodex-labs

AI Evaluation Engineering is an AI Engineering skill for Claude Code, published by devcodex-labs in devcodex.

202 stars26 forkson devcodex-labs/devcodexAdded 2026/09/05+16% in starsRepository updated 2026/08/31

View source on GitHub Download repo ZIP

ai-codingclaude-codecodexcoding-agentcursordeveloper-toolsgemini-cligithub-copilotgroklocal-firstworkflow-runtime

Install in seconds

Install AI Evaluation Engineering

Copy AI Evaluation Engineering into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.

terminal

npx degit https://github.com/devcodex-labs/devcodex/tree/main/content/skills/ai-evaluation-engineering ~/.claude/skills/ai-evaluation-engineering

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/devcodex-labs/devcodex.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file: content/skills/ai-evaluation-engineering/SKILL.md in devcodex-labs/devcodex
Installs to: ~/.claude/skills/ai-evaluation-engineering
Collection: One of 25 skills cataloged from this repository
Category: AI Engineering — 3101 skills

What AI Evaluation Engineering does

AI Evaluation Engineering designs evidence for model and prompt quality using golden sets, rubrics, judge calibration, repeat sampling, variance reporting, and regression decisions. Use it when comparing models, prompts, or upgrades and you need reproducible results with contamination checks.

AI Evaluation Engineering is cataloged under AI Engineering on DirSkills. AI Evaluation Engineering comes from a repository tagged ai-coding, claude-code, codex, coding-agent and cursor.

Documentation

README

Excerpt

字段	要求
evaluationDatasetManifest	来源、版本、许可/隐私、任务分层、难例、污染风险和 split
goldenCaseSet	输入、期望属性/答案、允许变体、失败标签和维护 owner
metricRubric	deterministic/semantic/human 指标、权重、阈值和不可聚合项
judgeCalibration	Judge 模型/Prompt/版本、盲测、与人工一致性、偏差和漂移
samplingProtocol	temperature/seed、重复次数、置信区间、停止规则和失败重试
varianceReport	均值、分布、尾部失败、跨 run/provider 差异和不确定性
costLatencyQualityFrontier	token/费用/延迟/成功率/质量的 Pareto 权衡
regressionDecision	baseline/candidate、显著性、阻断阈值、例外和 rollback

执行流程

This is the opening of the README. Read the full README on GitHub.

Frequently asked about AI Evaluation Engineering

What else does devcodex-labs publish alongside AI Evaluation Engineering?
AI Evaluation Engineering is one of 25 skills that DirSkills catalogs from devcodex-labs/devcodex, the repository it ships in. Its siblings there include AI Agent System Architecture, API Contract Architecture and Accessibility I18n. Each one is a separate skill with its own page in this directory, installs the same way AI Evaluation Engineering does, and is maintained by devcodex-labs in that same repository. The rest of the collection is listed on the devcodex-labs/devcodex page.
How does AI Evaluation Engineering compare to other AI Engineering skills?
AI Evaluation Engineering ranks #2826 by stars among the 3101 AI Engineering skills in this catalog. The most-starred ones next to it are Architecture Decision Records, AI-First Engineering and Agentic OS. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of AI Evaluation Engineering against them. Open each page to compare what they document and how they install.