AI Evals

by RefoundAI

AI Evals is an AI Engineering skill for Claude Code, published by RefoundAI in lenny-skills.

1.3K stars161 forkson RefoundAI/lenny-skillsAdded 2026/08/20Repository updated 2026/07/16

View source on GitHub Download repo ZIP

ai-agentsai-assistantclaudeclaude-codelenny-rachitskyllmpm-toolsproduct-managementproduct-skillsskills

Install in seconds

Install AI Evals

Copy AI Evals into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.

terminal

npx degit https://github.com/RefoundAI/lenny-skills/tree/main/skills/ai-evals ~/.claude/skills/ai-evals

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/RefoundAI/lenny-skills.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file: skills/ai-evals/SKILL.md in RefoundAI/lenny-skills
Installs to: ~/.claude/skills/ai-evals
Collection: One of 25 skills cataloged from this repository
Category: AI Engineering — 2451 skills

What AI Evals does

AI Evals helps teams build robust evaluation infrastructure for measuring, monitoring, and iterating on AI product performance using human, code-based, and LLM-as-a-judge methods.

AI Evals is cataloged under AI Engineering on DirSkills. AI Evals comes from a repository tagged ai-agents, ai-assistant, claude, claude-code and lenny-rachitsky.

Documentation

README

Excerpt

AI Evaluation Strategy

Move beyond vibe checks to systematic, empirical measurement of AI product quality and reliability.

Help the user with ai evaluation strategy using insights from 11 guests and posts across Lenny's Podcast and Newsletter.

How to Help

Identify Failure Modes - Help the user conduct error analysis on real traces to find where the system specifically breaks.
Select Eval Methods - Recommend the right mix of human, code, and LLM judges based on the specific technical use case.
Build Gold Sets - Assist in curating a reference dataset of high-quality examples to act as the ground truth for your application.
Operationalize - Guide the user in integrating these evaluations into a CI/CD pipeline for continuous quality improvement.

This is the opening of the README. Read the full README on GitHub.

Frequently asked about AI Evals

What else does RefoundAI publish alongside AI Evals?
AI Evals is one of 25 skills that DirSkills catalogs from RefoundAI/lenny-skills, the repository it ships in. Its siblings there include AI Product Strategy, AI-Assisted Prototyping and AI-Native UX Design. Each one is a separate skill with its own page in this directory, installs the same way AI Evals does, and is maintained by RefoundAI in that same repository. The rest of the collection is listed on the RefoundAI/lenny-skills page.
How does AI Evals compare to other AI Engineering skills?
AI Evals ranks #1126 by stars among the 2451 AI Engineering skills in this catalog. The most-starred ones next to it are Architecture Decision Records, AI-First Engineering and Agentic OS. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of AI Evals against them. Open each page to compare what they document and how they install.