BigCode Evaluation Harness

by Orchestra-Research

BigCode Evaluation Harness is a Quality skill for Claude Code, published by Orchestra-Research in AI-Research-SKILLs.

11.6K stars844 forkson Orchestra-Research/AI-Research-SKILLsAdded 2026/08/12+1% in starsRepository updated 2026/06/16

View source on GitHub Download repo ZIP

aiai-researchclaudeclaude-codeclaude-skillscodexgeminigpt-5grpohuggingfacemachine-leanringmegatronskillsvllm

Install in seconds

Install BigCode Evaluation Harness

Copy BigCode Evaluation Harness into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.

terminal

npx degit https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/11-evaluation/bigcode-evaluation-harness ~/.claude/skills/bigcode-evaluation-harness

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/Orchestra-Research/AI-Research-SKILLs.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file: 11-evaluation/bigcode-evaluation-harness/SKILL.md in Orchestra-Research/AI-Research-SKILLs
Installs to: ~/.claude/skills/bigcode-evaluation-harness
Collection: One of 50 skills cataloged from this repository
Category: Quality — 1354 skills

What BigCode Evaluation Harness does

BigCode Evaluation Harness evaluates code generation models on HumanEval, MBPP, MultiPL-E, and other benchmarks with pass@k metrics. Use it to benchmark coding ability, compare models, and measure multi-language code generation quality.

BigCode Evaluation Harness is cataloged under Quality on DirSkills. BigCode Evaluation Harness comes from a repository tagged ai, ai-research, claude, claude-code and claude-skills.

Documentation

README

Excerpt

BigCode Evaluation Harness - Code Model Benchmarking

Quick Start

BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).

Installation:

git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config

Evaluate on HumanEval:

accelerate launch main.py \
  --model bigcode/starcoder2-7b \
  --tasks humaneval \
  --max_length_generation 512 \
  --temperature 0.2 \
  --n_samples 20 \
  --batch_size 10 \
  --allow_code_execution \
  --save_generations

View available tasks:

python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"

Common Workflows

This is the opening of the README. Read the full README on GitHub.

Frequently asked about BigCode Evaluation Harness

What else does Orchestra-Research publish alongside BigCode Evaluation Harness?
BigCode Evaluation Harness is one of 50 skills that DirSkills catalogs from Orchestra-Research/AI-Research-SKILLs, the repository it ships in. Its siblings there include AWQ Quantization, Autoresearch and Axolotl. Each one is a separate skill with its own page in this directory, installs the same way BigCode Evaluation Harness does, and is maintained by Orchestra-Research in that same repository. The rest of the collection is listed on the Orchestra-Research/AI-Research-SKILLs page.
How does BigCode Evaluation Harness compare to other Quality skills?
BigCode Evaluation Harness ranks #172 by stars among the 1354 Quality skills in this catalog. The most-starred ones next to it are Benchmark, Benchmark Optimization Loop and API Design Patterns. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of BigCode Evaluation Harness against them. Open each page to compare what they document and how they install.