Paper Autoraters

Paper Autoraters is an AI Engineering skill for Claude Code, published by Ar9av in PaperOrchestra.

644 stars91 forkson Ar9av/PaperOrchestraAdded 2026/08/24+1% in starsRepository updated 2026/08/09

agentic-aiai-researchanthropicantigravityarxivautomated-paper-writingclaude-codeclaude-skillscoding-agentscursorlatexliterature-reviewllm-agentsmulti-agentpaper-orchestrapaperorchestraresearch-papersemantic-scholarskill-pack

Install in seconds

Install Paper Autoraters

Copy Paper Autoraters into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.

terminal

npx degit https://github.com/Ar9av/PaperOrchestra/tree/main/skills/paper-autoraters ~/.claude/skills/paper-autoraters

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/Ar9av/PaperOrchestra.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file: skills/paper-autoraters/SKILL.md in Ar9av/PaperOrchestra
Installs to: ~/.claude/skills/paper-autoraters
Collection: One of 9 skills cataloged from this repository
Category: AI Engineering — 2451 skills

What Paper Autoraters does

Paper Autoraters runs PaperOrchestra’s four LLM-as-judge evaluators for paper quality, citation coverage, and side-by-side comparisons. Use it to score a draft, compare two papers, or validate a paper-writing pipeline.

Paper Autoraters is cataloged under AI Engineering on DirSkills. Paper Autoraters comes from a repository tagged agentic-ai, ai-research, anthropic, antigravity and arxiv.

Documentation

README

Excerpt

Paper Autoraters (App. F.3)

Faithful implementation of the four LLM-as-judge autoraters used in PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).

These are the metrics the paper uses to demonstrate that PaperOrchestra beats single-agent and AI-Scientist-v2 baselines. Use them to:

Score a generated paper against a ground-truth paper.
Compare two paper-writing pipelines side-by-side.
Validate your own host-agent execution of the paper-orchestra pipeline.

The four autoraters

This is the opening of the README. Read the full README on GitHub.

Frequently asked about Paper Autoraters

What else does Ar9av publish alongside Paper Autoraters?
Paper Autoraters is one of 9 skills that DirSkills catalogs from Ar9av/PaperOrchestra, the repository it ships in. Its siblings there include Agent Research Aggregator, Content Refinement Agent and Literature Review Agent. Each one is a separate skill with its own page in this directory, installs the same way Paper Autoraters does, and is maintained by Ar9av in that same repository. The rest of the collection is listed on the Ar9av/PaperOrchestra page.
How does Paper Autoraters compare to other AI Engineering skills?
Paper Autoraters ranks #1864 by stars among the 2451 AI Engineering skills in this catalog. The most-starred ones next to it are Architecture Decision Records, AI-First Engineering and Agentic OS. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of Paper Autoraters against them. Open each page to compare what they document and how they install.