📚
DataPython

Dedupe Rank

by WILLOSCAR

Dedupe Rank is a Data skill for Claude Code, published by WILLOSCAR in research-units-pipeline-skills.

499 stars39 forkson WILLOSCAR/research-units-pipeline-skillsAdded 2026/08/26Repository updated 2026/08/26
View source on GitHubDownload repo ZIP
claudeclaude-codecodexgptpipelineresearchresearch-paperresearch-projectresearch-toolskillstoolsunitsvibevibe-codingvibecoding
Install in seconds
Install Dedupe Rank
Copy Dedupe Rank into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.
terminal
npx degit https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/dedupe-rank ~/.claude/skills/dedupe-rank

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/WILLOSCAR/research-units-pipeline-skills.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file
.codex/skills/dedupe-rank/SKILL.md in WILLOSCAR/research-units-pipeline-skills
Installs to
~/.claude/skills/dedupe-rank
Collection
One of 25 skills cataloged from this repository
Category
Data — 668 skills

What Dedupe Rank does

Dedupe Rank turns a broad paper candidate pool into a deduped JSONL and a stable core set. Use it when retrieval output needs deterministic deduplication, ranking, and repeatable paper IDs.

Dedupe Rank is cataloged under Data on DirSkills. Dedupe Rank comes from a repository tagged claude, claude-code, codex, gpt and pipeline.

Documentation

README

Dedupe + Rank

Turns a raw candidate pool into a deduped pool and a stable core set.

Input

  • papers/papers_raw.jsonl

Outputs

  • papers/papers_dedup.jsonl
  • papers/core_set.csv

Script boundary

scripts/run.py should own only:

  • title/year deduplication
  • deterministic ranking
  • stable paper_id generation

Use shared domain packs or pipeline contract metadata for topic-specific or product-specific behavior.

Contract-driven behavior

The script should prefer pipeline contract metadata over profile-name branching.

This is the opening of the README. Read the full README on GitHub.

Frequently asked about Dedupe Rank

  • What else does WILLOSCAR publish alongside Dedupe Rank?

    Dedupe Rank is one of 25 skills that DirSkills catalogs from WILLOSCAR/research-units-pipeline-skills, the repository it ships in. Its siblings there include Agent Survey Corpus, Anchor Sheet and Appendix Table Writer. Each one is a separate skill with its own page in this directory, installs the same way Dedupe Rank does, and is maintained by WILLOSCAR in that same repository. The rest of the collection is listed on the WILLOSCAR/research-units-pipeline-skills page.

  • How does Dedupe Rank compare to other Data skills?

    Dedupe Rank ranks #606 by stars among the 668 Data skills in this catalog. The most-starred ones next to it are Benchmark Methodology, Jupyter Notebook and Solana. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of Dedupe Rank against them. Open each page to compare what they document and how they install.

More from WILLOSCAR/research-units-pipeline-skills

Dedupe Rank is one of 25 skills cataloged on DirSkills from WILLOSCAR/research-units-pipeline-skills.

See all 25 skills →
📚
5d ago

Agent Survey Corpus

Agent Survey Corpus downloads open-access arXiv survey PDFs about agentic systems and extracts text for local reference. Use it to study how real surveys organize sections, subsections, and evidence-backed comparisons.
Writing
49939
🪝
5d ago

Anchor Sheet

Anchor Sheet extracts subsection-specific evidence hooks from draft packs, focusing on numbers, benchmarks, metrics, and limitations. Use it when you want sections to be grounded in cited facts instead of generic summaries.
Writing
49939
📋
5d ago

Appendix Table Writer

Appendix Table Writer curates reader-facing survey tables for paper appendices using only in-scope evidence and existing citation keys. Use it when you have evidence packs, anchor sheets, and citations and need publishable tables instead of internal logs.
Writing
49939
📚
5d ago

ArXiv Search

ArXiv Search retrieves paper metadata from arXiv keyword queries and writes normalized results to JSONL. Use it to build an initial paper set for ranking, taxonomy, or citation workflows, or to import and enrich offline exports.
Data
49939
🧾
5d ago

Argument Selfloop

Argument Selfloop builds a final machine-readable snapshot from current H3 sections, refreshes section hashes, and checks consistency before merge. Use it after section edits when you need an updated argument ledger and manifest.
Writing
49939
🧾
5d ago

Artifact Contract Auditor

Artifact Contract Auditor checks whether completed Units and the locked Pipeline have all required output files present, then writes output/CONTRACT_REPORT.md. Use it for mid-run coverage snapshots or final delivery completeness, not provenance checks.
Quality
49939

More Data skills

Dedupe Rank is one of 668 Data skills cataloged on DirSkills, ranked #606 by stars among them.

Browse all Data skills →
📊
3w ago

Benchmark Methodology

Benchmark Methodology scores competitors across nine weighted dimensions using explicit 1–5 rubrics and a tension plot. Use it after competitive-platform-analysis to produce comparable, evidence-anchored profile cards for a competitive report.
Data
239.8K36.4K
📓
3w ago

Jupyter Notebook

Jupyter Notebook provides a stateful Python REPL via a live Jupyter kernel (hamelnb) so variables persist across executions. Use it for iterative data exploration, API inspection, and complex code development.
Data
229.7K45.3K
🪙
3w ago

Solana

Solana queries wallets, token holdings, transactions, NFTs, whale transfers, and network stats on the Solana blockchain with USD pricing via CoinGecko. Use it when you need to inspect an address, verify a transaction, check token metadata or prices, or monitor network health.
Data
229.7K45.3K