---
name: Dedupe Rank
slug: dedupe-rank
category: Data
description: Dedupe Rank turns a broad paper candidate pool into a deduped JSONL and a stable core set. Use it when retrieval output needs deterministic deduplication, ranking, and repeatable paper IDs.
github: "https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/dedupe-rank"
language: Python
stars: 499
forks: 39
install: "npx degit https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/dedupe-rank ~/.claude/skills/dedupe-rank"
installs_to: ~/.claude/skills/dedupe-rank
source_path: .codex/skills/dedupe-rank/SKILL.md
collection_size: 25
category_size: 668
collection_url: "https://dirskills.com/collections/WILLOSCAR/research-units-pipeline-skills"
added: 2026-08-26T05:12:55.094Z
last_synced: 2026-08-26T05:12:55.094Z
canonical_url: "https://dirskills.com/skills/dedupe-rank"
---

# Dedupe Rank

Dedupe Rank turns a broad paper candidate pool into a deduped JSONL and a stable core set. Use it when retrieval output needs deterministic deduplication, ranking, and repeatable paper IDs.

**Install:**

```bash
npx degit https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/dedupe-rank ~/.claude/skills/dedupe-rank
```

## README

# Dedupe + Rank

Turns a raw candidate pool into a deduped pool and a stable core set.

## Input

- `papers/papers_raw.jsonl`

## Outputs

- `papers/papers_dedup.jsonl`
- `papers/core_set.csv`

## Script boundary

`scripts/run.py` should own only:
- title/year deduplication
- deterministic ranking
- stable `paper_id` generation

Use shared domain packs or pipeline contract metadata for topic-specific or product-specific behavior.

## Contract-driven behavior

The script should prefer pipeline contract metadata over profile-name branching.

Current important field:
- `quality_contract.candidate_pool_policy.keep_full_deduped_pool`
- `quality_contract.candidate_pool_policy.include_domain_pins`
- `quality_contract.candidate_pool_policy.minimum_domain_surveys`
- `quality_contract.candidate_pool_policy.survey_title_bonus`

If true, the script keeps the full deduped pool in `papers/core_set.csv` unless the user explicitly overrides core size.
Focused briefing Workflows can disable unconditional domain pins, reserve a
smaller survey floor, and remove the generic survey-title bonus without
changing broad Survey defaults.

## Acceptance

- deduped JSONL exists
- core-set CSV exists
- reruns are stable for the same inputs

## Non-goals

- retrieval
- screening
- manual topic authoring inside the script