--- name: ArXiv Search slug: arxiv-search-2 category: Data description: ArXiv Search retrieves paper metadata from arXiv keyword queries and writes normalized results to JSONL. Use it to build an initial paper set for ranking, taxonomy, or citation workflows, or to import and enrich offline exports. github: "https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/arxiv-search" language: Python stars: 499 forks: 39 install: "npx degit https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/arxiv-search ~/.claude/skills/arxiv-search" installs_to: ~/.claude/skills/arxiv-search source_path: .codex/skills/arxiv-search/SKILL.md collection_size: 25 category_size: 668 collection_url: "https://dirskills.com/collections/WILLOSCAR/research-units-pipeline-skills" added: 2026-08-26T05:12:51.434Z last_synced: 2026-08-26T05:12:51.434Z canonical_url: "https://dirskills.com/skills/arxiv-search-2" --- # ArXiv Search ArXiv Search retrieves paper metadata from arXiv keyword queries and writes normalized results to JSONL. Use it to build an initial paper set for ranking, taxonomy, or citation workflows, or to import and enrich offline exports. **Install:** ```bash npx degit https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/arxiv-search ~/.claude/skills/arxiv-search ``` ## README # arXiv Search (metadata-first) Collect an initial paper set with enough metadata to support downstream ranking, taxonomy building, and citation generation. When online, prefer rich arXiv metadata (categories, arxiv_id, pdf_url, published/updated, etc.). When offline, accept an export and convert it cleanly. ## Load Order Always read: - `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior Domain packs (loaded by topic match): - `assets/domain_packs/llm_agents.json` — pinned IDs, query rewrite rules for LLM agent topics ## Script Boundary Use `scripts/run.py` only for: - arXiv API retrieval and XML parsing - offline export conversion (CSV/JSON/JSONL normalization) - metadata enrichment via `id_list` backfill Do not treat `run.py` as the place for: - hardcoded topic detection or query rewriting (use domain packs) - domain-specific pinned paper lists (externalize to `assets/domain_packs/`) ## Contract-driven behavior - Domain-pack query rewriting is the default for broad discovery Workflows. - A focused Workflow may set `quality_contract.retrieval_policy.domain_pack_query_mode: explicit`; in that mode, the query list in `queries.md` remains authoritative and the domain pack must not replace its topic focus. - `quality_contract.retrieval_policy.minimum_records` turns a Workflow's raw candidate-pool floor into a strict quality-gate check. ## Input - `queries.md` (keywords, excludes, time window) ## Outputs - `papers/papers_raw.jsonl` (JSONL; 1 paper per line) - Each record includes at least: `title`, `authors`, `year`, `url`, `abstract` - When using the arXiv API online mode, records also include helpful metadata: `arxiv_id`, `pdf_url`, `categories`, `primary_category`, `published`, `updated`, `doi`, `journal_ref`, `comment` - Convenience index (optional but generated by the script): - `papers/papers_raw.csv` ## Decision: online vs offline - If you have network access: run arXiv API retrieval. - If not: import an export the user provides (CSV/JSON/JSONL) and normalize fields. - Hybrid: if you import offline but still have network later, you can **enrich missing fields** (abstract/authors/categories) via arXiv `id_list` using `--enrich-metadata` or `queries.md` `enrich_metadata: true`. ## Workflow (heuristic) 1. Read `queries.md` and expand into concrete query strings. 2. Retrieve results (online) or import an export (offline). 3. Normalize every record to include at least: - `title`, `authors` (array), `year`, `url`, `abstract` 4. Keep the set broad at this stage; dedupe/ranking comes next. 5. Apply time window and `max_results` if specified. ## Quality checklist - [ ] `papers/papers_raw.jsonl` exists. - [ ] Each line is valid JSON and contains `title`, `authors`, `year`, `url`. ## Side effects - Allowed: create/overwrite `papers/papers_raw.jsonl`; append notes to `STATUS.md`. - Not allowed: write prose sections in `output/` before writing is approved. ## Script ### Quick Start - `uv run python .codex/skills/arxiv-search/scripts/run.py --help` - Online: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace --query "" --max-results 200` - Offline import: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace --input ` ### All Options - `--query `: repeatable; multiple queries are unioned - `--exclude `: repeatable; excludes applied after retrieval - `--max-results `: cap total retrieved - `--input `: offline mode (CSV/JSON/JSONL) - `--enrich-metadata`: best-effort enrich via arXiv `id_list` (needs network) - `queries.md` also supports: `keywords`, `exclude`, `time window`, `max_results`, `enrich_metadata` ### Examples - Online (multi-query + excludes): - `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace --query "LLM agent" --query "tool use" --exclude "survey" --max-results 300` - Fetch a single paper by arXiv ID (direct `id_list` fetch): - `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace --query 2509.02547 --max-results 1` - Offline auto-detect (no flags): - Place `papers/import.csv` (or `.json/.jsonl`) under the workspace, then run: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace ` - Offline import + time window (via `queries.md`): - Set `- time window: { from: 2022, to: 2025 }` then run offline import normally ## Troubleshooting ### Common Issues #### Issue: `papers/papers_raw.jsonl` is empty **Symptom**: - Script exits with “No results returned …” or output file is empty. **Causes**: - Network is blocked (online mode). - Queries are too narrow or `queries.md` is empty. **Solutions**: - Use offline import: place `papers/import.csv|json|jsonl` in the workspace or pass `--input`. - Broaden keywords and reduce excludes in `queries.md`. - Run with explicit `--query` to sanity-check the parser. #### Issue: Offline import records miss fields **Symptom**: - Downstream steps fail because records miss `authors/year/abstract/url`. **Causes**: - Export columns don’t match expected fields; upstream export is incomplete. **Solutions**: - Ensure the export contains at least `title`, `authors`, `year`, `url`, `abstract`. - If you later have network, use `--enrich-metadata` to backfill missing fields (best effort). ### Recovery Checklist - [ ] Confirm `queries.md` has non-empty `keywords` (or pass `--query`). - [ ] If offline: confirm workspace has `papers/import.*` and rerun. - [ ] Spot-check 3–5 JSONL lines: valid JSON + required fields.