---
name: DeepSwarm
slug: deepswarm
category: AI Engineering
description: Spawn parallel AI workers for any long-running batch API task. Auto-calculates optimal worker count and stagger, with tiered delegation to reduce costs.
github: "https://github.com/amanning3390/deepswarm"
language: Python
stars: 122
forks: 4
install: "git clone https://github.com/amanning3390/deepswarm"
added: 2026-07-16T03:45:41.640Z
last_synced: 2026-07-16T03:45:41.640Z
canonical_url: "https://dirskills.com/skills/deepswarm"
---

# DeepSwarm

Spawn parallel AI workers for any long-running batch API task. Auto-calculates optimal worker count and stagger, with tiered delegation to reduce costs.

**Install:** `git clone https://github.com/amanning3390/deepswarm`

## README

# DeepSwarm 2.0 — Task-Agnostic Parallel Worker Orchestration

Spawn N parallel API workers for **any** batch task. Auto-optimizes worker count + stagger. Tiered model delegation: orchestrator plans (V4 Pro) → workers execute (V4 Flash). 99.95% API success rate.

## Install

```bash
hermes skills tap add amanning3390/deepswarm
```

## Quick Start

```bash
# 1. Define your task
cp task.yaml my_task.yaml
# Edit: prompt_template, worker_model, max_tokens

# 2. Generate seeds
python3 scripts/seed.py --count 1000 --template "Generate {{seed}}" > seeds.jsonl

# 3. Launch (auto-optimizes everything)
export DEEPSEEK_API_KEY=sk-...
python3 scripts/swarm.py --task my_task.yaml --total 1000
```

## Tiered Delegation

```yaml
orchestrator_model: deepseek-v4-pro   # Plans (few calls, frontier quality)
worker_model: deepseek-v4-flash       # Executes (many calls, cheaper)
```

V4 Pro costs ~3× V4 Flash per token. For batch tasks with thousands of calls, tiered delegation saves 60-70%.

## Auto-Optimization

| Call Duration | Workers | Stagger | Success | Throughput |
|--------------|---------|---------|---------|------------|
| <10s | 16 | 1s | 99.9% | ~5,760/hr |
| 10-30s | 12 | 2s | 99.9% | ~1,440/hr |
| 30-60s | 8 | 5s | 99.95% | ~440/hr |
| 60-90s | 6 | 10s | 99.9% | ~240/hr |

Omit `workers` and `stagger` in task.yaml — DeepSwarm runs a calibration call and picks optimal values.

## Task Types

Built-in: `generation`, `translation`, `summarization`, `classification`, `custom`

For multi-turn tasks (tool calling, conversation loops):
```yaml
multi_turn: true
max_turns: 20
```

## Files

```
deepswarm/
├── SKILL.md              # Hermes skill definition
├── README.md
├── task.yaml             # Sample task config
├── architecture.html     # Pipeline diagram
├── scripts/
│   ├── seed.py           # Seed generator (simple + rich templates)
│   ├── swarm.py          # Orchestrator (auto-optimize + launch)
│   ├── worker.py         # Task-agnostic batch processor
│   └── filter.py         # Quality filter with JSON repair
├── templates/
│   └── prompts.py        # v2 prompt templates
└── references/
    ├── api-rate-limits.md
    └── generation-patterns.md
```

## Post-Processing

After generation, filter the raw output to remove malformed traces:

```bash
python3 scripts/filter.py --input-dir output/ --output clean.jsonl --errors errors.jsonl
```

The filter applies 3 passes: JSON repair (fixes 17% API error rate), structural validation
(think blocks, tag balance), and length thresholds. Typically lifts pass rate from ~28% to ~62%.

## Provenance

Built from the [DeepSeek Hermes Reasoning Traces](https://huggingface.co/datasets/ning423/deepseek-hermes-reasoning-traces) project:
- 19,331 traces · 192K tool calls
- 96 workers · 31K API calls
- 99.95% success rate
- 8 workers + 5s stagger = the magic formula