Distributed LLM Pretraining with TorchTitan

Distributed LLM Pretraining with TorchTitan is a DevOps skill for Claude Code, published by Orchestra-Research in AI-Research-SKILLs.

11.6K stars838 forkson Orchestra-Research/AI-Research-SKILLsAdded 2026/07/19+1% in starsRepository updated 2026/06/16

View source on GitHub Download repo ZIP

aiai-researchclaudeclaude-codeclaude-skillscodexgeminigpt-5grpohuggingfacemachine-leanringmegatronskillsvllm

Install in seconds

Install Distributed LLM Pretraining with TorchTitan

Copy Distributed LLM Pretraining with TorchTitan into your Claude Code skills folder. Run the command in your terminal, or review the source on GitHub before installing.

terminal

npx degit https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/01-model-architecture/torchtitan ~/.claude/skills/torchtitan

Requires Node.js. Downloads this skill only — not the rest of the repository — into your Claude Code skills folder.

Without Node.js

git clone https://github.com/Orchestra-Research/AI-Research-SKILLs.git

Clones the whole repository, then copy the skill’s own directory into your skills folder yourself.

In this catalog

Source file: 01-model-architecture/torchtitan/SKILL.md in Orchestra-Research/AI-Research-SKILLs
Installs to: ~/.claude/skills/torchtitan
Collection: One of 50 skills cataloged from this repository
Category: DevOps — 1075 skills

What Distributed LLM Pretraining with TorchTitan does

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Distributed LLM Pretraining with TorchTitan is cataloged under DevOps on DirSkills. Distributed LLM Pretraining with TorchTitan comes from a repository tagged ai, ai-research, claude, claude-code and claude-skills.

Documentation

README

Excerpt

TorchTitan - PyTorch Native Distributed LLM Pretraining

Quick start

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.

Installation:

# From PyPI (stable)
pip install torchtitan

# From source (latest features, requires PyTorch nightly)
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt

Download tokenizer:

# Get HF token from https://huggingface.co/settings/tokens
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...

This is the opening of the README. Read the full README on GitHub.

Frequently asked about Distributed LLM Pretraining with TorchTitan

What else does Orchestra-Research publish alongside Distributed LLM Pretraining with TorchTitan?
Distributed LLM Pretraining with TorchTitan is one of 50 skills that DirSkills catalogs from Orchestra-Research/AI-Research-SKILLs, the repository it ships in. Its siblings there include AWQ Quantization, Autoresearch and Axolotl. Each one is a separate skill with its own page in this directory, installs the same way Distributed LLM Pretraining with TorchTitan does, and is maintained by Orchestra-Research in that same repository. The rest of the collection is listed on the Orchestra-Research/AI-Research-SKILLs page.
How does Distributed LLM Pretraining with TorchTitan compare to other DevOps skills?
Distributed LLM Pretraining with TorchTitan ranks #94 by stars among the 1075 DevOps skills in this catalog. The most-starred ones next to it are Backend Patterns, API Connector Builder and Migration. DirSkills ranks by the star count of the repository each skill ships in, so that order reflects how popular those repositories are rather than any review of Distributed LLM Pretraining with TorchTitan against them. Open each page to compare what they document and how they install.