Documentation
README
LLM Integration
Purpose
Integrate a language model into a production application, where the API is slow, rate-limited, occasionally down, and billed per token — none of which the quickstart mentions.
When to Use
- Adding an LLM to a production application.
- An LLM feature that is slow, expensive, or unreliable.
- Handling rate limits, streaming, or provider failover.
- Deciding where the model call belongs in the architecture.
Capabilities
- Streaming responses and partial rendering.
- Retry, backoff, and rate-limit handling.
- Timeouts and cancellation.
- Prompt caching and response caching.
- Multi-provider fallback.
- Token accounting and cost control.
Inputs
This is the opening of the README. Read the full README on GitHub.