---
name: Model Selection
slug: model-selection
category: AI Engineering
description: Helps select the right LLM for a task by evaluating capability, cost, latency, and routing strategies. Use when choosing a model for a new feature, reducing costs, or evaluating model migrations.
github: "https://github.com/nimadorostkar/Claude-Skills-collection/tree/main/skills/ai/model-selection"
language: Python
stars: 18
forks: 2
install: "git clone https://github.com/nimadorostkar/Claude-Skills-collection"
added: 2026-07-16T13:43:19.939Z
last_synced: 2026-07-18T04:41:53.105Z
canonical_url: "https://dirskills.com/skills/model-selection"
---

# Model Selection

Helps select the right LLM for a task by evaluating capability, cost, latency, and routing strategies. Use when choosing a model for a new feature, reducing costs, or evaluating model migrations.

**Install:** `git clone https://github.com/nimadorostkar/Claude-Skills-collection`

## README

# Model Selection

## Purpose

Choose the model that meets the task's requirements at the lowest cost and latency. Most production LLM features run on a model several times more expensive than the task requires, because nobody measured the cheaper one.

## When to Use

- Choosing a model for a new feature.
- Reducing the cost or latency of an existing feature.
- Evaluating whether a newly released model is worth migrating to.
- Designing a routing strategy across several models.

## Capabilities

- Task-to-capability matching.
- Cost and latency modeling at real volume.
- Routing: cheap model first, escalate on difficulty.
- Benchmark interpretation and its limits.
- Migration and re-evaluation.

## Inputs

- The task, and the accuracy it genuinely requires.
- Volume, latency budget, and cost budget.
- An evaluation set — without one, this is guesswork.

## Outputs

- A model choice justified by measurement on your data.
- A cost and latency projection at real volume.
- A routing strategy, where one is warranted.

## Workflow

1. **Start with the cheapest plausible model** — Not the best one. Measure it on your evaluation set. Escalate only if it fails, and only as far as necessary.
2. **Measure on your data, not on benchmarks** — A model that leads on MMLU may be worse at your specific extraction task. Public benchmarks measure general capability, and your task is not general.
3. **Model the cost at real volume** — A per-call cost difference that looks trivial becomes the dominant line item at a million calls a month. Do the arithmetic before choosing.
4. **Consider routing** — Send everything to a small model; escalate the cases it flags as low-confidence to a larger one. This frequently captures most of the accuracy at a fraction of the cost.
5. **Re-evaluate on model updates** — Provider models change under the same name. A prompt tuned six months ago on a model that has since been updated may no longer be optimal.

## Best Practices

- The most common mistake is defaulting to the most capable model for everything. Classification, extraction, and routing tasks are usually solved by a small model at a tenth of the cost and a fifth of the latency.
- Public benchmarks are contaminated and gamed. They are a rough capability ordering, not a prediction of performance on your task.
- Latency matters more than cost for user-facing features, and cost matters more than latency for batch processing. Optimize for the one that binds.
- A cheaper model with better prompting often beats a more expensive model with worse prompting. Improve the prompt before upgrading the model.
- Test the failure mode, not just the accuracy. A small model that fails loudly is safer than a large one that fails plausibly.
- Do not hard-code the model name across the codebase. Configure it, so migration is a config change rather than a search-and-replace.

## Examples

**Measuring rather than assuming:**

```text
Task: extract structured fields from support emails. 800,000/month.

Model         Accuracy   p95 latency   Cost/1k calls   Monthly cost
-----------   --------   -----------   -------------   ------------
Small          91.2%        340ms          $0.42            $336
Medium         96.8%        890ms          $3.10          $2,480
Large          97.4%       2,100ms        $15.00         $12,000

The large model buys 0.6 accuracy points over the medium for 5x the cost and
2.4x the latency. It is not a serious candidate.

The real question is whether 91.2% is sufficient. It is not — the failures
concentrate on emails with attachments and multi-issue threads.

Routing solution:
  - Small model handles everything, and returns a confidence score.
  - The 14% of cases below the confidence threshold escalate to medium.

  Effective accuracy:  96.5%   (within 0.3 points of medium-for-everything)
  Monthly cost:        $336 + (0.14 x $2,480) = $683
  p95 latency:         410ms   (most requests never touch the medium model)

Chosen: routing. 96.5% accuracy at $683/month, versus 96.8% at $2,480.
```

**Routing implemented:**

```python
async def extract(email: str) -> Extraction:
    result = await small_model.extract(email)

    # Escalate on the model's own uncertainty, or on a structural signal
    # that we know correlates with failure.
    needs_escalation = (
        result.confidence < 0.75
        or email_has_attachment(email)
        or count_distinct_issues(email) > 1
    )

    if needs_escalation:
        metrics.increment("extraction.escalated")
        return await medium_model.extract(email)

    return result
```

## Notes

- The routing threshold should be tuned on the evaluation set by plotting accuracy against escalation rate. There is usually a knee where a small increase in escalation buys most of the remaining accuracy.
- Structural escalation signals (has an attachment, exceeds a length, contains a table) are often better predictors of failure than the model's own confidence score, and they are free to compute.
- When a provider releases a new model, the correct response is to run your evaluation set against it, not to migrate on the announcement. Sometimes the newer model is worse at your specific task.