Small models where they win

The question "which model should we use" is usually malformed. For classification-shaped work, the answer is two models and a threshold.

The pattern: label big, serve small

When a pipeline needs to classify at volume, categories, spam, intent, language, we do not call a frontier model per item. We call it once per dataset: have the big model label 30,000 historical examples with a carefully engineered rubric, spot-check a sample by hand, then distill those labels into a small model that runs on our own box. The frontier model's judgment gets compiled into something that serves at 15 ms for the price of electricity.

The router handles the hard tail

const p = small.classify(x);            // ~15 ms, on-box, effectively free
if (p.confidence >= 0.92) return p.label;
return frontier.classify(x);            // ~1.5 s, the 7% that deserve it

On our distribution, 93% of items clear the threshold locally and agreement with the frontier model on that slice is above 99%. The ambiguous 7% get the expensive brain. Blended cost lands two orders of magnitude below calling the frontier model on everything, and p50 latency drops from seconds to milliseconds, which matters when the classifier sits inside a user-facing request.

Where fine-tuning actually loses

We keep re-testing fine-tuning and it keeps losing to prompting for our shapes of work. Style, format and policy compliance are handled better by a versioned prompt with few-shot examples riding a warm prefix cache: changes deploy in minutes, roll back in seconds, and A/B cleanly. A fine-tune is a build artifact with a days-long iteration loop, and it goes stale every time the base model improves, which lately is often. The distillation pattern above is the exception that earns it, and even there we distill into small open-weight models we host, so the artifact is ours.

The drift tax

The distilled model is frozen; the world is not. So the router double-writes a 1% shadow sample to the frontier model continuously, and disagreement above a threshold pages us to re-label and re-distill. Small models are cheap to run and easy to forget. The shadow sample is what keeps "we automated this last spring" from decaying into "we have been silently misclassifying since spring".