The multilingual cache tax

Two notes on this blog, one about a cache floor English reaches last and one about padding a prompt to clear it, turned out to be two faces of a single inequality. That was worth more than a blog post, so we spent August doing it properly: twelve languages, four tokenizer families, live billing validation, a controlled quality experiment, and a preprint with everything needed to reproduce it.

The Multilingual Cache Tax, INTFRAME Research, DOI 10.5281/zenodo.22081426. Paper CC BY 4.0, code MIT, reproduction package included.

The claim, in one line

A cache floor denominated in tokens turns cross-lingual token inflation into a cost difference that has nothing to do with the service delivered. At depth, caching preserves the familiar inflation gap because the discount scales every language by the same factor. Near the floor the sign flips and the least inflated language, usually English, is the one left paying full price. Which side of the tax you pay depends only on where your prefix sits relative to a number in a pricing table.

The part that made it a paper

Write i for the inflation of a language pair and r for the cached-rate multiplier. The near-floor penalty exists only when r · i < 1. Padding out of it is profitable under exactly the same condition. The trap and the escape hatch are the same inequality, which means escapability is structural rather than a lucky parameter setting.

There is a history in that: at the 0.5× cache rate of 2024, a high-inflation pair like Thai under the older tokenizer (4.4×) had r · i = 2.2 and no near-floor penalty at all. The industry move to 0.1× is what created this artifact, and the same move guaranteed a way out of it.

Measuring the inputs where they live

Benchmark prose overstates what a real service pays. Production system prompts are mostly language-neutral scaffolding, JSON schemas, tags, format rules, and nobody translates those. In our corpus scaffolding is 32% to 66% of the English tokens, median 54%. Localize only the instruction core, as real deployments do, and effective inflation drops sharply.

language	benchmark prose	real system prompt	excess over English, reduced by
Korean	1.47×	1.26×	45%
Thai	1.97×	1.48×	50%
Chinese	1.25×	1.12×	52%

Every dollar figure in the paper uses the conservative real-prompt numbers, not the benchmark ones. A service reusing one system prompt across a million requests a month, on a mid-tier price sheet: a naive 900-token English prompt sits below the floor and costs about $1,880 in prompt spend, the Korean version at 1,134 tokens caches and costs about $505, so the unaware English deployment pays roughly 3.7× more for the same prefix. Padding the English prompt to the floor takes it to about $285, the cheapest of the three. Deeper in, where both cache, the durable inflation gap is about $1,960 a year and no padding removes it.

What we got wrong

Our first draft argued the near-floor penalty was a structural cost. It is not, and the padding experiment we ran to defend that claim is what refuted it. We withdrew the strong version, republished, and the retraction is stated in the record rather than quietly edited out.

We also cut an entire results section. We had a latency axis, and single-client time-to-first-token was too noisy to support the per-language claim we wanted to make. The measurement script stays in the reproduction package, labelled exploratory and not in the paper. Deleting your own favourite finding is cheaper before publication than after.

And the scope is narrower than the headline: live billing validation is OpenAI only, the cross-provider figures are modelled from published floors and public tokenizers, cache-write premiums and TTL expiry sit outside the cost model, and the padding result covers four short machine-gradable task groups. All of it is in the threats-to-validity section, because a paper that only lists its strengths is marketing.

The fix is one line of pricing policy

The floor exists for a reason: each cached prefix occupies KV-cache memory, and a floor keeps that memory from filling with tiny high-churn entries. That rationale is denominated in tokens because the storage is, and it ranks the candidate fixes. Cache aligned blocks below the floor as well, keeping the granularity the storage already has. Or scale the floor per tokenizer so it means the same amount of content in every language. Denominating it in characters, the fix that sounds simplest, is the weakest of the three because it decouples the eligibility rule from the thing being metered.

Until one of those happens, the operator-side version is three lines in a deploy check: know your floor, measure cache hit rate per language, and never leave a prompt sitting in the band between the floor and one tenth of it.