The cache discount has a floor

One of our extraction pipelines runs the same system prompt in three languages. In June the per-language cost report stopped making sense: the Korean and Vietnamese calls were billed at roughly a tenth of the English calls, for prompts that say the same thing to the same model. Nothing was broken. English was not caching, and the other two were.

The rule nobody reads

Prefix caching is gated. A prefix earns the cached rate only after it passes a minimum token count, and below that floor you pay full price forever, no matter how many times you send the identical bytes. The floors are published and they are all denominated in tokens.

provider	floor	block	cache read
OpenAI	1,024 to 2,048 by model	128 tokens	0.1×
Anthropic	512 / 1,024 / 2,048 / 4,096 by model	breakpoints	0.1×, opt-in
Gemini	2,048, newer models 4,096	n/a	0.1×, implicit

The block size is the second half of the rule. Caching moves in whole 128-token blocks, so the remainder is re-billed at full price on every single call:

eligible = prompt_tokens >= FLOOR          // 1024 on our models
cached   = Math.floor(prompt_tokens / 128) * 128
full     = prompt_tokens - cached + tail   // paid every call, forever

Tokens are not language neutral

The same meaning costs a different number of tokens in different languages, and English is almost always the cheapest. Everyone quotes that as a fairness problem for other languages, and at depth it is. But run it against a floor and the sign flips: because English is the least inflated, English is the last language to reach the floor. There is a band of prompt sizes where every translation of your prompt caches and the original does not.

We confirmed it against live billing rather than trusting the arithmetic. Same content, translated, sent at increasing depth, reading usage.prompt_tokens_details.cached_tokens back off each response.

language	D=20	D=24	D=28	D=33
English	0	0	0	1024
Korean	0	1024	1280	1536
Vietnamese	1024	1152	1280	1536
Thai	1280	1536	1792	2048
Japanese	1024	1280	1536	1792

Cached tokens, not billed dollars, so there is nothing to interpret. At D=24 and D=28 English is the only language in the table paying full price. The floor itself is exact: a Korean prefix at 971 tokens does not cache, the same prefix at 1,157 tokens does. And every eligible cell matched floor(p / 128) * 128 with zero error, which means the block rule is not a rounding heuristic, it is the billing.

What we changed the same afternoon

• Cache hit rate is now a per-language metric. The aggregate looked healthy the whole time, because two languages out of three were pulling it up. An average across languages hides exactly the failure it is supposed to surface.
• Every production prompt logs its length against the floor. Not its length. Its distance from the floor, signed.
• Below-floor English is an alert, not a note. It is the cheapest bug we have ever fixed and it was invisible from the invoice, which only says the total went up.

The obvious fix is to pad the short prompt until it clears the floor. That felt like cheating, and cheating usually costs something, so we measured what it costs before shipping it. That is the next note.