800 KB to 1.8 KB, and Your Bill Does Not Change

Anthropic put message threads into public beta yesterday. Instead of your application resending the entire conversation on every turn, the API keeps it and you send only what is new. Anthropic reports median request size in long conversations falling from around 800 KB to about 1.8 KB. The number is striking and it is being read as a cost story, which it is not.

What the feature does

The first request sends the full conversation, including the system prompt and tools, and starts a thread. The API stores that conversation and the reply under the response ID. Every later turn sends only the messages since the previous response plus that response ID, and the API reconstructs the rest before calling the model.

In practice it is a parameter and a beta header, with a helper in the official SDKs that manages the header, holds the conversation in memory and replays the whole thing once if a continuation fails. A modest change to adopt, which is part of the appeal.

What moves and what does not

The performance figures come from Anthropic's own Claude Code rollout and have not been independently reproduced. The direction is plausible, because sending 1.8 KB over the wire is obviously faster than sending 800 KB, and the exact magnitude is a vendor claim until somebody outside tests it.

The last line of that table is the one with money attached. Prompt caching is the mechanism that changes what you pay, and threads do not enable it for you.

The detail worth absorbing

Set the feature aside and look at the baseline it replaces. Long agent conversations were re-uploading the full history on every single turn, reaching roughly 800 KB per request.

Most businesses running agents have no idea that is how it works. They picture a conversation that accumulates somewhere, and the reality is that every turn resends everything, which is why an agent that chats for forty turns costs so much more than forty separate questions would.

That re-reading is also why the token bill does not fall here. The upload got smaller; the reading did not. If you have been surprised by an agent invoice, this is usually the explanation, and the modelling for it is in cash flow forecasting when AI costs are variable.

Latency and cost are different problems

Worth being precise about which one you have, because the fixes do not overlap.

A latency problem looks like a person waiting on a screen, a support reply that arrives too slowly to feel live, or a batch job that misses its window. Threads help here, and so do a smaller model, shorter outputs and streaming.

A cost problem looks like an invoice growing faster than usage. Threads do nothing. Prompt caching, fewer calls per run, a cheaper model for the parts that permit it, and capped retries are the levers, as set out in consumption pricing and unpredictable bills.

Adopting a latency fix and expecting a cost result is how an improvement gets recorded as a failure, which is the same accounting trap described in where AI actually moves your gross margin.

Who should take it

Businesses running agents that hold long conversations with a person waiting at the other end. Customer-facing assistants, interactive tools, anything where the pause before the first word is the thing people notice. The change is small and the gain is real.

Businesses running short interactions or batch work overnight get very little, because the upload was never the bottleneck. And anyone whose actual complaint is the monthly invoice should spend the same hour on caching instead.

It is also a public beta, which means rough edges and a fallback path worth keeping. Testing it against your own workload with a baseline, rather than adopting on the strength of a vendor figure, is the method in designing a proof of concept that decides, and knowing how many agents you have running in the first place is a prior question.

Read the next one the same way

Infrastructure improvements get announced with the most dramatic number available, and the most dramatic number is frequently not the one that affects you. A 400-fold reduction in request size is genuinely impressive engineering and leaves your unit economics exactly where they were.

The question to carry into any such announcement is simply which line on your own bill or clock this touches. That habit is the same one applied to pricing changes in the fine print on AI price cuts.

Frequently Asked Questions