How the model input is cached — and kept fresh
The expensive prefix is reused; the freshest signals stay fresh; junk drops out early — four moving cache points, an always-fresh tail, marked cold turns, pruning, and compaction.
Caching only works on a prefix that is byte-for-byte identical to last time. So the whole strategy comes down to one design choice: put the stable things first, the volatile things last.
In #1 we mapped the shape of the model input: a stable per-agent front (instructions · tool catalog · skill catalog), a timeline that grows oldest → newest, and a volatile tail at the very end. That shape wasn’t an accident. It’s what makes the input cheap to reuse and safe to keep fresh at the same time. This is #2 of three: what happens across calls — how most of the input is reused instead of re-sent, how the freshest signals stay fresh, and how the agent can drop junk early without throwing away the cache. The trick isn’t pretending context never changes. It’s shaping what stays visible.
01Why caching is even possible here
Sending a long input to the model is expensive. The instruction envelope, the catalogs, and the older history don’t change between one round and the next — so re-sending them every time is pure waste. Prompt caching lets the model reuse an exact prefix it has already seen, and only pay full price for what’s new after it. The catch: change one thing near the front and everything after that point has to be re-sent. The expensive part is reused; the freshest part is never cached, so it can change every round without a thought. To make that reuse precise, the runtime drops cache points along the way.
02The tail is never cached — on purpose
The tail is never cached. The sources pool and the live view sit at the very end, after every cache point, and they are rebuilt every round.
This is deliberate. Fast-moving state — the current budget, a live user interrupt, a fresh projection of the workspace — has to be current on every call. If it were baked into the cached prefix, you’d face a bad choice: either churn the expensive prefix every round to update it, or let the agent read stale state. By keeping that state in an uncached tail, you sidestep both. The prefix stays calm; the tail stays fresh. (The tail’s own structure is the subject of #3.) Caching by shaping visibility, not by freezing the input.
03The four cache points
A cache point is a boundary that says “everything up to here is stable; reuse it.” There are four, and where each sits is the whole design.
- Envelope / prefix point — anchors the whole instruction envelope: the runtime and agent/domain instructions plus the tool and skill catalogs. This is the stable per-agent prefix from #1, and the boundary that lets the entire front be reused across calls.
- Previous-turn point — right after the last block of the prior turn. The cross-turn anchor: once a new turn begins, the whole conversation up to here can be reused for the rest of the turn.
- Pre-tail point — the last block of round N−4, a deliberate extra checkpoint set a few rounds back from the newest activity (it appears only once the turn has enough rounds). This is what opens the editable zone.
- Tail point — right after the most recent stable round, the last block in the cacheable stream.
Everything after point 4 — the sources pool and the live view — is the uncached tail; no cache point ever reaches it.
04The extra checkpoint lets the agent forget garbage early
The pre-tail point earns its keep. Without it, the agent’s only safe cache boundary would be at the very edge of activity — no room to drop anything without busting the cache. With the extra checkpoint a few rounds back, there’s a small editable zone between it and the tail. If the agent pulls a huge, useless result — a giant page it shouldn’t have fetched — it can hide that block: the bulky payload leaves the model’s eyes, a short note takes its place, and the original stays recoverable by its path. Because the hide happens below the pre-tail point, the cached prefix above it is untouched.
05Cache points move with the turn
The previous-turn point is fixed once the turn starts. The other two are not — they march forward as rounds accumulate.
Early on there may not be enough rounds to place a pre-tail point at all; later it appears, and both it and the tail point slide downward as the agent works. That’s what you want in a live loop: the cached part keeps growing while the editable frontier stays near the action.
06When the user changes the agent: cold turns, marked and governed
Everything above assumed the front of the input holds still — and between rounds it does. But since per-user agent customization, the user can change the front itself: toggle a tool group, an MCP server, a namespace, or a skill from the chat composer, or switch the model. The “same bytes across calls” is exact — per selection: the front is stable precisely as long as the selection that produced it stands.
The cold turn is marked, not silent. Each conversation keeps the last-applied selection snapshot alongside its warmness signal (cache_last_touch_at plus the stored TTL). When a changed selection applies on a warm conversation, the runtime sets a cold-turn marker: the agent sees a one-line [CACHE] section in the live view, the decision call’s accounting metadata carries cache_cold_turn, and a log line correlates the change with the next cache attempt. Accounting already records cache_creation_tokens and cache_read_tokens on every call — the marker joins that rebuild premium to the action that caused it, one identifiable component within the turn’s spend sum.
And the user decides when to pay. The user pays for the cache, so the user holds the policy for selection changes: accept, confirm (the platform default), defer_cold, or defer_conversation — admin config sets the default and bounds the allowed set. Under confirm, the composer turns a costly change into an inline choice at the moment it matters: apply now, from the next conversation, or when the cache is cold — where applying is free. A deferred change parks as a pending delta the runtime promotes when its trigger fires.
07Keeping it small over time: pruning vs. compaction
Reuse handles cost within a warm window. Two other mechanisms handle the input getting large over a long conversation, and they’re different in kind.
TTL pruning is the freshness rule. After a cache window goes cold, older payloads are hidden behind short replacement text — the recent turns stay visible, the old bulk doesn’t. Crucially, pruning hides payloads, not availability: the hidden block is still there, still addressable by its path, still re-openable on demand. Compaction is the hard ceiling. When the visible input genuinely approaches the model’s window (today, roughly 90% of the budget), the runtime summarizes a stretch of older blocks into a single summary block and renders only from that summary onward. The summarized turns still survive behind their paths and turn indexes.
| TTL pruning | Compaction | |
|---|---|---|
| When | a cache window goes cold | input nears the model’s hard limit |
| What it does | hides old payloads, keeps recent | replaces an old stretch with a summary |
| Stays whole | hidden blocks, reachable by path | summarized turns, reachable by path/index |
| Role | keep the hot prompt light | the ceiling that can’t be crossed |
Neither one is amnesia. Both keep logical availability even as they shrink what’s visible.
08Render order is part of the design
All of this happens in a fixed order each render, and the order is the point.
Cache points are computed last among the stable steps, on a stream that’s already settled — so they never get placed on bytes that are about to move. And the tail goes on at the very end, keeping the uncached part truly at the bottom. Stable first, volatile last: maximal reuse, by construction.
09Recovery is deliberate, not a lucky cache hit
Because pruning and compaction shrink what’s visible, the agent has a real path back to what left the prompt — and it’s a deliberate route, not a hope that something’s still cached. A visible path reads back directly. A summary points at a turn index, which lists the exact refs to re-open. Topic-only? A search returns the refs or the index to read. The summaries are the semantic map; the logical paths are the handles; the stored turn logs and artifacts are the source of exact data.
This is why the honest promise isn’t “the model never forgets.” It’s stronger and more truthful: the model can forget aggressively when it should, and still recover what matters, because the system leaves enough path-and-summary structure behind.
10The point: availability, not perfect recall in the prompt
Caching here isn’t about freezing context. It’s about keeping the expensive prefix reusable, the freshest signals fresh, and everything that scrolls out of view still reachable. The input’s shape — stable front, growing middle, volatile tail — is what makes all three possible at once.
- #1 — The model input: the structure. The five parts of the input and where each comes from.
- #3 — The live view — the agent’s attention zone. The tail’s own world, and why it’s rebuilt every single round.