How the model input is cached

In #1 we mapped the shape of the model input: a stable per-agent front (instructions · tool catalog · skill catalog), a timeline that grows oldest → newest, and a volatile tail at the very end. That shape wasn't an accident. It's what makes the input cheap to reuse and safe to keep fresh at the same time.

This is #2 of three. Here we look at what happens across calls — how most of that input is reused instead of re-sent, how the freshest signals stay fresh, and how the agent can drop junk early without throwing away the cache. The trick isn't pretending context never changes. It's shaping what stays visible.

Why caching is even possible here

Sending a long input to the model is expensive. The instruction envelope, the catalogs, and the older history don't change between one round and the next — so re-sending them every time is pure waste. Prompt caching lets the model reuse an exact prefix it has already seen, and only pay full price for what's new after it.

The catch: caching only works on a prefix that is byte-for-byte identical to last time. Change one thing near the front and everything after that point has to be re-sent. So the whole strategy comes down to one design choice: put the stable things first, the volatile things last.

That's exactly the shape from #1: the expensive part is reused, and the freshest part is never cached, so it can change every round without a thought. To make that reuse precise, the runtime drops cache points along the way — one anchoring the envelope, three more inside the timeline. They're the next section.

The tail is never cached — on purpose

The simplest and most important rule: the tail is never cached. The sources pool and the live view sit at the very end, after every cache point, and they are rebuilt every round.

This is deliberate. Fast-moving state — the current budget, a live user interrupt, a fresh projection of the workspace — has to be current on every call. If it were baked into the cached prefix, you'd face a bad choice: either churn the expensive prefix every round to update it, or let the agent read stale state. By keeping that state in an uncached tail, you sidestep both. The prefix stays calm; the tail stays fresh. (The tail's own structure is the subject of #3.)

So the rule of thumb is: caching by shaping visibility, not by freezing the input. The stable things get cached; the volatile things are quarantined at the end where caching never reaches them.

The four cache points

A cache point is a boundary that says "everything up to here is stable; reuse it." There are four, and where each sits is the whole design. The first anchors the envelope; the next three live in the timeline; the uncached tail follows the last one:

① Envelope / prefix point — anchors the whole instruction envelope: the runtime and agent/domain instructions plus the tool and skill catalogs. This is the stable per-agent prefix from #1, and it's the cache boundary that lets the entire front of the input be reused across calls without being re-sent.
② Previous-turn point — right after the last block of the prior turn. The cross-turn anchor: once a new turn begins, the whole conversation up to here can be reused for the rest of the turn.
③ Pre-tail point — the last block of round N−4, a deliberate extra checkpoint set a few rounds back from the newest activity (it appears only once the turn has enough rounds). This is what opens the editable zone — see the next section.
④ Tail point — right after the most recent stable round, the last block in the cacheable stream.

Point ① keeps the expensive front reusable. Point ② is about reuse across turns. Points ③ and ④ are the current turn's moving frontier. And everything after point ④ — the sources pool and the live view — is the uncached tail; no cache point ever reaches it.

All four cache points in order — and the uncached tail that follows ④, never reached by any of them.

The extra checkpoint lets the agent forget garbage early

The pre-tail point earns its keep. Without it, the agent's only safe cache boundary would be at the very edge of activity — which means there'd be no room to drop anything without busting the cache.

With the extra checkpoint a few rounds back, there's a small editable zone between it and the tail. If the agent pulls a huge, useless result — a giant page it shouldn't have fetched — it can hide that block: the bulky payload leaves the model's eyes, a short note takes its place explaining what was hidden, and the original stays recoverable by its path. Because the hide happens below the pre-tail point, the cached prefix above it is untouched.

That's the point of the extra checkpoint: a cache boundary that lets early junk drop out cheaply, instead of forcing the agent to either keep staring at it or re-send the whole expensive prefix to be rid of it.

Cache points move with the turn

The previous-turn point is fixed once the turn starts. The other two are not — they march forward as rounds accumulate.

Early on there's no pre-tail point yet; later it appears at N−4 and both it and the tail point slide downward.

Early on there may not be enough rounds to place a pre-tail point at all; later it appears, and both it and the tail point slide downward as the agent works. The editable zone isn't a fixed range — it's a window that tracks the turn's structure. That's what you want in a live loop: the cached part keeps growing while the editable frontier stays near the action.

Keeping it small over time: pruning vs. compaction

Reuse handles cost within a warm window. Two other mechanisms handle the input getting large over a long conversation, and they're different in kind.

TTL pruning is the freshness rule. After a cache window goes cold, older payloads are hidden behind short replacement text — the recent turns stay visible, the old bulk doesn't. Crucially, pruning hides payloads, not availability: the hidden block is still there, still addressable by its path, still re-openable on demand. It's out of the hot prompt, not out of existence. A one-time notice tells the agent it happened and how to read hidden content back.

Compaction is the hard ceiling — the floor under everything. When the visible input genuinely approaches the model's window (today, roughly 90% of the budget), the runtime summarizes a stretch of older blocks into a single summary block and renders only from that summary onward. It's not the everyday cleanup; it's what stops the input from ever overflowing. The summarized turns still survive behind their paths and turn indexes.

TTL pruning vs. compaction — different mechanisms, same promise of availability.
	TTL pruning	Compaction
When	a cache window goes cold	input nears the model's hard limit
What it does	hides old payloads, keeps recent	replaces an old stretch with a summary
Stays whole	hidden blocks, reachable by path	summarized turns, reachable by path/index
Role	keep the hot prompt light	the ceiling that can't be crossed

Neither one is amnesia. Both keep logical availability even as they shrink what's visible.

Render order is part of the design

All of this happens in a fixed order each render, and the order is the point: prune the stale stuff first, compact if it's still too big, drop in any hidden replacements, then recompute the cache points, and only then append the uncached tail.

Prune → compact (if needed) → hidden replacements → recompute cache points → append the uncached tail.

Cache points are computed last among the stable steps, on a stream that's already settled — so they never get placed on bytes that are about to move. And the tail goes on at the very end, keeping the uncached part truly at the bottom. Stable first, volatile last: maximal reuse, by construction.

Recovery is deliberate, not a lucky cache hit

Because pruning and compaction shrink what's visible, the agent has a real path back to what left the prompt — and it's a deliberate route, not a hope that something's still cached. A visible path reads back directly. A summary points at a turn index, which lists the exact refs to re-open. Topic-only? A search returns the refs or the index to read. The summaries are the semantic map; the logical paths are the handles; the stored turn logs and artifacts are the source of exact data.

This is why the honest promise isn't "the model never forgets." It's stronger and more truthful: the model can forget aggressively when it should, and still recover what matters, because the system leaves enough path-and-summary structure behind.

The point: availability, not perfect recall in the prompt

Caching here isn't about freezing context. It's about keeping the expensive prefix reusable, the freshest signals fresh, and everything that scrolls out of view still reachable. The input's shape — stable front, growing middle, volatile tail — is what makes all three possible at once.

That leaves one part we've kept pointing at and never opened: the tail itself.

#1 — the structure. The five parts of the input and where each comes from.
#3 — the live view. The tail's own world: the agent's attention zone, and why it's rebuilt every single round.

Documentation on GitHub

The live docs behind this entry:

How the model input is cached — and kept fresh