Every search embeds its query. The repeats land on another worker.

A hybrid search embeds the query on every call. The vector it gets back is a pure function of (query, model, dimension) — the same three inputs always produce the same numbers — so every repeated search re-pays for an answer the system already computed.

The obvious fix is to remember the vector on the index object. In a runtime where turns hop between workers, that memo is worth almost nothing, and the reason is not that it is small. It is which repeats actually happen.

The repeats all cross a process boundary

Look at what a real repeat is:

• Pagination. “Load more” is the same query with a bigger window. The next page may be served by another worker.
• A later turn. The same person searches the same term ten minutes on, in a new turn — which is a new turn wherever the scheduler put it.
• Two people, one obvious term. Different sessions by definition.
• A search-backed tool. Called once per turn, across many turns.

Not one of those is a burst of identical searches inside a single process. A per-process memo catches the case that rarely happens and misses the four that do — and, being per-process, N workers hold N copies that none of them can invalidate.

So the cache is shared and nothing else. It is a kind of the platform KV cache — the favicon cache's sibling — under kdcube:cache:query-embedding, prefixed by tenant and project like every other namespaced cache. A query is not secret, but the vector belongs to a project's model configuration.

The key carries the model, not just the query

Normalization stops at trimming, collapsing whitespace, and casefolding. Two searches differing only in spacing or capitalization do produce the same vector from the embedder, so they should share a key. Stemming or stop-word removal would go further and make the key disagree with the text that was actually embedded — a cache returning a vector for a query nobody ran.

The digest keeps the key bounded and safe to log. The model and the dimension sit in front of it because they are part of the identity: ask for 1536 floats and get 1024 back, and the honest answer is a miss.

A vector is big enough to need a ceiling

A TTL — a day, by default — bounds how old an entry gets. It says nothing about how many there are. At roughly 8KB apiece, an unbounded vector cache is a slow leak that happens to answer questions.

So each scope keeps a recency index — a sorted set of its own keys, scored by last use — beside the entries.

That makes the ceiling enforceable and the eviction least-recently-used rather than whatever happened to expire first. The default cap is 5000 entries per tenant/project — about 40MB — and Redis' own maxmemory-policy stays underneath as the backstop.

Failing soft, in both directions

Every path returns nothing rather than raising: a miss, a malformed payload, a vector of the wrong width, a Redis that is not answering. The index embeds and carries on. The recency index is best-effort too — if it cannot be maintained, the cache still serves.

A search must never fail because a cache did. The cache exists to spare an embedder call, and a spared call is not worth a broken search.

The rule underneath

The cache is small. The rule it follows is not:

State that can be rebuilt is shared or rebuilt — never remembered on a long-lived object in one process.

A turn can execute on any worker, and an index instance is rebuilt freely. Under those two facts, anything held inside one process is a coin flip that mostly loses, and N copies that quietly disagree the moment anything changes. When the repeats you care about cross a process boundary, the cache has to cross it too.

Related reading

The search this cache sits under: your conversations are now searchable — by you.

Documentation on GitHub

The live docs behind this entry:

• The KV cache and its kinds — namespaces, prefixing, and the query-embedding kind in full
• Hybrid index — how a caller configures the shared query-embedding cache