How we manage agent context

Every time an agent takes a step, it sends the model everything again: the instructions, the tool definitions, every message and every tool result, from the top. A task that reads four files and writes a summary is five or six requests, and a long working session is hundreds. The model has already seen nearly all of each one.

Providers cache for exactly this reason. When a request begins with the same bytes as an earlier one, that stretch is read back from the cache at a fraction of the normal input price (a tenth or less on Claude, a tenth on GPT-5.6 and later), and the answer starts sooner. The match runs byte for byte from the first byte and stops at the first difference, so one changed character near the top means everything after it is processed, and billed, as new.

That makes managing an agent’s context mostly a question of what you let yourself change once it has been sent. This is how we answered it in the SotaAgents gateway.

A broken cache is silent

When the cache breaks, nothing else does. The answer is still right and nothing gets logged. The only signs are a slower first token and, a few weeks later, an invoice bigger than anyone can explain.

Providers read a request in a fixed order: tool definitions, then the system prompt, then the messages. Claude caches at breakpoints the caller places, up to four per request. GPT models find the shared prefix themselves. Gemini can work either way.

That is why a timestamp in the system prompt is so expensive: it moves the first differing byte to the top of every request. Manus called it “a common mistake” in 2025, and it still is one. A tool list that grows when the agent finds a new tool does the same, as does a status note re-inserted before the latest message every turn, or an old tool result cut down to a preview once it’s a few turns old. All of them are common, because each is a sensible idea taken alone.

We wrote down six rules the gateway keeps within a conversation on one model:

1. Append-only. Each request starts with the whole previous request plus the model’s reply to it, both between the steps of a turn and across turns.
2. The tool list is frozen. Finding a tool, loading a skill or switching a tool off never changes tools[].
3. A tool result is fixed when it’s written. What the model sees of it is decided when the tool returns.
4. Skills sit at the tail. A skill’s text is added once, where it was used, and survives compaction.
5. Live equals replay. A request rebuilt from the database on a later turn is byte-identical to the one sent live.
6. An answer is a result. When the agent asks the user a question, the answer comes back as that call’s tool result.

Five events are allowed to break the prefix, at the cost of one miss each:

• the user switches model;
• the conversation is compacted (we call this an epoch, the one point where history may shrink; more on it below);
• someone changes a system block they own, such as the user’s instructions, a workspace admin’s prompt or an app’s prompt hook;
• an app or MCP server is installed or removed (the first one brings two tools with it, and the system prompt lists what’s installed);
• the user edits or regenerates an earlier message.

Apart from those, and a few edge cases we’ve documented, anything that breaks the prefix is a bug, and we treat it as one.

One function builds every request

The simplest way to keep requests append-only is to never build the same message two ways. In our gateway, a request’s messages come from one function with two inputs: the conversation’s stored messages and its active epoch. The live turn calls it, and so does every later turn, over the same stored rows. Nothing in it looks at how far a message is from the end, so a message comes out as the same bytes on turn 3 and on turn 30.

That moves the work to write time. Whatever affects how a message looks to the model is saved with the message, exactly as it was sent: the model’s view of each tool result, the text description we write for an attachment the model can’t read directly, a message the user typed mid-turn (stored where it arrived), the reasoning the model returned. When the database holds all of that, a rebuild can’t drift from the live request, because it’s the same function run on the same input.

The same rule decides how tool calls are stored. Models often get arguments slightly wrong in harmless ways; Qwen, for one, likes to send "offset": "0" where the schema wants a number. We convert the value before the tool runs, but history keeps the call exactly as the model wrote it, so the next request, and every rebuild after it, carries what the model actually emitted.

DeepSeek’s open-source harness treats this property as a theorem and tests it: a fresh session must be able to rebuild the same provider request from what was stored. Our main test is built around the same idea.

The tool list never changes

Tool definitions come first in a request, ahead of the system prompt, so adding one halfway through a conversation invalidates everything after it. In one of our test tasks the agent found an app’s tool and called it. With a builder that adds discovered tools to tools[], the two requests after the tool was added read nothing from the cache on Claude Sonnet 5, and each wrote about 18,000 tokens. Claude charges 1.25 times the input price to write to its cache and 0.1 times to read from it, so that’s the most expensive way there is to send those tokens, twice.

A workspace can have dozens of tools once its apps and MCP servers are installed, sometimes hundreds. Declaring all of them on every request is costly and makes the model worse at picking one; declaring them as they come up breaks the cache. So a conversation declares the core tools every conversation uses (the clock, files, skills, asking the user, sub-agents, reading back a stored result), plus two more when the workspace has any app or MCP tools: searchTools and callTool.

searchTools returns its matches with their full input schemas as a tool result, which puts the schema at the tail of the conversation, where adding it costs nothing. callTool({ name, params }) is a dispatcher. It looks the name up in the conversation’s catalog, checks params against that tool’s real schema, and runs the tool with everything it would normally get: permissions, approvals, output limits, tracing. A wrong name, or input that doesn’t fit, comes back as an error the model can act on. The gateway renames the call to the real tool as it streams, so people using the app still see “Jira · create issue”.

Switching a tool off works the same way: a tool the user disabled stays declared, and calling it returns a plain message saying it’s off.

Anthropic’s API lets tools be declared with defer_loading and found through a tool search, which in the docs’ words “preserves prompt caching”. OpenAI’s Responses API loads searched tools at the end of the context for the same reason, and Claude Code and Codex CLI both keep MCP tools behind a search. We needed one mechanism for every model we serve, including those we reach through OpenRouter’s chat-completions API, which has no field for deferred tools. Two ordinary tools do the job, for several hundred tokens of fixed prefix that is read from the cache from the second request on.

A tool result is written once

Tool output is where context gets big: a database query can return 48 KB, and a web page more. The usual answer is to trim old results as the conversation goes on, by their age or by how full the window is. That works, but it changes bytes the model has already seen, and it keeps changing them as a result gets older.

We decide once, when the tool returns. The model’s view of a result comes from the tool’s own formatter and then a size limit: past 24,000 bytes, the full output is stored and the model gets a preview with a pointer to the rest (read_tool_result, which pages through it). That view is saved on the message and replayed unchanged on every later turn. It doesn’t depend on the app still being installed, on the stored copy still existing, or on whatever the settings are today.

Images inside tool results work the same way: the exact bytes the model saw are stored under their hash and put back on every rebuild.

Shrinking context costs a cache miss, so we do it rarely and on purpose. When a result does have to go, it goes at an epoch.

What belongs to a turn goes on that turn’s message

The system prompt holds what’s true for the whole conversation: the agent’s instructions, the user’s and the workspace’s instructions, the skills it can load, the tools that exist. It’s assembled once per turn and is identical on every step of that turn. Nothing in it reads the clock (the model calls get_current_time when it needs the time), and nothing in it depends on the history.

What belongs to a single turn goes on that turn’s user message instead: attached files, the project’s reference data when it has changed, a conversation the user @-mentioned, a skill invoked with a slash command. These are worked out when the turn starts, saved on the message, and rendered after what the user typed by the same function, live and on every rebuild.

Two cases took more thought than they seem to need.

A failed read is not a change. Several system blocks come from a live read on every turn: an installed app’s prompt hook, a knowledge base’s trained prompt, the project’s instructions. When a read fails (a timeout, a 500 from the app, a network blip), the obvious fallback is to leave the block out. That changes the system prompt for one turn and back again on the next, two misses for something that never changed. So a successful read is taken as the truth, even when it comes back empty because the app was uninstalled, and a failed read reuses the last value this conversation rendered, with the failure logged.

A question and its answer. When the agent needs to ask the user something mid-task, it calls a tool, ask_user. The answer arrives as the user’s next chat message, but the model sees it as that call’s result, matched by an id: The user replied: followed by the answer. Both alternatives touch history: most providers reject a tool call that has no result, and patching a result in later rewrites a message that was already sent.

Claude Code puts updates in a <system-reminder> tag “in the next user message or tool result”, and Codex CLI keeps its environment and project instructions in the first user message and appends changes as diffs. Anthropic’s newer models accept system messages mid-conversation, which would be tidier, but not every model we serve has them (Sonnet 5 doesn’t), and every model has a user message.

History shrinks in one place

Append-only can’t go on forever. Context windows end, and long before they do, big prompts get slow and expensive. So a conversation does shrink, but only at an epoch, never a little at a time. An epoch changes the messages only, never the tools or the system prompt, so those are still read from the cache; everything after the first thing it changed is written again, once. After that, the conversation is append-only again.

Each model gets a budget: its context window, minus room for the answer and a small margin. We cap it at 256,000 tokens even on million-token models, so compaction, latency and price behave alike across models.

Against that budget there are two tiers, both checked when a user turn starts:

• Tier 1 clears, from 70%. Old tool outputs, large old tool inputs and old attachments are replaced by short placeholders that say what they replaced and point at a stored copy the model can read back. No model call is made, and at the start of a turn it runs only when it frees at least 64 KiB, so a miss is never spent on a small gain (that batching threshold is Cline’s). It never touches the current turn.
• Tier 2 summarizes, from 80%, when clearing alone can’t bring the request back under that line. Everything before a verbatim tail of up to 15,000 tokens becomes one marker message: the summary, the text of skills used before the boundary (up to 5,000 tokens each and 25,000 in all, the caps Claude Code uses), the ids of the newest files and slide decks the agent made, and a line on how to get anything back.

The summary request is itself a fork of the conversation: the same tools, system prompt and messages, byte for byte, with one instruction added at the end. Claude Code’s team calls this cache-safe forking. The summary reads the conversation’s cache instead of sending it all cold: on Sonnet 5, summarizing about 76,000 tokens cost $0.151 sent cold and $0.011 to $0.018 as a fork.

Between the steps of a turn, when the model has called a tool and is about to read the result, nothing is reset, even past a threshold. There are two exceptions: a next request that wouldn’t fit the budget at all, and one that would go over the provider’s limits on images or request size. A summary that fails, or that leaves the request above the 80% line, counts against a circuit breaker, which waits longer after each failure before trying again.

Counting from the provider’s number

Each of these decisions needs the size of the next request. Like most harnesses we read, we start from the provider’s count for the previous request and estimate only what has been appended since. Where we differ is the estimate. The usual four characters to a token is close for English prose and off for Vietnamese, Japanese and JSON, so ours keeps a ratio per tokenizer family, counts non-ASCII characters separately, prices images, audio, video and PDFs by their pixels, length and pages rather than their bytes, and corrects itself against the provider’s numbers as the conversation goes on. A reported count far above what we expected is set aside until the next report confirms it, so a single outlier can’t trigger a summary nobody needed.

Same rules, every provider

Our requests go to many models: most through OpenRouter, some directly with a customer’s own OpenAI, Google or Anthropic key, some to self-hosted GPUs behind an OpenAI-compatible endpoint. The six rules hold on all of them. What differs is how a request is marked for caching, how reasoning is carried from one step to the next, and how a conversation keeps landing on the server that holds its cache.

We keep those differences as data. Each model family has a profile, and the compiler that turns a prompt into a request reads it; the compiler itself never branches on a provider. A family-specific rule only goes in with the provider’s documentation, or a measurement, cited next to it. Adding a provider takes a descriptor and an adapter that moves bytes, not another copy of the history logic.

• GPT caches on its own, so we send nothing.
• Claude caches only at breakpoints, four at most. We put them at the end of the core instructions, at the memory block, at the end of the system prompt and on the latest message, because those parts change at different rates. A breakpoint looks back only 20 blocks for an earlier cache entry, so when one step appends 20 or more (a burst of parallel tool calls, say), we mark the previous request’s last position again so the read still lands there. The memory breakpoint gives up its slot for that request; the one at the end of the system prompt still covers it.
• Gemini, left to its implicit cache through OpenRouter, read 41 to 46% of the prompt in our runs. With one explicit anchor that trails the end of the conversation and moves forward in steps of about 4,000 tokens, never backward, it reads about 90% of the prompt, and 96 to 97% of the warm prefix: the part an earlier request already sent, which is all any cache can serve.
• Pools and auto-routing, where a request can go to any of several models and the one that serves it isn’t known in advance, get one marker on the whole request. Claude upstreams honor it, and it doesn’t narrow which upstream can serve the request.

A cache also lives on a particular server. OpenRouter tends to send a conversation back to the same upstream provider, and it starts doing so from the first request only if the request carries a session id. We scope ours to the workspace and the model rather than to the conversation, because the session scopes the cache as well. With an id per conversation, every new conversation would pay full price for the same system prompt: on a 9,500-token prefix, a second conversation costs 11 credits that way and 2 with the workspace-wide id. (Credits are how SotaAgents meters usage.) The id never spans workspaces, so a conversation only reads cache entries its own workspace wrote, and a read needs identical bytes from the very first one, so what conversations share is only the start they already have in common, such as the tools and the workspace’s instructions. Side calls, such as repairing a malformed tool call, writing a title or suggesting follow-ups, get keys of their own, so they can’t pull the main loop onto a different upstream mid-turn.

Reasoning gets a rule of its own. Models return reasoning (plain, summarized, signed or encrypted) that they expect to get back on the next step. We replay it as it streamed (less a few unsigned pieces one provider library drops, the same way every time), but only to the model that produced it. Every part of an assistant message carries a stamp naming the model that served it, and parts from another model go back as plain content, without reasoning or provider-specific fields. Switching model drops the old model’s reasoning at the switch, which is a miss anyway.

Side by side

Very little here is new on its own. Most of it shows up somewhere in the harnesses we read while building this, and we borrowed from several of them. This is how five of them handle the same four questions.

Harness	Tools during a conversation	Old tool output	Compaction	Size of the next request
Claude Code	MCP tools deferred behind a tool search; their stubs stay the same	Cleared first, before anything is summarized	A summary sent as a cache-safe fork; invoked skills re-injected (5k tokens each, 25k in all)	The last response's reported input
Codex CLI	MCP tools deferred behind a client-side search; the tools parameter doesn't change	Cut to about 10k tokens, head and tail, when written	OpenAI's encrypted compaction item, or a local summary on other providers	The server's total, plus bytes ÷ 4 for newer items
Gemini CLI	All declared on every call; a new MCP server rewrites the list	Older outputs masked once they pass a threshold	A state snapshot at half of a 1M window, keeping the newest 30%	The provider's count, plus a characters heuristic
Cline	MCP tools declared; a change to them restarts the session	Every result cut to 8,000 characters as the request is built; stale reads rewritten in batches of at least 64 KiB	A summary at 90% of the usable input, keeping at least 20k tokens	Characters ÷ 3, calibrated against the provider's count
OpenCode	Declared, sorted by name	Optional pruning of old outputs	A summary and a tail of up to 15k tokens	The provider's count (characters ÷ 4 to size the tail)
SotaAgents	Fixed: the core tools, plus searchTools and callTool	Fixed when written; cleared only at an epoch, normally at least 64 KiB at a time, each behind a pointer	Clearing first, then a summary sent as a cache-safe fork; skills re-injected (5k / 25k)	The provider's count for the last request, plus a calibrated estimate of what was appended

Where we differ is mostly in how strictly the line is held. Outside an epoch and the five declared events, nothing is allowed to change a byte the provider has already cached, and a test fails when something does.

How we know it holds

A break that makes no noise needs a test that goes looking for it. Claude Code’s team has written that they “alert on cache breaks and treat them as incidents”. We also wanted a test that catches breaks before they ship.

Ours drives the real turn pipeline, from the chat endpoint down to the code that serializes the request for the provider, and captures every request body. It compares each request with the one before, in the order the provider reads them, and reports the first byte where they part and which rule was broken. There are 71 scenarios, from plain chat to a tool found mid-turn, compaction, a model switch, an admin changing the workspace prompt and 20 images in one turn. The same scenarios run live against real models, where we read the providers’ own count of cached tokens.

On the final live pass, the six paid models read between 96.3% and 100% of their warm prefix (what earlier requests had already sent) from the cache. To put a price on that, we ran the same seven scripted tasks, with the same text, models and settings, through two builders. One was a conventional builder, one we actually ran, built from the common patterns above: a sliding window over old turns, a status note re-inserted each turn, tools added as they’re found, Gemini left to its implicit cache, and a summary request sent cold. The other was this one. On Claude Sonnet 5, GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek, a full set of the seven tasks cost $1.93 through the conventional builder and $0.67 through ours, 65% less. On Claude and GPT, the longer and more tool-heavy the task, the wider the gap: the task that ran into compaction cost 71% less on Sonnet. Gemini gained the most almost everywhere, 87% even on plain chat, because its implicit cache had read so little.

Applying the cache rates we measured live to a month of our staging traffic gives a smaller figure: just over 28% of model spend, somewhere between 21% and 37% depending on assumptions. Most of those conversations are a single turn, and the saving grows with length. The savings are all on input: tokens read from the cache instead of processed and billed as new.

What it costs

• A bigger fixed prefix. Two more tools (the dispatcher and the local-machine tool), longer tool schemas and a slightly longer system prompt add about 450 to 950 tokens to every request, depending on the model. They’re read from the cache, but a plain chat with no tools costs 3 to 5% more on Claude and GPT than through the conventional builder.
• Bigger prompts later on. Keeping tool results whole makes later prompts about 7.5% larger than trimming would. Those extra tokens are cache reads, and the task built around big results still came out 12 to 72% cheaper, depending on the model.
• Earlier compaction. Counting the system prompt and tools in the budget brings compaction sooner: on Sonnet, the long task was summarized twice where a looser count summarized once. The forked summary more than pays for it.
• Less help from strict decoding. callTool declares its params loosely, since it has to carry any tool’s input, so a provider that constrains output to a schema can’t enforce the inner tool’s. Some models then send numbers as strings, which the dispatcher converts, where the tool’s schema expects a number, before it validates.
• A miss whenever the shape changes. Any change to how the prompt is assembled costs every existing conversation one miss when it ships.

One thing the numbers don’t tell you is whether the answers were as good. Every run finished without errors on both builders, but we didn’t grade the answers against each other, and the quality of summaries in particular hasn’t been measured yet.

What’s next

• Grading answers and summaries on the same tasks, next to their cost.
• A one-hour cache on Claude where the upstream honors it (customers calling Claude with their own key), and keeping a cache warm while a long tool runs. Both wait on data about how long people pause between turns.
• Anthropic’s native deferred tools, in place of our dispatcher, for those same customers.

If you’re building agents and would rather not discover these rules one invoice at a time, we’d be glad to compare notes.