Prompt caching: same agent, same rules, about a quarter of the bill

Prompt caching has been covered a lot. The big providers' guides explain how it works, and many posts tell how a team's hit rate jumped after one fix. We want to add a way to watch it happen: one agent's prompt, drawn the same way at every step, with the measured number in each picture. Sometimes the same idea, shown another way, is what makes it click.

To show it, we built a small recipe agent based on what happened in our own agents: a prompt that looks fine, fixed in four moves, every version sent to a real model API. Same model, same rules, not a word changed: its prompts got 73% cheaper, and the whole test cost less than a cent. At the end we show a test that keeps them from coming undone, and what the same moves did in ours.

How a prompt cache decides what you pay

Every call to a model sends the whole request again: the tools, the instructions and the conversation. A prompt cache lets the provider keep the work it already did on the start of a request, and reuse it when the next request starts exactly the same way. In the guides from OpenAI and Anthropic, reading from the cache costs a tenth of the normal input price or less, and writing to it a quarter more.

The rule behind it is short: a prompt cache reuses everything up to the first thing that changes. That is the whole game: what never changes goes first, and what changes goes last.

Four moves on a recipe agent

Our test agent writes recipes. It has two tools, find_in_pantry and save_recipe, about a thousand words of rules, and a profile for each person. The app has two plans: Pro can save recipes, free cannot. Ana is on the Pro plan and vegetarian; Leo is on the free plan. During one evening, each of them sends three messages.

A prompt that looks fine

This is the prompt many agents start with. The date goes on top, so the agent knows what day it is. Ana's details come next, so it can personalize. Her plan decides which tools she gets. And like many harnesses, it builds the whole prompt again before every call to the model.

The cache was on, and nothing came back from it. Every message wrote its whole prompt into the cache and none read it back, so the evening cost more than it would have with no cache at all.

Step 1: Move what changes on every message to the end

The date changes every minute, and it sits near the top. Whatever comes after it is new work on every message, the thousand words of rules included. Both guides say the same: what changes on every request goes at the end. So the date moves into the user message.

Each person's later messages now read almost their whole prompt from the cache, and the evening went from 0% to 65%. Anything that changes on every message goes at the end.

And yet Leo's first message still read nothing.

Step 2: Keep personal details out of the shared part

Ana's details change with the person. With them in the instructions, nobody else can reuse Ana's prompt. So they move into the user message, and the instructions become the same text for everyone.

The evening went down a little, to 64%, because Ana's details now travel with each of her messages. And Leo still read nothing. What belongs to one person goes after what everyone shares, but that alone was not enough: something still differed, and it came before the instructions.

Step 3: Give every plan the same tool list

The tools come before the instructions, and Leo's free plan gave him a shorter list. His request differed from Ana's at its very first line, so it could not reuse anything she had left in the cache.

The fix is to send every plan the same list. The list only describes the tools; it does not decide what Leo may use. That is decided apart: OpenAI's guide says to "restrict which tools are callable while keeping the supplied tools list stable", and the code that runs the tools refuses anything his plan does not allow. Give everyone the same menu, and decide at the table what each one may order.

Leo's first message now reused what Ana had left in the cache, and the evening reached 80%. The plan still decides what each person can do; it just stopped deciding what the prompt says.

Step 4: Tool calls almost free: append, don't rebuild

To answer Ana, the agent first checks her pantry: the model asks for find_in_pantry, the agent runs it and calls the model again with the result. If that call only adds the tool call and its result at the end, everything before them comes from the cache. OpenAI's guide puts it plainly: "Append new messages rather than rewriting earlier turns."

The call after the tool read 94% of its prompt from the cache. An agent makes many calls like this one, so they end up almost free. In the first version, the agent rebuilt the whole prompt before every call, with a fresh time on top, and the same call read nothing. A tool call should add to the request, never rebuild it.

What the four moves add up to

Together, the four moves changed what the evening cost. At a price of $1 per million input tokens, its prompts went from $1.76 to $0.48 per thousand calls, and the call after the tool cost $0.25.

None of the four moves changed a word of the rules. We ran the test on gpt-6-luna with reasoning off; for any model, multiply these costs by its price per million input tokens.

Check your own agent

Take two requests your agent sends, from two people a minute apart, and find the first place where they differ. Everything before it can come from the cache; everything after it is paid on every call. If that place is near the top, look at what sits there: a date, a name, a tool list built for one plan.

Then keep it that way with a test. This one never calls a model: it builds two requests and compares how they start.

# two people, two times: their requests must start the same
ana = build_request(ANA, "6:42 PM", "Dinner with chickpeas?")
leo = build_request(LEO, "6:43 PM", "A quick lunch with eggs?")
assert ana["tools"] == leo["tools"]              # same tool list
assert ana["messages"][0] == leo["messages"][0]  # same instructions

# the call after a tool must start with the whole call before it
second = add_tool_result(ana, PANTRY_CALL, PANTRY_RESULT)
assert second["messages"][:2] == ana["messages"]

If someone later puts the date back on top, writes a name into the instructions or trims the tool list for one plan, the first check fails. If someone rebuilds the prompt after a tool call, the second one does.

How we did it in our own agents

In an earlier post on owning the harness we said caching deserved a post of its own. This October we made the same kind of moves on our real agents: the time went to the end of each request, the tool definitions stopped varying by project, role or language, and each tool call adds to the request instead of rebuilding it.

We saw even more improvement than in this small evening, because our agents often make more than one tool call for each message, and those are the calls that read the most from the cache. The more tool calls, the higher the average. Our dashboard went from the 60s to 93%. What we are building is still in development, with few people and long gaps between messages, so many calls find the cache expired, like Ana's first message of the evening; with continuous use we expect 96 to 98%.

Look at the first lines of your prompt. If they change from one call to the next, move them to the end, and let the cache do the rest.