A shared prompt prefix does not automatically mean a shared cache hit. This matters when an agent coordinator is about to launch a batch of parallel jobs—say, fifty document classifications that all need the same policy, tool definitions, examples, and reference packet. If every worker starts at once, they can all become cache-writing requests. You pay fresh-input or cache-write costs repeatedly, and the slowest cold start may still set the batch’s visible latency.
The practical move is to make cache warm-up an explicit stage before the fan-out. Send one bounded request containing the exact reusable prefix, wait until the provider reports that the cache is available, then release the workers that share it. Treat the warm-up result as a small receipt, not a hopeful delay: it should identify the prefix version, cache partition or routing key, response usage, expiry, and the observed cache-read result from a second request.
This is a coordination technique, not a new way to arrange a prompt. A stable prefix remains necessary, but it is not sufficient when many requests arrive together. Anthropic says a cache entry becomes available only after the first response starts; for concurrent requests that need hits, it recommends waiting for the first response before sending subsequent requests.[1] OpenAI similarly describes cache reuse as dependent on reaching a machine that holds an unexpired matching entry, with routing influenced by organization, region, model, request load, and the initial prompt content.[2]
Make the first request a deliberate warm-up
Choose a lightweight representative suffix: a harmless classification, extraction, or “ready” task that exercises the same model, tools, developer instructions, cache settings, and reusable context as the eventual workers. Do not warm the cache with a different prompt and assume it transfers. Cache matching is about the eligible rendered prefix, so a changed tool definition, reordered instruction, tenant-specific addition, or different processing region can turn a warm-up into an unrelated entry.[1][2]
The warm-up request needs an ordinary useful result only if your product has one. Its main job is to create the reusable state and return provider usage data. Then send a single probe with the same prefix plus a different tiny suffix. Release the batch only if the probe shows the expected cached-input or cache-read evidence for your provider.
The probe catches the expensive false positive: a successful warm-up that does not create a reusable entry.
Copy this cache warm-up gate
Put this gate in the coordinator immediately before a batch whose requests intentionally share context.
BATCH CACHE CONTRACT
- prefix_version: [hash/version of reusable instructions, tools, and corpus]
- model + region: [exact deployment]
- cache partition/routing key: [tenant-safe stable key, if supported]
- cache settings: [TTL, explicit/implicit breakpoints, retention]
- expected minimum cached tokens: [provider/model-specific threshold]
- batch size and deadline: [N jobs, start-by time]
WARM-UP
1. Build request = exact shared prefix + harmless warm-up suffix.
2. Send one request; record request ID, prefix_version, and usage fields.
3. Wait for the provider’s documented cache-ready point.
4. Send one probe = exact shared prefix + distinct tiny suffix.
5. Accept only if probe reports cache-read evidence at or above the expected threshold.
RELEASE RULE
- cache_ready: enqueue workers with the identical prefix contract.
- cache_not_ready but deadline allows: retry one warm-up with bounded backoff.
- cache_not_ready and deadline does not allow: run a deliberately limited cold batch,
record the miss, and do not claim cache savings.
RECEIPT
{prefix_version, model, region, partition, warmed_at, expires_at,
warm_usage, probe_usage, decision: cache_ready|cold_fallback|blocked}
Version the inputs that affect reuse: tool schemas, developer instructions, reference version, cache settings, and model. Partition tenant-specific policies before warming; a cache key is not a shortcut around data or billing boundaries. OpenAI says caches are not shared across organizations or regional processing boundaries; Anthropic describes workspace-level isolation.[1][2]
Release workers with a cache contract, not a vague hope
After the probe passes, preserve the prefix contract: job builders must not inject a timestamp, reorder tools, or add instructions before the cache boundary. Put job-specific text after it. Log cache-read evidence per worker and calculate hit rate by prefix version and partition.
A cache is an optimization, not durable state. Record its expiry estimate and re-run the gate after it passes. A miss may cost more or take longer, but must not change the answer or cross a tenant boundary.[1][2]
Failure mode: warming work that cannot be reused
Do not add a warm-up stage to a two-job batch with a small prompt, or to workers whose prefixes differ materially. The coordination overhead can exceed the saving. It can also create a thundering herd if every queue consumer independently decides to warm the same key.
Use one owner per {prefix_version, model, region, partition} and make other consumers await its short-lived receipt. If the warm-up fails, let one bounded retry happen; then choose cold fallback, a smaller batch, or a later run. The goal is not to force a hit. It is to make a cache miss visible before parallel work multiplies it.
The habit to keep
before a large agent fan-out, prove that the shared context is reusable in this provider, this partition, and this moment. Warm once, probe once, then release the batch with a receipt that can explain the cost.
Sources
Previous issues
- Make Every Tool Argument Point Back to Evidence
Require an agent to attach an allowed evidence reference to every consequential tool argument, then have code construct the final payload from those references instead of trusting reconstructed values.
- Tell the Agent When Not to Use a Tool
Put exclusion rules, safe alternatives, and output limits in each tool definition so an agent can reject a tempting near-match before it produces a bad call.
- Name the Unknowns Before You Start AI Research
Turn a broad request into a short decision frame that separates fixed requirements from open variables, so an AI researches the question you actually need answered instead of a plausible substitute.
- Draw the Dependency Line Before You Parallelize Agents
Split only the work that can finish from the same stable input, then give each parallel agent a bounded deliverable and a single synthesis rule.
- Put Every Production Prompt in a Versioned Helper
Treat a production prompt as a typed, reviewed, testable code artifact, so a prompt edit has an owner, a diff, a fixture, and a rollback path.
- Make Each Retrieval Slot Earn a Different Job
Diversify a broad candidate set before it reaches the model, so a fixed context budget covers distinct evidence instead of five near-duplicate chunks.
- Locate Before You Read the PDF
Turn a large PDF question into a page-finding pass and a visual-evidence pass, so the model spends attention on the pages that can actually support the answer.
- Validate the Tool Result Before It Enters the Agent Loop
Put a typed result gate between every tool response and the model, so malformed, partial, stale, or out-of-scope data becomes a safe receipt instead of unexamined context.
- Partition a Semantic Cache Before You Trust It
Treat a semantic-cache hit as a candidate answer that must match the caller’s policy partition, source version, and output contract before it can bypass a fresh model call.
- Put an Egress Gate Before an Agent Calls an External Tool
Check every model-to-tool payload against a destination-specific data policy before it leaves the application, so an agent cannot turn useful context into an accidental disclosure.