Claude API context compaction: you pick when to summarize

Long-running agents rarely die on the first tool call. They die when history gets fat: every turn resends the same failed stack traces, latency climbs, the bill swells, and the model quietly forgets a filename you settled thirty minutes ago.

We already covered how to build an application-level memory system. That is your store, your retrieval, your write policy. This piece is about something narrower: Anthropic turning “replace old turns with a server-written summary” into a Messages API feature. Around 14 September 2026 the docs added an on-demand path under beta compact-2026-09-04. ML Pills’ weekly builder roundup framed it as context management moving into the model platform. Vendors will stretch that sentence. You still own the swap, the billing aggregation, and the eval that catches a dropped order ID.

Two betas, two ideas of control

Do not mix the two styles on one request.

Threshold compaction (beta compact-2026-01-12) lives in context_management.edits with edit type compact_20260112. You set an input-token trigger. Default is 150,000; minimum is 50,000. When an ordinary chat request crosses the line, the API writes a summary mid-request and can continue into the answer you asked for. The threshold docs pitch this at multi-turn chat and long task threads where you do not want to schedule summarization yourself.

On-demand compaction (beta compact-2026-09-04) uses a top-level "compaction": {"type": "summarize"} parameter. That request returns only a signed compaction block. Success means stop_reason is "compaction". There is no normal assistant reply. You replace the summarized span of history with that block and keep sending the signature unchanged. AI Codex’s walkthrough of the September 14 parameter focuses on control: you choose the moment, and the summary can run while the agent keeps talking. You can also leave the last few turns verbatim after the block.

Rule of thumb: use threshold when a chat product is fine letting the API manage context inside ordinary requests. Use on-demand when your own agent loop cannot pause for a summary, needs keep-tail behavior, or wants preserved thinking in the recent turns to stay valid.

Platform caveats matter before you copy-paste. On-demand is documented for the Claude API, Claude Platform on AWS, and Microsoft Foundry. It is not available on Amazon Bedrock or Google Cloud for that beta. Supported model IDs include Fable and Mythos 5.1/5, Opus 5 and several 4.x Opus builds, Sonnet 5 and 4.6. Confirm with the Models API field capabilities.compaction instead of trusting a blog list six weeks later.

On-demand: request, then swap

Minimal flow: ask for a summary, then swap it in.

import anthropic

client = anthropic.Anthropic()
SYSTEM = "You are a data-modeling assistant for a recipe app."

history = [
    {"role": "user", "content": "Name the main entities for a recipe app."},
    {
        "role": "assistant",
        "content": "Start with Recipe, Ingredient, and Step. Add RecipeIngredient for quantity and unit.",
    },
    {"role": "user", "content": "Good. Suggest field names for Recipe."},
]

summary = client.beta.messages.create(
    model="claude-opus-5",
    max_tokens=8000,
    system=SYSTEM,
    messages=history,
    betas=["compact-2026-09-04"],
    extra_body={"compaction": {"type": "summarize"}},
)

assert summary.stop_reason == "compaction"
# Keep content exactly as returned, signature included
history = [{"role": "assistant", "content": summary.content}]

Hard rejects to bake into your client before you chase 400s in production:

• The last assistant turn ends in a tool call with no tool result yet. Finish that turn first.
• You cannot send context_management on the same request as compaction.
• Certain tool_choice values (any / pinned tool), structured-output format settings, and stop_sequences block the summarizer call.
• The conversation still has to fit the model’s context window when you ask for a summary. Compacting after you already overflowed is too late.

Background work is where swap logic usually breaks. If the agent kept chatting while the summary request ran, remember sent_count at submit time:

block = {"role": "assistant", "content": summary.content}
history = [block] + history[sent_count:]

Drop only the span you sent to the summarizer. Keep turns appended after that snapshot. Official rules: the compaction block must sit first in messages, or you get compaction_block_misplaced. Leaving the old summarized text after the block is often accepted, which is worse in a quiet way: you pay for those tokens again and the model sees a duplicate story.

To compact a second time, send compaction on a history that already starts with a block. The new block summarizes the old summary plus everything after it. From then on, keep only the newest block.

Keep-tail, background compaction, custom instructions

Agents often want “old work as a summary, last two turns as raw text.” Keep recent turns does that by omitting the tail from the compaction request. After the block arrives, order is block → kept verbatim turns → new user message.

That pattern matters for preserved thinking. On models that keep thinking blocks, the thinking in the kept turns can stay valid if those turns originally followed the summarized span and if system plus non-defer_loading tools match what you sent on the compaction request. Edit history between request and swap, or change the system prompt while trying to keep a tail, and the thinking check fails. Expect a 400, not a gentle degradation.

Background compaction is the async version: when your token estimate crosses a threshold, fire a separate request with compaction on a worker; the main loop keeps using the full history; swap when the future completes. You temporarily hold two open requests, both billed and rate-limited, while the full history keeps growing. Start the job while you still have headroom, not when you are thousands of tokens from the wall.

Custom summaries use instructions. That field replaces the default summarization prompt entirely; it does not append. Always tell the model not to call tools and to return summary text only. Otherwise the summarizer can emit tool_use against tools you declared for the agent, and you walk away with empty content or a useless stop_reason. Support bots should demand account IDs, order numbers, attempts already tried, and the open question. Coding agents should demand paths, failing test names, and rejected approaches.

Threshold for chat products that want less glue

If you only need the API to manage context inside normal messages.create calls, threshold is shorter:

response = client.beta.messages.create(
    betas=["compact-2026-01-12"],
    model="claude-opus-5",
    max_tokens=4096,
    messages=messages,
    context_management={
        "edits": [
            {
                "type": "compact_20260112",
                "trigger": {"type": "input_tokens", "value": 100000},
            }
        ]
    },
)
messages.append({"role": "assistant", "content": response.content})

When a compaction content block appears, append the full response.content as usual. On later requests the API ignores everything before that block.

If you need to re-insert recent messages after the summary, set pause_after_compaction: true. Then stop_reason is "compaction" and the response is basically the summary block. Rebuild messages and continue. Anthropic’s own examples multiply compaction count by the trigger threshold as a rough cumulative budget, then inject a “please wrap up” user turn when you blow past a total. It beats pretending context is free.

Prompt caching works if you put cache_control: {type: ephemeral} on the end of the system prompt and on the compaction block, so a new summary does not force a full system rewrite into cache. Details stay in the threshold documentation.

Billing, silent failures, what the summary eats

Compaction is another sampling pass. Under threshold, top-level input_tokens and output_tokens exclude the compaction iteration. Sum usage.iterations entries of type compaction and message. On-demand successes often show zeros at the top level; the real cost still sits in iterations. Cost dashboards that only read the top-level fields undercount.

On-demand failures frequently return HTTP 200 with empty content. Always inspect stop_reason:

stop_reason	Usual meaning
max_tokens	Summary truncated; raise max_tokens into the thousands
tool_use	Model called a tool; ban tools in instructions
model_context_window_exceeded	No room left for the summarization prompt
refusal / end_turn	Declined or produced nothing; retry or skip

A retryable 529 with compaction_unavailable shows up under load. Forget the beta header and you may see a generic compaction: Extra inputs are not permitted with no mention of the missing header. That one burns support time.

Once a span is summarized, images, documents, container_upload blocks, and fetched URLs inside that span are gone. Mid-conversation system messages in the span get summarized away and stop applying. Anything a later turn still needs should be restated on the next user turn, or kept outside the summarize window on purpose.

A minimal agent loop sketch

Here is a sketch you can adapt without pretending it is production-ready. The point is the control points: when you measure tokens, when you fire compaction, and how you swap without deleting work that arrived late.

COMPACT_AT = 80_000  # your choice; far below the model window
pending = None
sent_at_submit = None

def approx_tokens(messages):
    # replace with count_tokens in real code
    return sum(len(str(m)) // 4 for m in messages)

for user_text in user_stream:
    if pending is not None and pending.done():
        summary = pending.result()
        if summary.stop_reason == "compaction":
            history = [{"role": "assistant", "content": summary.content}] + history[sent_at_submit:]
        pending = None
        sent_at_submit = None

    history.append({"role": "user", "content": user_text})
    response = client.beta.messages.create(
        model="claude-opus-5",
        max_tokens=4096,
        system=SYSTEM,
        tools=TOOLS,
        messages=history,
        betas=["compact-2026-09-04"],
    )
    history.append({"role": "assistant", "content": response.content})

    if (
        approx_tokens(history) > COMPACT_AT
        and pending is None
    ):
        sent_at_submit = len(history)
        pending = executor.submit(
            client.beta.messages.create,
            model="claude-opus-5",
            max_tokens=8000,
            system=SYSTEM,
            tools=TOOLS,
            messages=list(history),
            betas=["compact-2026-09-04"],
            extra_body={"compaction": {"type": "summarize"}},
        )

Three boring habits save more pain than clever prompts. First, log every compaction with sent_at_submit, stop_reason, and the usage.iterations entry. Second, never mutate system or the tool list between the compaction request and the swap if you are keeping a thinking-bearing tail. Third, after each swap, run a cheap probe question the agent should already know (“what was the failing test name?”) and treat a miss as a regression, not a surprise.

If you are building chat rather than an agent, skip the executor and use threshold compaction with a trigger you actually measured on your traffic. A default of 150,000 is convenient for demos; many support threads get expensive and fuzzy much earlier. Measure median and p95 input tokens for successful sessions, then set the trigger with room for the next user message and a fat tool result.

Managed Agents and similar hosted loops may already compact for you. In that case this parameter is the wrong layer to touch. Read the product docs before you wire a second summarizer that fights the platform.

When you still need your own memory layer

Compaction answers “how does this conversation or agent run keep fitting in the window?” It is not a vector store, not an audit log, and not cross-session user preference storage. The patterns in our memory management tutorial still apply when a fact must survive for days. Use both: compaction for prompt size, durable memory for “the same user comes back on Thursday.”

Likewise, Herdr-style terminal multiplexers keep processes alive, and Coder Agent Relay moves the filesystem boundary. Neither replaces Messages API history accounting. Do not delete your custom summarizer the day the beta ships. Run one long workflow through a few compaction cycles. Measure task success, tool accuracy, latency, and tokens against your current memory strategy. Keep whichever stack wins on that table.

One more operational note: token counting via /v1/messages/count_tokens applies existing compaction blocks but does not trigger a new compaction. Use it to check effective size after a swap. Do not use it as a substitute for watching usage.iterations when a live request actually compacted. Teams that mix both APIs without separating those metrics end up arguing about “why did the meter jump” in Slack for half a day.

Start with five checks: correct beta header; for on-demand, assert stop_reason; signed block first with signature untouched; cost from usage.iterations; a fixed eval for what the summary drops, not vibes.