Compaction

A long chat or a multi-step agent loop keeps adding messages. At some point the transcript passes the model's context limit and the call fails. You want the conversation to keep working without hitting that wall.

withCompaction shrinks provider context before each model call. When the context passes maxTokens, a strategy rewrites what the model sees. The canonical transcript does not change. Add this ChatMiddleware to the middleware array of any chat() call.

Install#

pnpm add @tanstack/ai-compaction

Quick start#

The default strategy drops the oldest messages once the transcript passes maxTokens and keeps the recent ones.

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { withCompaction } from "@tanstack/ai-compaction";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: openaiText("gpt-5.5"),
    messages,
    middleware: [withCompaction({ maxTokens: 100_000 })],
  });

  return toServerSentEventsResponse(stream);
}

Pick a strategy#

Pass strategy to change how the history shrinks. Three are built in.

evictOldest#

Cheapest. Keeps the recent tail, drops the older head, and leaves a short marker in its place. This is the default, so you only name it to tune keepRecentTokens.

import { withCompaction, evictOldest } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: evictOldest({ keepRecentTokens: 40_000 }),
});

summarizeOldest#

Keeps the gist of old turns instead of dropping them, at the cost of one summarization call. Pass a summarize callback. It gets the messages about to be dropped and returns the summary text. Wire it to summarize() or any model call.

import { chat, summarize, toServerSentEventsResponse } from "@tanstack/ai";
import { openaiText, openaiSummarize } from "@tanstack/ai-openai";
import { withCompaction, summarizeOldest } from "@tanstack/ai-compaction";
import type { ModelMessage } from "@tanstack/ai";

async function summarizeHistory(messages: Array<ModelMessage>) {
  const text = messages
    .map((m) => `${m.role}: ${typeof m.content === "string" ? m.content : ""}`)
    .join("\n");

  // Return the whole result. Compaction reports its `usage`.
  return summarize({
    adapter: openaiSummarize("gpt-6.1-sol"),
    text,
  });
}

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    messages,
    middleware: [
      withCompaction({
        maxTokens: 100_000,
        strategy: summarizeOldest({ summarize: summarizeHistory }),
      }),
    ],
  });

  return toServerSentEventsResponse(stream);
}

summarize can return the summary text, or an object with summary and usage. Return the summarize() result as it is, so compaction can report what the summary cost.

clearToolResults#

Best for agent loops. Tool output (file reads, command output) is usually most of the tokens. This strategy replaces the content of old tool results with a stub and keeps every message and its tool-call pairing in place. The conversation shape does not change.

import { withCompaction, clearToolResults } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  // Keep the 5 most recent tool results in full, stub the older ones.
  strategy: clearToolResults({ keepRecentToolResults: 5 }),
});

Write your own#

A strategy is a function. It gets the messages and the budget, and returns the rewritten messages, or null to change nothing. It runs when the count is over maxTokens, and when compactNext forces it. Then the count can be under maxTokens.

import { withCompaction } from "@tanstack/ai-compaction";
import type { CompactionStrategy } from "@tanstack/ai-compaction";

// Keep only the last message.
const keepLastOnly: CompactionStrategy = (messages) => {
  if (messages.length <= 1) return null;
  return messages.slice(-1);
};

withCompaction({
  maxTokens: 100_000,
  strategy: keepLastOnly,
  strategyKey: "keep-last-v1",
});

Set strategyKey when you combine a custom strategy with persistence. Change the key when the strategy can produce different output. This prevents an old checkpoint from using stale behavior.

Combine strategies#

composeStrategies runs several strategies in order and escalates: it stops as soon as the result is back under maxTokens. Put the cheap, targeted strategy first and a broad fallback last. Here it clears old tool output first, and only drops old messages if that was not enough.

import {
  withCompaction,
  composeStrategies,
  clearToolResults,
  evictOldest,
} from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: composeStrategies(clearToolResults(), evictOldest()),
});

Let the provider compact#

Some providers can compact the conversation on their own servers. The model then keeps more of the old turns than a marker gives. It costs one compaction call.

Pass the adapter of the chat() call as native. When the count passes maxTokens, the adapter's compact method runs in place of strategy.

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { withCompaction } from "@tanstack/ai-compaction";

const adapter = openaiText("gpt-6.1-sol");

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter,
    messages,
    middleware: [withCompaction({ maxTokens: 100_000, native: adapter })],
  });

  return toServerSentEventsResponse(stream);
}

• An adapter with no compact method uses strategy.
• When compact fails, strategy runs in its place. onCompact and compaction:ended get the failure in error. Some providers on the OpenAI Responses base have no compaction endpoint.
• After a 404, 405, or 501 (the provider has no compaction endpoint), that adapter object and model use strategy until the process restarts. Create the adapter once, outside the request handler, so that no more compaction calls go out.
• After any other failure, such as a 400, a 429, or a network error, the next compaction tries compact again.
• The compaction call gets the system prompts of the chat() call. It does not get the tools: the compaction endpoint does not take them.
• The compacted messages can hold an encrypted provider item. The adapter sends it back as it is to the same model. Another model does not get it.

The OpenAI Responses adapters have compact. See Native compaction.

When the chat is still too long#

The token count is an estimate. So once in a while a call is still too long for the model, and the provider rejects it.

You do not have to catch that error. When a call fails with CONTEXT_OVERFLOW_CODE, withCompaction compacts and calls the model again, in the same run. Your code only sees the answer. This works with auto: false too.

It retries once per model call. If the compacted call is still too long, the run ends with a RUN_ERROR:

• code is CONTEXT_OVERFLOW_CODE ('context_overflow').
• message is the provider's own text.

The OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, Grok, Cloudflare, and BytePlus adapters set this code. Other OpenAI-compatible adapters set it when the provider sends context_length_exceeded.

Give the retry something to cut#

The retry compacts even under maxTokens. But a strategy can still find nothing to cut:

• composeStrategies stops when the estimate is under maxTokens.
• evictOldest keeps maxTokens / 2 by default.

Set a small keepRecentTokens, so the retry always drops something:

import { evictOldest, withCompaction } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: evictOldest({ keepRecentTokens: 20_000 }),
});

Compact on your own schedule#

Want the next call on a thread to compact, even under maxTokens? Call compactNext(threadId). For example, do this before you switch the thread to a model with a smaller context window.

Create the middleware once, so compactNext reaches the same instance:

import { withCompaction } from "@tanstack/ai-compaction";

const compaction = withCompaction({ maxTokens: 100_000 });

// The next model call on this thread compacts first.
compaction.compactNext("thread-1");

Options#

withCompaction#

Strategy options#

The token count is a rough characters / 4 estimate. It is good enough to trigger on, not exact. Pass estimateTokens for provider-accurate counts.

Count with real usage#

The characters / 4 estimate drifts on long agent runs. Code, JSON, and other languages do not split into tokens like English text. Then compaction runs too late and the call fails, or it runs too early and drops context you still need.

Set countTokens: 'usage'. After each model call, compaction saves the usage that the provider reported. The next check starts from that number and estimates only the messages that came after it.

import { chat, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'
import { withCompaction } from '@tanstack/ai-compaction'

export async function POST(request: Request) {
  const { messages } = await request.json()

  const stream = chat({
    adapter: openaiText('gpt-6.1-sol'),
    messages,
    middleware: [withCompaction({ maxTokens: 100_000, countTokens: 'usage' })],
  })

  return toServerSentEventsResponse(stream)
}

• Before the first model call there is no usage yet, so compaction uses the estimate.
• With a metadata store, the usage is saved. The next request and a restarted server start from the real count.
• The strategies still use the estimate to choose which messages to keep.

Set the limit from the context window#

Most models publish a context window. Keep room for the answer: set maxTokens to the window minus a reserve.

import { summarizeOldest } from '@tanstack/ai-compaction'
import { summarize } from '@tanstack/ai'
import { openaiSummarize } from '@tanstack/ai-openai'

const contextWindow = 200_000
const reserveTokens = 20_000

const windowed = withCompaction({
  maxTokens: contextWindow - reserveTokens,
  contextWindow,
  countTokens: 'usage',
  strategy: summarizeOldest({
    keepRecentTokens: 8_000,
    summarize: (messages) =>
      summarize({
        adapter: openaiSummarize('gpt-6.1-sol'),
        text: messages
          .map((m) => `${m.role}: ${typeof m.content === 'string' ? m.content : ''}`)
          .join('\n'),
      }),
  }),
})

Some providers accept a request that is too large. They cut the input and answer without an error. On a host with a session log, compaction with countTokens: 'usage' and durable: true checks once more after the last model call of a run. If the usage passed maxTokens or contextWindow, it compacts right away, so the next run starts small. With auto: false, only contextWindow counts. The answer is not sent again. The only extra model calls are the summary calls.

Without a session log, there is no check after a run. The next request compacts before its first model call.

Prepare the summary in the background#

A summarizeOldest call takes time. When compaction runs at maxTokens, the user waits for that call before the model answers. Set background to prepare the summary earlier, while the chat goes on:

import { chat, summarize, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiSummarize, openaiText } from '@tanstack/ai-openai'
import { summarizeOldest, withCompaction } from '@tanstack/ai-compaction'

// Create it once, so a summary from one request reaches the next.
const compaction = withCompaction({
  maxTokens: 100_000,
  background: { atTokens: 80_000 },
  strategy: summarizeOldest({
    summarize: (messages) =>
      summarize({
        adapter: openaiSummarize('gpt-6.1-sol'),
        text: messages
          .map((m) => `${m.role}: ${typeof m.content === 'string' ? m.content : ''}`)
          .join('\n'),
      }),
  }),
})

export async function POST(request: Request) {
  const { messages, threadId } = await request.json()

  const stream = chat({
    adapter: openaiText('gpt-6.1-sol'),
    messages,
    threadId,
    middleware: [compaction],
  })

  return toServerSentEventsResponse(stream)
}

• atTokens must be below maxTokens. If it is not, withCompaction throws.
• When the count at a model call passes atTokens, the summary starts on a copy of the messages. The model call does not wait for it.
• The ready summary waits in the metadata store, under the namespace @tanstack/ai-compaction:background. Without a store, it waits in memory.
• The summary applies at the first model call of the next run on the same threadId, never in the middle of a run.
• A model call over maxTokens while the summary runs waits for it and applies it. It compacts inline only when the messages are still over maxTokens. A cancel of the run stops the wait, not the summary.

Sometimes a summary does not apply:

• A newer compaction cut past the message that the summary keeps. The summary is dropped and reported with stale: true.
• The summary call failed. Nothing applies, and the failure is reported with error. The next model call past atTokens starts a new summary.

onCompact always gets these reports. A compaction:ended event for a failed summary reaches the stream only while the run that started the summary still runs.

What it keeps safe#

• The system prompt is never dropped. chat() keeps it separate from messages, so compaction only touches the conversation.
• Tool calls stay paired with their results. The built-in strategies never leave an orphaned tool result, so the request stays valid.
• It runs before every model call. Compaction skips init. It runs on beforeModel and structuredOutput. Each later call can compact again.
• The canonical transcript stays complete. Compaction writes provider-only context. Persistence and other middleware still read ctx.messages.

Better summaries#

The basic summarizeOldest setup works for a chat. A coding agent needs more. It must keep the goal, the decisions, and the files it touched across many compactions. Three opt-in parts help:

1. cut: 'turn' on summarizeOldest: a better summary when the cut falls inside a turn. The older turns get one summary. The start of the current turn gets its own short summary, under ## Turn context.
2. conversationSummarizer: a ready summarizer with a structured prompt (goal, constraints, progress, key decisions, next steps, critical context). When an older summary exists, it updates that summary. It reports the usage of each summary call.
3. details on conversationSummarizer: your own facts after the summary, for example the files the agent read. The next compaction gets them back as previousDetails.

import { openaiText } from '@tanstack/ai-openai'
import {
  conversationSummarizer,
  summarizeOldest,
  withCompaction,
} from '@tanstack/ai-compaction'

function pathOf(args: string) {
  const input: unknown = JSON.parse(args)
  return typeof input === 'object' &&
    input !== null &&
    'path' in input &&
    typeof input.path === 'string'
    ? input.path
    : undefined
}

const agentCompaction = withCompaction({
  maxTokens: 180_000,
  contextWindow: 200_000,
  countTokens: 'usage',
  strategy: summarizeOldest({
    cut: 'turn',
    keepRecentTokens: 8_000,
    summarize: conversationSummarizer({
      adapter: openaiText('gpt-6.1-sol'),
      details: ({ messages, previousDetails }) => {
        const files = new Set(previousDetails?.split('\n').filter(Boolean))
        for (const message of messages) {
          for (const call of message.toolCalls ?? []) {
            const path =
              call.function.name === 'read_file'
                ? pathOf(call.function.arguments)
                : undefined
            if (path) files.add(path)
          }
        }
        return files.size > 0 ? [...files].join('\n') : undefined
      },
    }),
  }),
})

• In the text sent to the summarizer, each tool result is cut to 2,000 characters. Set maxToolResultChars to change it.
• A summarize function of your own gets a second argument, summarize(messages, input):
  • previousSummary and previousDetails: the older summary and its details.
  • turnPrefix: true when the call summarizes the start of the current turn.
  • turnPrefixMessages: the start of the current turn. Only the call that writes the details gets it, so the details cover each dropped message.
  • signal: aborts when the run aborts.

DevTools#

After a compaction, the chat stream includes three CUSTOM events in order: compaction:started, compaction:state, then compaction:ended. compaction:started is sent before the strategy runs. Durability flushes each event as soon as it is emitted, so a slow summarizeOldest call still shows as started on the client while it runs. The state and ended events follow when the strategy returns.

compaction:ended also says why compaction ran and what it cost:

• reason: 'threshold' (the count passed maxTokens), 'forced' (compactNext, or the retry after a context overflow), or 'background' (a background summary).
• usage: the token usage of the summary calls.
• error: the message when the strategy failed. By default the run then fails. Set continueOnError: true to send the full messages instead.
• stale: true when a ready background summary no longer fit the messages and was dropped.

A compaction after the last model call of a run has reason: 'after-turn'. The stream is closed by then, so only onCompact reports it. A failed strategy or onCompact there does not fail the run, because the answer is complete. Only a failed write to the log fails it.

TanStack AI DevTools has a Compaction tab on the hook. Each compact shows:

• started, state, and ended rows
• when it ran, with token and message counts
• the maxTokens budget
• dropped messages
• the transcript sent to the model

The conversation timeline also keeps onCompactStart, onCompact, and onCompactEnd steps.

Open the AI plugin in the DevTools panel (the ts-react-chat example mounts it). Select the Compaction hook, then open the Compaction tab.

The /compaction route in examples/ts-react-chat uses a small maxTokens so this fires after a few turns. It also uses withPersistence (SQLite). Reload after a compact: the chat still shows every message. The banner is example UI, not part of useChat.

Compaction and persistence#

Compaction and server-side withPersistence use two message views:

• messages is the complete canonical transcript. Persistence saves this view.
• providerMessages is temporary model context. Compaction rewrites this view.

Middleware order does not change this split. Dropped, summarized, and stubbed content remains in the message store.

If the persistence adapter has a metadata store, compaction also saves a small checkpoint. The next request validates the canonical prefix, restores the last compacted result, and adds only new messages. A changed prefix or strategy key invalidates the checkpoint.

When a checkpoint is reused, a later summarizeOldest pass sees the previous summary plus new messages. Then it folds the old summary into the new one. Folding needs a metadata store and a strategy key.

The default strategy, standard evictOldest, summarizeOldest, clearToolResults, and safe compositions get a strategy key automatically. Set strategyKey for custom strategies, custom estimators, or custom marker functions. Change strategyKey when your summarize function can change. Without a metadata store or safe key, compaction stays stateless.

With durable: true on a host with a session log, compaction writes its result into the log instead. See Keep compaction in a session log.

Keep compaction in a session log#

Some hosts keep a durable log for each thread and rebuild the model context from it when a session opens again. A metadata checkpoint can miss on that rebuild. Set durable: true so the compaction goes into the log, and the rebuilt session sees the same context.

1. The host provides LogRecordsCapability. See Append records to the session log.
2. Add withCompaction({ durable: true }) to the same chat() call.
3. Fold each compaction record into the messages with projectCompaction: for the next model call, and when the host rebuilds the session.

import type { ModelMessage } from '@tanstack/ai'
import { projectCompaction, withCompaction } from '@tanstack/ai-compaction'

const compaction = withCompaction({
  maxTokens: 180_000,
  contextWindow: 200_000,
  countTokens: 'usage',
  durable: true,
})

type LogEntry =
  | { kind: 'message'; message: ModelMessage }
  | { kind: 'record'; record: { type: string; [key: string]: unknown } }

// Fold the log in order. A compaction record replaces the messages before it.
function contextOf(log: Array<LogEntry>): Array<ModelMessage> {
  let messages: Array<ModelMessage> = []
  for (const entry of log) {
    if (entry.kind === 'message') messages = [...messages, entry.message]
    else messages = projectCompaction({ messages, record: entry.record }) ?? messages
  }
  return messages
}

• Each compaction writes one record with the type 'tanstack.compaction' (COMPACTION_RECORD_TYPE). The host must accept this type.
• projectCompaction returns undefined for a record of another type, and for a record that does not fit the messages. Then keep the messages as they are.
• Without the fold for the next model call, each later call of the run compacts the full list again.
• The log only grows, so the original messages stay in the store.
• Without LogRecordsCapability, compaction uses the metadata checkpoint, and there is no check after the run.

Next steps#

• Middleware: the full hook reference and how middleware composes
• Built-in Middleware: ready-made middleware that ships in @tanstack/ai