Optiak Docs
Gateway

Context Engine

Reduce the input tokens each request sends to the model, and measure the effect before you apply it.

The Context Engine shapes the context an application sends to a model: it compresses redundant content, summarizes retrieved data, and trims long conversations. Fewer input tokens lower the cost of each request and keep long conversations within a token budget you set.

Required role

Organization AdminWorkspace AdminProject AdminApp Admin

Applies to

LLM requests
Admins of an application add the Context Engine's components and choose whether each one only measures or also changes requests. App Viewers can see the configuration but can't change it.

Why use it

Input tokens often account for a significant fraction of a request’s cost. As conversations continue and applications include tool results or documents, the context can grow. The Context Engine helps reduce it centrally across all requests from an application, as part of the optimization described in the Introduction.

  • Lower input costs without changing application code.
  • Measure before you change anything. Every component can run in Audit mode, which records what it would remove while the model still receives the original request.
  • Choose how far to go. Removing redundant content is a low-risk start. Summarizing retrieved data and trimming history save more, with more risk of removing something the model needs.

How it works

Optiak offers several context reduction components. When a request arrives, each enabled component applies its configured method to the request’s messages in sequence. The Context Engine then sends the resulting messages to the model. Components operate on messages or collections of messages of the following types:

  • System - System and developer instructions.
  • User - A message from the user.
  • Assistant - A reply from the model.
  • Tool exchange - An assistant message that calls one or more tools, the tool results, and the assistant reply that uses them.

Within the Context Engine, each message or complete tool exchange is treated as an item. A user message plus every item that follows it until the next user message is one turn. For example:

System
Turn 1
UserAssistant
Turn 2
UserTool exchange
Active turn
User...

Two rules apply to every component:

  • Attachments, metadata and other non-textual information is never altered. Images, files, tool-call IDs and provider metadata are kept unchanged. Components only remove whole items or rewrite text.
  • Some content is never removed: system and developer instructions, the active user message and all content that follows it as long as the turn is not finished, incomplete tool exchanges, and data that Optiak can't interpret safely, such as reasoning blocks and provider-specific items.

Components run in the fixed order you see in the console: Smart Compression, Context Compression, Sliding Window, then Context Budgeting, which works on what the others leave.

Audit and Optimize

Each component can be enabled or disabled. When enabled, it applies one Action to the request:

Component stateActionComponent behaviorWhat the model receivesRequest trace
Disabled—Does not runOriginal requestNo component result
EnabledAudit (no action)Runs without applying changesOriginal requestComponent result is recorded
EnabledOptimizeRuns and applies its resultUpdated requestComponent result is recorded

Audit is the default action for a new component. In both Audit and Optimize modes, Optiak records what each component removed or changed and how much that impacted the input token count, so you can compare them in the request traces.

Start each component in Audit, send real traffic, and read the measured savings in the request traces. Switch to Optimize when the numbers and the retained context look right. The effect on the model's answers shows only in Optimize, because in Audit the model still receives the original request.

Components

Add the components your application needs; each one runs under its own Action. They range from removing only redundant content to removing content outright:

ComponentWhat it doesRemoves content
Smart CompressionFolds repeated content in history and tool resultsOnly redundant content
Context CompressionSummarizes retrieved data against the user's questionYes, from retrieved data
Sliding WindowKeeps the start and the latest part of the conversationYes, older history
Context BudgetingCaps the context tokens sent to the modelYes, oldest items first

Smart Compression

Smart Compression removes redundant content from conversation history and tool results. Repeated content is folded, and nothing is summarized or reworded, and user messages and instructions are never compressed.

It helps most with verbose tool output:

  • Logs and build output - Terminal colors are stripped, and repeated lines collapse to a repetition marker.
  • Search results - A file path repeated on every match becomes one heading above its matches.
  • Repeated blocks - Identical chunks in YAML, TOML, INI or plain text become a reference.
  • Diffs - Git bookkeeping the model doesn't need is dropped. The changed lines stay as they are, so the diff still applies.

Settings

  • Minimum tokens to compress - Content below this token count is left untouched. The default is 250 tokens, and the lowest accepted value is 20.
Smart Compression is the lowest-risk place to start. It only removes redundant content, so the model still receives all the information in the original. Savings are modest.

Context Compression

Context Compression summarizes the data that Optiak retrieves from Data Integrations against the user's current question. Only the part relevant to the question reaches the model. Requests without retrieved data pass through unchanged.

Settings

  • Dynamic ratio - Let the engine choose how much to compress. This is the default.
  • Compression level - With dynamic ratio off, choose how much of the retrieved data to remove: Conservative (about 25%), Balanced (about 50%) or Aggressive (about 75%). Aggressive compression can remove information the model needs.

The right level depends on your data. Dense data, where most of the content matters, needs a more conservative level. Data with a lot of noise can take a more aggressive one.

Context Compression runs before the model call, so it adds latency.

Sliding Window

Sliding Window keeps the start and the most recent part of a conversation and drops the history in between.

Messages are removed. What Sliding Window drops is gone from the request. In multi-step agents, a large cut can remove the agent's record of its own work, so it spends extra steps rebuilding it, and it can break the provider's cache.
We recommend that you run it first in Audit mode. Thus, you can adjust the behavior of the component to your applications. For example, this component can be your ally in long chat conversations where older turns are no longer needed. Read more in Prompt caching.

Settings

  • Retention unit - What counts as one unit:
    • Conversation turns - One unit is one conversation turn: user + tools + asisstant response. A turn with many tool calls is still one unit. This is the default.
    • Context items - One unit is a message type, except for the tool exchanges that are maintained as one item. This can keep an assistant message without the user message before it.
  • Keep first units - Units kept from the start of the conversation, which often frame the task. The default is 1.
  • Keep latest units - Units kept from the most recent history. The default is 2.

Content that is never removed (see How it works) doesn't count toward either quota. In conversation-turn mode, a turn that contains such content is kept whole, and so is an incomplete turn. Nothing is removed when there are no more units than the quotas allow, and the latest unit is kept when a window would leave only instructions.

Example - Retention unit Conversation turns, keep first 1, keep latest 2:

System prompt
Turn 1
Turn 2
Turn 3
Turn 4
Turn 5
Active turn
Never removedKeptDropped

Example - Retention unit Context items, keep first 1, keep latest 2. Each item is its own unit, so the window can split a turn:

System prompt
User
Assistant
User
Tool exchange
User
Assistant
Active user message
Never removedKeptDropped

Context Budgeting

Context Budgeting limits the number of context tokens sent to the model. It runs last, after the other components.

Context may be omitted or trimmed. When the context exceeds the budget, whole items are left out, oldest first, and with Allow partial content on, a message can be cut mid-text. As with Sliding Window, a large cut can make an agent repeat work and break the provider's cache.
We recommend that you run it first in Audit mode to see how much a budget would remove. Thus, you can adjust the behavior of the component to your applications. For example, it can help to not go over the context window of the model for some applications. Read more in Prompt caching.

Settings

  • Max tokens - The application's budget, between 8,192 (the default) and 2,000,000 tokens. Leaving the field empty tells the component to use a budget based on the target model's context window size. Setting a manual budget larger than the model's context window maximum budget has no effect.
  • Allow partial content - Trim a message to fit when the remaining budget is smaller than the message. With this off, a message that doesn't fit is left out whole.

Required content is kept first and uses budget first. The remaining context is added from newest to oldest until the budget runs out. Tool calls and their results are never split.

The budget isn't a hard cap. Content that is never removed (see How it works) is always kept, even when it alone exceeds the budget. In that case, only that content is sent.

The model's budget and the response length

The model's context budget leaves room for the response. It's 90% of the model's context window minus the room reserved for the response, or the model's input limit if that's lower. That room is the first of these that is set:

  1. The request's own maximum output tokens, such as max_completion_tokens.
  2. The model's maximum output tokens.

A larger response cap leaves less room for context, so Context Budgeting may remove more.

Prompt caching

Model providers cache the start of a prompt and bill the cached part at a lower price on later requests. The cache matches only up to the first difference from the previous request. Everything after that point is billed at the full price.

Check your cache usage before you turn on Sliding Window or Context Budgeting. If your requests already read much of their input from the provider's cache, these components can raise your cost instead of lowering it. Audit doesn't show this effect, because the request is unchanged. Check Cache read in your request traces first, and again after you switch to Optimize.

Sliding Window and Context Budgeting change where the kept history starts, so they can move that point close to the start of the conversation:

  • Sliding Window - Once the conversation holds more units than the window keeps, each new unit pushes the oldest of the latest units out. Keep first units sets where that change starts: the system prompt and the units kept from the start stay the same and can still be read from the cache, and everything after them is billed at the full price. The more units you keep from the start, the larger the part that stays cached.
  • Context Budgeting - Once the context exceeds the budget, the budget is filled first with the content that is never removed, such as the system prompt and the active user message. The rest of the history is then added from newest to oldest until the budget runs out, so each new item pushes the oldest kept item out. The history right after the required content changes on that request. While the context stays over budget, this can happen on every request.

For example, with Sliding Window keeping the first turn and the latest 2, when turn 6 arrives, turn 3 leaves the window. The cache then matches only up to turn 1, so turns 4 and 5 are billed at the full price again:

Request with turns 1 to 5

System prompt
Turn 1
Turn 2
Turn 3
Turn 4
Turn 5

Next request, with turn 6

System prompt
Turn 1
Turns 2 and 3
Turn 4
Turn 5
Turn 6
DroppedRead from cacheBilled at full priceNew

Each move trades fewer tokens on every later request for one request that bills the kept history at the full price. Large cuts that happen rarely recover that cost quickly. A cut point that moves on every request bills the kept history at the full price every time.

Benchmarks

We measured each component on internal workloads: conversations with long retrieved documents, and multi-step coding agents with verbose tool output.

ComponentSavingsEffect on answers
Smart CompressionAbout 1–2% fewer input tokens, more on log-heavy tool outputNo change measured; only redundant content is removed
Context CompressionFewer tokens of retrieved data, based on the compression level you set. The cost saving depends on the model's priceNo measurable change at Conservative and Balanced; lower accuracy at Aggressive, mostly on questions that combine several documents
Sliding Window, Context BudgetingThe largest cutsAgents can need extra steps to recover removed context

What this means for your application:

  • Start with Smart Compression. It only removes redundant content, so it's the lowest-risk component. Expect about 1–2% fewer input tokens, and more if your tasks produce a lot of logs, build output, search results or other repetitive tool output.
  • Match Context Compression to your data. Use a conservative level for dense data and a more aggressive one for noisy data.
  • Use Sliding Window and Context Budgeting with care in agents. They remove the most tokens, but an agent can need extra steps to recover what was removed. Check your cache usage first; see Prompt caching.

These results vary with your workload and models. Measure each component in Audit mode on your own traffic before you switch it to Optimize mode.

Set it up

In the console, open the application, then Configuration, then Savings, and select the Context Engine tab. Select Add components, choose a component, and set its Action and settings. Components appear as steps in a pipeline, in the order they run, from top to bottom. Removing one leaves the rest of the configuration unchanged.

Each request trace in the application's Requests tab shows the Context Engine as its own stage, with one row per component: the action it ran under, items removed or modified, and the approximate token change. Opening request traces needs an admin role; see Members and Access.

Limits and behavior

  • Sliding Window and Context Budgeting can break the provider's cache. See Prompt caching.
  • The Context Engine can't rescue a request that is already over the model's input limit. Optiak checks that limit before any component runs, and rejects the request with a context-window error.
  • Context Budgeting doesn't run when the model's limits are unknown.
  • Context Compression affects only retrieved data.
Copyright © 2026