Context Caching

Long contexts can exhaust GPU memory before model weights do. Capacity planning for agent workloads must include KV cache, concurrency, and serving-engine behavior.

Agentic inference makes KV-cache locality and prefill pressure first-class scheduling concerns. Red Hat's case study improved latency and capacity, but P/D separation needs suitable traffic and fast networking.

Prefix Sliding drops stale reasoning tokens while retaining instructions and recent work, capping memory during long inference and reportedly making existing models 3× faster without training.

LFU was the strongest simple semantic-cache default, but answer validity mattered far more than eviction policy. Raw hit rates above 50% fell near 2% after quality adjustment.

In this AI-tutor evaluation, keeping full history beat compaction on recall and cost because cached tokens were cheap. Measure your constraint before shortening agent context.

Blast Radius predicts which code and context a prompt will reach, then reversibly archives unused history. Across seven OpenAI models, it cut token consumption by 17–26%.

Model routing can cut agent costs only when delegation, escalation, and context transfer are designed together; duplicating context across models can erase the savings.

TokTier targets a hidden agent-serving bottleneck: repeatedly tokenizing long transcripts after small tool results. Its stateful repair cut median vLLM time to first token by 16–34% in reported tests.

ReToken adds one learned retrieval embedding to select relevant visual tokens from a cached context, improving long-image and video benchmarks while fitting training and inference on one H100.

Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.

A delayed KV-eviction policy helps when memory reuse is sharp and observable, but loses its edge on natural-text benchmarks—a useful warning for agent-memory claims.

OmniRoute puts many model providers behind one OpenAI-compatible endpoint, with routing, quota failover, cost telemetry, compression, and coding-agent setup. Its breadth raises operational and trust questions.

Caveman compresses agent inputs and outputs, with local tooling for noisy commands, browsing, memory, and skills. Its own caveat matters: terse workloads can cost more once the skill prompt is included.

Context Mode keeps bulky tool output outside an agent’s prompt and restores relevant session state after compaction. Its MCP tools and hooks support 17 clients, though enforcement varies by platform.

Reasonix is a configurable terminal coding agent built to preserve DeepSeek prefix-cache reuse across long sessions, with optional planner/executor models and subprocess-based plugins.

oMLX serves local models to coding agents on Apple Silicon, combining continuous batching, persistent SSD-backed KV caching, and menu-bar control behind compatible APIs.