Agent Reliability

MCP Tasks aim to make long-running tool calls durable across disconnects and crashes. The proposed V2 removes session-heavy state, but polling scale and implementation complexity remain unresolved.

Nick Heiner argues that leaderboard gains can diverge from useful agent behavior through contamination, weak verifiers, reward hacking, and test conditions that users cannot inspect.

Cybersecurity agents need deterministic exploit oracles and tasks that reward finding distinct vulnerabilities, not self-reported wins or one crash. A capability ladder makes partial progress measurable.

Human task duration is a weak proxy for agent difficulty when agents use different shortcuts and tools. Long-horizon evals need environment complexity, queryable trajectories, and judges that inspect final state.

RLHF can make agents persuasive assistants without making them dependable autonomous decision-makers. Builders should separate human-pleasing interaction from calibrated automation and keep stakes bounded.

Training on a production harness could adapt an agent to real workflows, but non-replayable interactions break common RL assumptions and make feedback harder to turn into updates.

Post-training gains depend heavily on task selection, rollout quality, and environment design. For many enterprise agents, curated SFT may deliver most of the value before costly RL.

Long-running agents strain context, reward design, and GPU scheduling at once. Compaction and value models can help, but each introduces another optimization trade-off.

Biology agents need evaluators that verify analysis of large experimental datasets, not recall. LatchBio found human review essential because valid scientific paths can defeat brittle graders.

TokTier targets a hidden agent-serving bottleneck: repeatedly tokenizing long transcripts after small tool results. Its stateful repair cut median vLLM time to first token by 16–34% in reported tests.

AgentHPOBench tests whether agents can learn from experiment history, not merely produce code. Its results expose weaknesses in sustained refinement and log diagnosis across sequential ML runs.

OVI suggests interactive expert feedback can let a smaller learner imitate values without representing the expert's full policy. The result matters to agent distillation, but relies on specific assumptions.

For agent work without deterministic answers, mine production traces into tasks, build controllable simulators, and use scaled judges to derive rubrics while watching for reward hacking.

AI Gateway’s dedicated logs expose per-request cost, tokens, latency, routing, and provider fallbacks, making agent failures and spend anomalies easier to trace.

AISPA turns system-prompt review into an eight-dimension audit. Its survey suggests builders should test prompts for user protection and conflicting instructions, not merely check that safeguards exist.

OSReward finds that VLM judges often approve failed computer-use runs. Its benchmark and open reward models offer a more grounded way to evaluate trajectories without paying frontier-model costs.

At equal generated-token cost, repeated sampling matched or beat seven reflection, critique, selection, and debate methods. Agent evals should budget every generated token, not compare against one-shot baselines.

Socher’s automated-research design combines prior knowledge, measurement data, simulation, physical experiments, and agent orchestration, with early demonstrations in training and CUDA optimization.

Vercel Sandbox can now isolate agents as Linux users while exposing selected files through a shared group directory. This gives multi-agent harnesses a simpler permission boundary.

Finance agents need claim-level provenance, explicit uncertainty, consistency checks, surfaced contradictions, and logged approval. Fluent output without those controls will not survive diligence.

Shared agents need a different harness: action-boundary security, selective group memory, per-user privacy, and routing that decides both who receives information and when the agent should speak.

Kepler’s pattern keeps LLMs away from numeric mutation: agents choose sources and steps, while deterministic systems persist, calculate, and reject unverifiable values.

Intuit argues that domain context alone does not create experience. Its approach learns from verified state-action-outcome histories, using frontier models only to propose candidates.

Nubank uses simulated multi-turn traces to evaluate agent changes before production, shortening release cycles while checking simulation results against real data and human review.

Factory’s deployed-engineering model treats agent adoption as workflow design: instrument the path from signal to deploy, build validators, and tie autonomy to measurable business outcomes.

Varick treats enterprise agents as process-reengineering systems: capture how work really happens, encode that context, then automate only the steps whose risk permits it.

Desktop-Delta Bench tests whether computer-use models can verify what changed after an action, exposing a reliability gap hidden by end-task and single-frame scores.

Kubernetes security patches improved sharply when the model received live service dependencies, showing that scanner findings alone omit context needed to avoid breaking workloads.

Netflix’s performance-agent playbook feeds production profiles and exact deployed code to agents, then relies on canaries and engineers to validate proposed optimizations.

Controlled experiments suggest long-horizon agent planning depends on explicit state transitions, some compositional trajectories, and compatible teacher patterns—not atomic skills alone.

ERUnderstand shows vision-language models can recover common ERD elements but often miss rarer schema constructs, so image-to-schema agent workflows still need structural validation.

Vercel’s gateway now supports persistent WebSocket sessions for the Responses API, reducing repeated context transfer during long, tool-heavy agent runs.

DeepSWE uses original long-horizon tasks to reduce contamination and expose coding-agent behaviors hidden by saturated PR-mined suites. Its current task mix still underrepresents some everyday work.

SonderMind turns clinician-reviewed failures into release-gating evals, keeping mental-health guardrails modular and testing false positives, false negatives, category, and timing.

Replace unbounded coding-agent loops with measured, incremental changes: sense code state, choose one bounded task, open a reviewable PR, and pause until a human responds.

Reconstructing production traces as replayable environments lets teams compare agent stacks under fixed conditions and gate releases on task outcomes, cost, latency, and retries.

Uber’s image-editing agent uses routing, iterative QA, golden-set gates, and production feedback to avoid costly edits, hallucinated food, and quality regressions.

Production agent behavior should be improved through a loop of tool design, small eval sets, trace review, and calibrated judges—not prompt tweaks based on one bad run.

Agents can turn production traces, evals, repository context, and observability skills into investigated issues or draft fixes. Human review remains more important as changes grow.

Fixed-rubric judges miss failures that emerge across long, variable agent trajectories. Arize argues for adding agent-based analysis while retaining deterministic and LLM-judge evals.