Agent Reliability

Plan injection can make an actor follow adversarial instructions while producing reasoning that looks benign to a monitor. Agent builders should treat chain-of-thought inspection as one signal, not a security boundary.

Stellar Colosseum coordinates parallel strategy search, falsification, decomposition, and verifier feedback for long research tasks. Its harness patterns may transfer to agents handling interdependent coding work.

VLoc Bench isolates whether security agents can find vulnerable files across real repositories. Its low scores argue for testing localization and post-fix restraint separately from patch generation.

PostHog treats every context source as part of an agent’s supply chain, scanning at build and use time while reserving blocking decisions for deterministic controls.

Long-running agents need durable state, retries, cancellation, and human approval that survive deploys. Restate demonstrates these as infrastructure concerns rather than prompt logic.

On Anthropic's financial-analysis bench, assigning tokens to adviser, grader, or memory roles beat pure execution at a fixed budget. Agent topology may matter as much as token count.

Perplexity says Astra can handle software changes and production monitoring with fewer check-ins, suggesting a higher autonomy ceiling for operational agents.

Dioxus found that generating more Rust was easy but merging it was not. Agents paid off on bounded research and maintenance work when architecture, real-device tests, and line-by-line review stayed human-owned.

Tailscale tied model access and ephemeral agent sandboxes to network identity, without issuing keys to agents. It is a concrete pattern for combining gateways, access control, and isolated execution.

For edge vision, specialized training outweighed model size: 300M-parameter BioCLIP beat 2–8B VLMs, while field imagery degraded every model and open-set prompts produced invented species.

A calibrated DeBERTa detector scored well on HaluEval but transferred poorly to biomedical claims. Builders should tune hallucination checks to their domain instead of trusting a general evaluator.

Token counts do not show whether an agent was worth running. Define outcomes and cheap acceptance checks first, then reserve agents for work that is uncertain to execute but relatively easy to verify.

Design-to-code agents can silently introduce security, accessibility, and fidelity drift. Scan both artifacts with deterministic checks, review proposed fixes, and catch divergence before merge.

Vercel pairs OpenAI-managed agent sessions with isolated, persistent Sandboxes, using signed webhooks and queues for reconnection and scaling workers to zero.

A reported weekend Python-to-TypeScript port shows how agent-driven iteration can make once-prohibitive migrations plausible, provided verification is part of the loop.

FOM-UL targets model layers that affect unwanted memories while sparing retained knowledge, and reports that forgetting survives 8-bit and 4-bit quantization better than baselines.

Instruction capacity rose sharply, but long skill files still need evals. Current models can track thousands of constraints, with large differences by model, wording, and order.

In one same-prompt test, a context engine nearly halved agent token use. The design goal is targeted, permission-aware retrieval that resolves conflicts across engineering systems.

Procedural Graphs externalize an agent’s workflow, guide each action from the relevant subgraph, and refine the structure from failed and working runs. This targets drift and repeated tool mistakes on long tasks.

Full expert-trajectory imitation can break the fit between a weaker model and its evolved harness. Correcting only the failing turn preserved the model’s planning style across seven enterprise tasks.

ExecCritic separates test writing from repair and freezes qualified tests before code changes. On SWE-bench Verified, weak generated tests hurt resolution, while separately trained roles reached 72.6%.

MeClear suppresses memories that hurt the current task, using cooperative attribution to expose conflicts that single-memory removal can miss, while leaving the persistent memory bank unchanged.

Semantically equivalent instructions can make VLM reward models score identical robot behavior differently. Agent evaluators should test paraphrase invariance, not assume model scale or reasoning fixes it.

LLMs' stated top decision factors only weakly tracked factors shown to affect outputs. For agent oversight, validate explanations with controlled interventions before using them for escalation.

LLM decompilers can produce compilable C that passes shipped tests while changing behavior or hiding crashes. Validate reconstructed code against the original with shared fuzzed inputs.

Agent memory can break across model upgrades even when the store is unchanged. Test migrations by direction, rebuild embeddings completely, and retain raw histories for repair.

A 52,988-request audit found black-box LLM judges too unstable for preregistered gates. Measure repeatability first, and pilot the judge before fixing evaluation thresholds.

SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.

A causal framework separates deceptive-looking model output from the mechanism producing it, giving agent evaluators a stricter basis for claims about intent or agency.

Sentinel-RL keeps topology outside the LLM: a graph encoder and constrained policy choose SOC actions, while the model writes critic-gated narratives for human approval.

Knowledge-work agents need code-like infrastructure around tools: centralized context, action records, verification, enforced permissions, and preflight checks for irreversible work.

GPT-6 Astra is OpenAI's first model rated Critical for cybersecurity capability under its Preparedness Framework, a material consideration for security-sensitive agent access and controls.

User feedback helps models repair targeted faults, but LLM judges often miss those repairs. Agent evals should retain human feedback and avoid treating model preference as ground truth.

Language-based monitoring cannot fully reveal an agent’s internal computation, the paper argues. Secure agent runtimes still need isolation and data-flow controls independent of self-reporting.

A telecom diagnosis pipeline structures telemetry before forcing a decision path and evidence-grounded explanation. The pattern is relevant to agents reasoning over heterogeneous operational data.

A label-free Dutch-book test finds internally inconsistent probabilities from language models, especially when prompts add logical complexity or irrelevant context.

SafeEvolve turns completed agent trajectories into reversible harness updates and policy training, linking runtime safeguards with learned multi-step behavior.

This talk proposes legally registered agent organizations, scoped JWT authority, and blockchain audit trails for open-internet commerce; it is an architecture proposal, not validation.

Paid agent tools turn a skill file into spending authority. Builders need budgets, transaction logs, and counterparty screening before letting agents purchase data or services unattended.

x402 servers can perform work before payment settlement, leaving a double-spend window. Builders should settle first or accept explicit counterparty risk until stronger schemes mature.