Harness Engineering

Stellar Colosseum coordinates parallel strategy search, falsification, decomposition, and verifier feedback for long research tasks. Its harness patterns may transfer to agents handling interdependent coding work.

HypoEvolve turns multi-agent scientific work into explicit population updates governed by a genetic algorithm. The result suggests agent collaboration is easier to test when selection and revision are formalized.

A GitHub review agent is rebuilt with progressively less orchestration code, ending as instructions and setup files around general tools. Evals still remain the builder’s job.

Vercel’s data agent improved after replacing prescriptive specialist pipelines with a filesystem, Bash and company-specific context. Eve packages that pattern as conventions.

PostHog treats every context source as part of an agent’s supply chain, scanning at build and use time while reserving blocking decisions for deterministic controls.

A production agent's harness is everything around the model: tools, memory, runtime, isolation, and deployment controls. The practical lesson is to standardize that layer across agents.

Dioxus found that generating more Rust was easy but merging it was not. Agents paid off on bounded research and maintenance work when architecture, real-device tests, and line-by-line review stayed human-owned.

AI SDK now runs GitHub Copilot behind the same HarnessAgent interface as nine other coding harnesses, using an official adapter and ACP connection.

Token counts do not show whether an agent was worth running. Define outcomes and cheap acceptance checks first, then reserve agents for work that is uncertain to execute but relatively easy to verify.

Design-to-code agents can silently introduce security, accessibility, and fidelity drift. Scan both artifacts with deterministic checks, review proposed fixes, and catch divergence before merge.

A spatial agent harness can make parallel work, dependencies, and ownership visible on a shared canvas. Bridging that canvas to coding tools lets its nodes perform real work beyond drawing.

A reported weekend Python-to-TypeScript port shows how agent-driven iteration can make once-prohibitive migrations plausible, provided verification is part of the loop.

Show-Harness puts a compact semantic action layer between VLM intent and robot-specific control, a harness pattern for reusing frontier or small open models across embodiments.

LinkedIn scales a large internal agent catalog through search, schema lookup, and execution rather than exposing every tool at once. Its playbooks add task-specific operating knowledge.

RLMs keep large inputs in a programmable environment, letting a model inspect them with code and delegate subsets to submodels instead of attending to every token at once.

Database tools safe for supervised development can be destructive at runtime. Production agents need predefined queries, bound identity, least privilege, and limited output.

Procedural Graphs externalize an agent’s workflow, guide each action from the relevant subgraph, and refine the structure from failed and working runs. This targets drift and repeated tool mistakes on long tasks.

Full expert-trajectory imitation can break the fit between a weaker model and its evolved harness. Correcting only the failing turn preserved the model’s planning style across seven enterprise tasks.

ExecCritic separates test writing from repair and freezes qualified tests before code changes. On SWE-bench Verified, weak generated tests hurt resolution, while separately trained roles reached 72.6%.

EDGE synthesizes multi-step tool-call training data by retaining only API links verified through live execution. Its 9B model nearly matched an untuned 27B sibling on Korean public-API tasks.

Sentinel-RL keeps topology outside the LLM: a graph encoder and constrained policy choose SOC actions, while the model writes critic-gated narratives for human approval.

Knowledge-work agents need code-like infrastructure around tools: centralized context, action records, verification, enforced permissions, and preflight checks for irreversible work.

Cross-silo agents are primarily a context and privacy-boundary problem. Start with low-sensitivity data, explicit sharing policy, and approval at the moment information leaves a silo.

SafeEvolve turns completed agent trajectories into reversible harness updates and policy training, linking runtime safeguards with learned multi-step behavior.

Agent authorization should scale with stakes and reversibility: logs may cover coding changes, while open-ended payments need scoped, time-bound, independently verifiable mandates.

CordisBench finds that models lose reliability as harness lifecycle interactions grow, while deterministic execution semantics can replace costly reasoning on controlled cases.

This survey separates verbal feedback by when it acts and what it changes, helping agent builders distinguish task specification, test-time guidance, and parameter updates.

For fuzzy requests, an agent should identify the missing constraint with the most decision value, elicit it in the right modality, then choose a response format suited to the task.

Agent checkout needs more than payment tools: enforce user-set limits outside the agent, expose structured merchant data, honor cancellation, and log every decision.

A two-stage ensemble reconciles outputs from string, graph-embedding, and RAG-based ontology aligners. Composition matters: mixed paradigms favor precision, while LLM-only groups more often favor F1.

Vercel found that prose alone produced inconsistent agent-made pages, then paired design.md with fixed CSS primitives and repeatable evals to encode brand judgment.

An AWS demo treats a robot as an agent tool: an LLM selects existing hardware policies while separate agents handle thinking, communication, and voice. The live failures expose the safety gap.

Maersk’s production agents depend less on a clever loop than on executable SOPs, bounded tools, replayable traces, and a correction system shared by experts and engineers.

Navan treats production agents as stateful services needing persistence, composable skills, traces, authorization gates, and trajectory-based tests before multi-agent complexity.

Once agents mutate external state, timeouts mean unknown outcomes. Builders need idempotent tools, bounded retries, scoped credentials, durable traces, and explicit recovery paths.

Agent architectures are expiring quickly. Keep model, search, and orchestration choices replaceable, and evaluate platforms by how well they handle repeated change.

Shared agent skills can turn team know-how into portable workflows, but unmanaged catalogs create duplication and policy risk. Treat skills as versioned, evaluated dependencies.

Figma’s cautious agent adopters expose missing safeguards, not resistance. Use their feedback to improve verification, plans, review size, and the labeling of AI-generated work.

Amazon’s pilot suggests coding-agent gains come from redesigning the workflow: prepare agent-readable context, fast local feedback, scoped specs, and parallel work instead of constant chat.

A study of 8,351 Claude Code plugins finds instruction files and scripts often require coordinated maintenance, treating skills as evolving software rather than static prompts.