Harness Engineering

Training on a production harness could adapt an agent to real workflows, but non-replayable interactions break common RL assumptions and make feedback harder to turn into updates.

Post-training gains depend heavily on task selection, rollout quality, and environment design. For many enterprise agents, curated SFT may deliver most of the value before costly RL.

Long-running agents strain context, reward design, and GPU scheduling at once. Compaction and value models can help, but each introduces another optimization trade-off.

Repo-only tasks omit deployments, incidents, traffic, and distributed failures. Training infrastructure agents may require multi-node environments with real resources, despite much higher setup cost.

For agent work without deterministic answers, mine production traces into tasks, build controllable simulators, and use scaled judges to derive rubrics while watching for reward hacking.

MiniMax and Together describe why agent workloads change inference priorities: large codebase context, repeated tool turns, multimodality, and long-horizon RL all demand purpose-built environments and kernels.

Boundary replaces blanket code review with stable architecture rules, agent-generated tests, transcript inspection, execution traces, and human triage—a concrete harness pattern, not a license to skip verification.

Cursor’s cloud-agent adoption grew after it treated the dev environment as agent infrastructure: Linux parity, one discoverable CLI, end-to-end testing, and automated repair.

Production agent velocity depends less on model swaps than on automating the developer loop: isolate parallel work, encode workflows as skills, connect tools, and keep humans as verifiers.

Finance agents need claim-level provenance, explicit uncertainty, consistency checks, surfaced contradictions, and logged approval. Fluent output without those controls will not survive diligence.

Add agents beside deterministic fraud systems, not in place of them: project cross-service events into a semantic layer, send only uncertain cases to specialist agents, then emit the verdict back into the saga.

Treat agent skills as supply-chain dependencies. Nubank scans them locally and in CI with deterministic rules plus LLM review, then gates marketplace distribution and feeds findings into vulnerability management.

Kepler’s pattern keeps LLMs away from numeric mutation: agents choose sources and steps, while deterministic systems persist, calculate, and reject unverifiable values.

Intuit argues that domain context alone does not create experience. Its approach learns from verified state-action-outcome histories, using frontier models only to propose candidates.

Nubank uses simulated multi-turn traces to evaluate agent changes before production, shortening release cycles while checking simulation results against real data and human review.

FactSet treats skills as versioned product features and the harness as their runtime. Routing descriptions, model-specific evals, access controls, and governance matter as libraries grow.

MindForge trains a 27B coding model on source-free program synthesis, raising ProgramBench pass rate from 37.98% to 49.51% and improving across seven unseen benchmarks.

Factory’s deployed-engineering model treats agent adoption as workflow design: instrument the path from signal to deploy, build validators, and tie autonomy to measurable business outcomes.

Varick treats enterprise agents as process-reengineering systems: capture how work really happens, encode that context, then automate only the steps whose risk permits it.

Ramp pairs strict request scoping with agents across the delivery pipeline. Its intake agent reportedly saves about 20% of scoping time, while humans retain final judgment.

Decagon splits deployment between configuring each customer’s agent and turning repeated enterprise requests into product features. The scarce skill is resisting brittle one-offs.

Kepler frames forward deployment as product discovery: observe real work, ship the smallest useful fix, then turn repeated pain and customer vocabulary into durable product leverage.

Netflix’s performance-agent playbook feeds production profiles and exact deployed code to agents, then relies on canaries and engineers to validate proposed optimizations.

Controlled experiments suggest long-horizon agent planning depends on explicit state transitions, some compositional trajectories, and compatible teacher patterns—not atomic skills alone.

Real workflow traces may teach agents more than manufactured tasks, while benchmark scores can shift with the harness. Build pipelines around live work and test across scaffolds.

Replace unbounded coding-agent loops with measured, incremental changes: sense code state, choose one bounded task, open a reviewable PR, and pause until a human responds.

Reconstructing production traces as replayable environments lets teams compare agent stacks under fixed conditions and gate releases on task outcomes, cost, latency, and retries.

Agents can turn production traces, evals, repository context, and observability skills into investigated issues or draft fixes. Human review remains more important as changes grow.

CausalForge pairs a Lean-verified causal-inference library with an autonomous research pipeline and a semantic statement audit. Formal proof checks derivation, not whether the theorem matches the intended claim.

MineValiCoder filters generated tests, refines several code candidates, then scores code and tests together. The pattern targets a core agent risk: unreliable tests steering implementation.

MedGame turns static clinical cases into executable decision stories with separate narrative and orchestration stages, a useful architecture pattern for case-grounded learning agents.

DSPy separates an AI task’s contract from its model, prompt, tools, and harness, letting builders swap or optimize implementations while holding behavior against the same evals.

OpenForgeRL trains agents inside their deployment harnesses by proxying model calls into standard RL stacks and isolating rollouts in Kubernetes containers; error recovery remains weak.

Coding-agent loops can raise throughput without preserving maintainability. Keep human ownership of code, and use upfront alignment to make review affordable instead of trying to automate it away.

A fixed chain of specialist agents lost context and produced incoherent recommendations. The replacement separates deterministic detection, gives one agent end-to-end ownership, and uses subagents only for bounded investigations.

Cursor’s SQLite experiment suggests swarm harness design can matter more than model mix: separate planner and worker contexts reduced coordination churn while cheaper workers handled most tokens.

E3 makes agents estimate task scope, try the minimum viable path, and expand only after verification fails. In a controlled edit benchmark, it preserved task completion while sharply reducing work.

FormalAnalyticGeo shows a reusable synthetic-data pipeline: agents generate problems, compile them into a formal representation, render exact diagrams, measure answers, and retry failed checks.

A proposed agent harness separates planning from execution, represents tool calls as inspectable programs, and requires machine-checkable safety proofs before side effects run.

Osmani argues that agent throughput makes evidence-backed judgment the engineering bottleneck: builders should require explainable diffs, tests, logs, and explicit ownership before shipping.