Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang
Repo-only tasks omit deployments, incidents, traffic, and distributed failures. Training infrastructure agents may require multi-node environments with real resources, despite much higher setup cost.
Current coding tasks often keep agents inside one repository for **50–100 turns**. Emulated instead packages organizational context, deployments, live-service constraints, network failures, corruption, and clock skew into training environments.
If an agent will operate production infrastructure, evaluate the entire operational loop rather than only its patch. The proposed **multi-node sandbox** provisions real cloud resources so agents can practice deployment, rollback, observability, and blast-radius decisions.
Current coding tasks often keep agents inside one repository for **50–100 turns**. Emulated instead packages organizational context, deployments, live-service constraints, network failures, corruption, and clock skew into training environments. If an agent will operate production infrastructure, evaluate the entire operational loop rather than only its patch. The proposed **multi-node sandbox** provisions real cloud resources so agents can practice deployment, rollback, observability, and blast-radius decisions. This remains an emulation: customer traffic and failures that appear only at scale are still missing. Some stacks take **hours** to provision, making rollout duration, cost, reproducibility, and the remaining simulation-to-reality gap unresolved constraints.
This expands coding-agent evaluation from repository completion to operational competence across deployment, rollback, observability, and failure containment. It strengthens prior environment- and harness-centered approaches, but also narrows their production claims: even realistic cloud sandboxes omit scale-only failures and introduce substantial provisioning cost, duration, and reproducibility constraints.