VercelEngineering PostOfficial Source

Run Terminal-Bench and other Harbor evals on Vercel Sandbox

Harbor can run Terminal-Bench and related evals in isolated Vercel microVMs, enabling parallel model comparisons without putting injected credentials inside each sandbox.

Vercel · Sep 17, 2026

Open Source Open Markdown Open JSON

Source Summary

**Harbor 0.22.0 or later** can run Terminal-Bench, SWE-bench, tau3-bench, OSWorld, and other registry evals on Vercel Sandbox with `--env vercel`. Each trial gets an isolated **Firecracker microVM**.

Practical Implication

Move repeatable agent evaluations off a constrained local machine, parallelize trials, and swap the gateway `--model` value to compare providers while keeping the benchmark command stable.

Agent-Ready Context

**Harbor 0.22.0 or later** can run Terminal-Bench, SWE-bench, tau3-bench, OSWorld, and other registry evals on Vercel Sandbox with `--env vercel`. Each trial gets an isolated **Firecracker microVM**.

Move repeatable agent evaluations off a constrained local machine, parallelize trials, and swap the gateway `--model` value to compare providers while keeping the benchmark command stable.

Network policy is enforced outside the VM, and optional secrets are attached only to matching outbound requests. The material does not quantify cost, startup overhead, concurrency limits, or result reproducibility.

Connected Context · Feed7 Judgment

This turns Harbor’s reproducible-evaluation loop into a managed, parallelizable execution option with per-trial microVM isolation and externally enforced network controls. Stable commands can compare gateway models without local capacity being the immediate constraint, but credible comparisons still require documented resource settings because cost, startup overhead, concurrency, and reproducibility are not quantified.

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude InstituteThe integration operationalizes Harbor’s proposed loop of sandboxed trials, outcome verification, and model or harness comparison on managed infrastructure.Quantifying infrastructure noise in agentic coding evalsAnthropic’s measured score sensitivity to resource limits makes the new backend’s unspecified resource configuration and reproducibility material to interpreting cross-model results.How Tailscale built a customer-facing model router on AI GatewayBoth keep credentials and network policy outside isolated agent execution; the eval integration applies that pattern to benchmark trials rather than customer-facing agents.Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOceanSwapping the gateway model under a stable benchmark command supplies the workload-specific evaluation mechanism needed before adopting the candidate’s task-based routing conclusions.

Context Map

benchmarkcodingsecurity#agent-evals#sandboxing#gateways

Uncertainty

Network policy is enforced outside the VM, and optional secrets are attached only to matching outbound requests. The material does not quantify cost, startup overhead, concurrency limits, or result reproducibility.