VercelEngineering PostOfficial Source

Run Terminal-Bench and other Harbor evals on Vercel Sandbox

Harbor can run Terminal-Bench and related evals in isolated Vercel microVMs, enabling parallel model comparisons without putting injected credentials inside each sandbox.

Vercel · Sep 17, 2026

Open Source Open Markdown Open JSON

Source Summary

**Harbor 0.22.0 or later** can run Terminal-Bench, SWE-bench, tau3-bench, OSWorld, and other registry evals on Vercel Sandbox with `--env vercel`. Each trial gets an isolated **Firecracker microVM**.

Practical Implication

Move repeatable agent evaluations off a constrained local machine, parallelize trials, and swap the gateway `--model` value to compare providers while keeping the benchmark command stable.

Agent-Ready Context

**Harbor 0.22.0 or later** can run Terminal-Bench, SWE-bench, tau3-bench, OSWorld, and other registry evals on Vercel Sandbox with `--env vercel`. Each trial gets an isolated **Firecracker microVM**.

Move repeatable agent evaluations off a constrained local machine, parallelize trials, and swap the gateway `--model` value to compare providers while keeping the benchmark command stable.

Network policy is enforced outside the VM, and optional secrets are attached only to matching outbound requests. The material does not quantify cost, startup overhead, concurrency limits, or result reproducibility.

Connected Context · Feed7 Judgment

This makes a stable Harbor command usable across isolated cloud trials and gateway models, lowering local-capacity friction for repeated comparisons. It strengthens the empirical evaluation loop, but does not guarantee reproducibility: VM resources, startup behavior, concurrency, cost, and other infrastructure conditions still need to be recorded and controlled.

Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude InstituteHarbor’s rollout methodology provides the operating rationale for this integration: repeat trials, verify outcomes, inspect trajectories, and reevaluate harness or model changes.Quantifying infrastructure noise in agentic coding evalsAnthropic’s measured score swing from resource configurations shows why isolated microVMs alone do not make results comparable unless execution resources are documented and controlled.How Tailscale built a customer-facing model router on AI GatewayTailscale’s external identity, credential, and sandbox controls reinforce the integration’s design of enforcing network policy and secret attachment outside agent execution.Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOceanThe routing demonstration supplies a concrete reason to swap gateway models under one benchmark command, while its uncertain quality comparison reinforces the need for repeatable local evals.

Context Map

benchmarkcodingsecurity#agent-evals#sandboxing#gateways

Uncertainty

Network policy is enforced outside the VM, and optional secrets are attached only to matching outbound requests. The material does not quantify cost, startup overhead, concurrency limits, or result reproducibility.