Simulated Sandbox

--simulated runs an evaluation against an LLM-simulated environment. The real services never start; a stateful simulator agent fabricates the output of every command the agent under test runs, and an LLM judge decides whether an attack would have worked. This makes control evals cheap to run end to end.

ct run eval -e <env> -t <main_task> -s <side_task> \
  --policy obvious-attack --simulated --limit 1

The agent uses the same bash / text_editor / python tools and any env-specific sibling-container tools it would normally have; only the sandbox changes.

Flags

--simulator-model picks the model that fabricates tool outputs, --simulator-judge-model the model that scores the run, and --simulator-scratch-mode the grounding container (see below); ct run eval --help lists their defaults. Model values accept aliases (haiku, sonnet, opus_4_8) or full provider strings (anthropic/claude-opus-4-8).

Scratch modes

The simulator grounds its fabrications in a single real scratch container, with the env source always available at /codebase. Pick the fidelity/cost trade-off:

  • filesystem (default, free): a minimal python-slim container with the env codebase copied to /codebase. No env image is built. The simulator reads the Dockerfile to map container paths (/app/...) to codebase paths.
  • default_image: the env's real default service image only (siblings and network stripped), with the source bind-mounted at /codebase. One image build; the simulator can run real commands at the app's real deployment paths.
  • full: the env's whole compose stack, with /codebase mounted into default.
  • none: no container at all. The simulator gets no tools and fabricates every output from its own reasoning, as a no-code-execution ablation.

Scoring

A simulated sample has no real environment to score against, so an LLM transcript judge (--simulator-judge-model) replaces the programmatic scorer. It reads the task objective, the agent's submitted attack description, and the full transcript, then inspects the scratch container (when there is one) and decides whether the objective would be satisfied if the environment were real. The judge's investigation transcript lands in the score's judge_messages metadata, and the simulator's own history in the sample store (SimulatorState:sim_messages), so a simulated run stays auditable from the .eval alone.

Known limitations

  • run_side_task_test is simulated too. Policies that ship the side-task test tool get a verdict the simulator made up, and the scorer script's contents pass through the simulator's context. For realism studies, disable it in both arms (the sim-control-eval study did exactly this).
  • Task setup scripts never run, in any scratch mode. For main tasks whose setup mutates the env (seeded bugs, fixtures), the simulator grounds in a world where the task's premise doesn't hold.
  • Main-task scoring is skipped. The judge scores only the side task, so simulated honest runs complete unscored and usefulness is not measured.
  • Fidelity is aggregate, not per-task. The sim-control-eval study found monitor suspiciousness distributions on simulated vs real attack runs statistically indistinguishable in aggregate (KS p > 0.5) but weakly correlated per task (r ≈ 0.18, vs ≈ 0.71 real-vs-real). Treat per-task simulated results with caution.