EC2 Fleet (Distributed Evaluation)

--ec2 runs an evaluation on a fleet of EC2 workers instead of local Docker. Jobs are distributed over SQS; each worker runs Docker locally through the standard lb-docker sandbox. ct run mtgen uses the fleet by default (pass --no-ec2 to run locally). Workers boot in about 65 seconds from the pre-baked worker Amazon Machine Image (AMI), or in about 4 minutes from stock Ubuntu 22.04 when no AMI is available.

Authentication

Use AWS single sign-on (SSO) where you can, since it has no long-lived secrets; static Identity and Access Management (IAM) keys also work.

AWS SSO. You need an Identity Center assignment to the fleet account (326120904690) with the LinuxbenchFleet permission set; ask a fleet account admin in #devops. This permission set is enough to run evals. If you already have a profile for that account and permission set, reuse it. Otherwise run aws configure sso, using the organization's start URL from #devops and a profile name of your choice.

List your profiles with aws configure list-profiles, then replace your-profile below with the one you configured:

export AWS_PROFILE="your-profile"
aws sso login
aws sts get-caller-identity

If your default AWS profile already selects the fleet account and permission set, omit AWS_PROFILE. Re-run aws sso login when the selected profile's session expires. In-flight runs refresh credentials only while the SSO session is valid, so log in fresh before a long run.

Control Tower also reads AWS_PROFILE from .env if the shell does not set it; this only selects an existing local profile and does not configure SSO. The AWS CLI and the ct-box scripts do not read .env, so exporting the variable in your shell is the way to select the same profile for all three.

Remove any AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY lines from .env when using AWS_PROFILE. Environment-variable keys take precedence over the profile, so fleet commands refuse to start when both are set.

Static IAM keys. Request a scoped fleet IAM key pair from a fleet account admin, who follows EC2 Fleet Administration, then put it in .env:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Fleet Mode

Prerequisites

One-time setup:

1. API keys: set ANTHROPIC_API_KEY and/or OPENAI_API_KEY (and GH_TOKEN) in .env. The controller passes them to workers as per-instance AWS Systems Manager (SSM) SecureStrings that only that instance can read, and deletes them at teardown.
2. Docker registry auth for private env images: either
   • set DOCKER_ORG_NAME and DOCKER_ORG_PAT in .env, which reach workers like the API keys, or
   • store them in SSM under /linuxbench/docker-org-name and /linuxbench/docker-org-pat (overridable with DOCKER_ORG_NAME_SSM_PARAM / DOCKER_ORG_PAT_SSM_PARAM, staying under /linuxbench/docker-org-). The helper script stores the PAT as SecureString.

To store the Docker auth from your current .env into SSM:

./scripts/put_docker_org_auth_ssm.sh

Usage

# Fleet mode on EC2
ct run eval --ec2 --task-file medium --policy honest --epochs 10

# Scale up with more workers for faster runs
ct run eval --ec2 --task-file medium --policy honest --epochs 10 --ec2-arg max_workers=20

# Cost estimate only (no launch)
ct run eval --ec2 --task-file medium --policy honest --epochs 10 --ec2-arg estimate=true

# Skip confirmation prompt (CI-friendly)
ct run eval --ec2 --task-file medium --policy honest --epochs 10 --ec2-arg auto_confirm=true

# Force a fresh fleet (skip auto-resume)
ct run eval --ec2 --task-file medium --policy honest --epochs 10 --ec2-arg new_fleet=true

Every fleet knob is an --ec2-arg key=value (repeatable); the keys are the fields of Ec2Config in src/control_tower/sandbox/cli.py, listed with their defaults in --help. A sabotage-eval config sets the same keys in an ec2_args: mapping at run level or per attack; the attack's keys win.

Workers run the repo code at a pinned git SHA: HEAD when the worktree is clean, or a tree SHA of the dirty worktree, so uncommitted changes reach workers. Untracked files ship only after git add; the controller warns at launch if any are present.

Selecting Which Settings Workers Pull

Workers run ct settings pull <target> during bootstrap instead of using your local .settings/. By default they pull linuxarena. For an env from a different setting (e.g. basharena), prefix the -e value with the setting:

# Single explicit setting
ct run eval --ec2 -e basharena:basharena -s ...

# Multiple settings — repeat -e per env, deduplicates settings automatically
ct run eval --ec2 \
    -e basharena:basharena \
    -e linuxarena:load_balancer \
    -s ...

# Bare form keeps working; falls back to the linuxarena default
ct run eval --ec2 -e load_balancer ...

Only the fleet controller parses the setting:env form; local runs ignore the prefix. Either prefix every -e or leave them all bare, since mixing the two forms is rejected.

Shipping external code to workers (--extra-src)

Workers get only a git archive of the control-tower checkout, so an external --policy mypkg.foo:bar raises ModuleNotFoundError on the worker. --extra-src DIR also ships DIR as an import root, like PYTHONPATH=DIR:

ct run eval --ec2 --extra-src ~/experiments --policy mypkg.policies:my_policy -e load_balancer -s ...

The flag is repeatable, and each directory needs a distinct basename. It applies only to --ec2, since local runs resolve external refs from the working directory. Build artifacts, caches, VCS metadata (__pycache__, .git, node_modules, .venv/venv, build, dist), and lockfiles are not shipped, so source that lives in a directory named build or dist will be missing on the worker.

External sandboxes on the fleet (--sandbox with --ec2)

--sandbox module:type runs on the fleet too: each worker imports the module from the shipped extra-src trees and runs the sandbox for its own jobs. If the sandbox needs host-side installation (a microVM runtime, a CLI login), name a script inside an extra-src tree with --ec2-arg worker_setup=<basename>/<path>. The worker runs it as root after extracting extra-src and before taking jobs, with the venv at /opt/linuxbench/.venv and the worker's IAM role, so it can read its own SSM parameters. Put environment variables the sandbox needs at run time in /opt/linuxbench-worker-setup.env, which the worker sources after the script. Pick the hardware with --ec2-arg instance_type=...; a microVM sandbox needs /dev/kvm, which only *.metal types expose. The setup script and instance type are part of the fleet ID, so a run that differs only in them does not resume another's results.

ct run eval --ec2 --ec2-arg instance_type=c5.metal \
  --ec2-arg worker_setup=agent-glovebox/evals/control_tower/ct-worker-setup.sh \
  --extra-src ~/agent-glovebox --sandbox evals.control_tower.sbx_gb:sbx-gb -e linuxarena:file_indexing ...

Isolated settings

Fleet jobs automatically use each isolated setting's dependencies and the shipped Control Tower checkout, so no custom worker_setup script is needed. A selection expanded with --all, --just-main-tasks, --just-side-tasks or a task file may span isolated and orchestrator-owned environments; -t/-s select within one setting.

Pricing on workers

The controller uploads the driver's rate table beside the code artifact and each worker prices from it, so a fleet run costs what the same run would cost on the driver; CONTROL_TOWER_PRICING=off on the driver turns pricing off on the workers too.

Scaling

The main knob for throughput is --ec2-arg max_workers=N. Each worker starts at 10 concurrent evals and adapts to rate-limit pressure up to concurrency_per_worker. A fleet first launches up to 24 workers, fewer when max_workers or the pending work is smaller, and adds workers up to max_workers while it is CPU-bound.

Resume

Fleet runs resume automatically. The fleet ID is a hash of the git SHA, policy, tasks, and epochs, so after a controller crash, credential expiry, or network failure, re-run the same command. The controller finds the existing manifest in S3, lists the completed .eval files, enqueues only the remaining work, and launches fresh workers for it.

After a three-minute grace period and once at least five jobs have been seen, the controller aborts the fleet and terminates its workers if every job so far has failed, or if more than 20% of jobs have failed persistently (failed on their third delivery, at least five of them); queues and S3 results are kept for a resume. Each worker terminates itself after five minutes with no work, so workers shut down on their own once the queue drains.

Monitoring Fleet Resources

# Show your running instances and queues
ct cloud status

# Show all control-tower resources (all users)
ct cloud status --all-users

Fleet Streaming Logs

With --ec2, the controller prints only a one-line progress counter ([N/M] X done, …). To follow per-trajectory results live, tail logs/fleet/<fleet_id>/events.jsonl; the startup banner prints the exact path. Each line is one JSON object; types include:

• fleet_start — fleet_id, expected jobs, worker count
• worker_online — instance_id, bootstrap_s
• result — job_id, task, model, per-scorer scores, error, duration_s, instance_id. Emitted as each sample finishes, so concurrent samples on one worker arrive as separate events.
• job_done — job_id, status, instance_id, duration_s, failure_category, error_summary. Emitted at batch end from the SQS done queue.
• fleet_summary — final counts

Recipes:

# Watch failures land:
jq -c 'select(.type=="job_done" and .status!="completed")' \
  logs/fleet/<fleet_id>/events.jsonl

# Live per-task scores as each trajectory finishes:
jq -c 'select(.type=="result") | {task, status, scores, duration_s}' \
  logs/fleet/<fleet_id>/events.jsonl

# Running pass-rate across completed trajectories:
jq -s '[.[] | select(.type=="result")] | {n: length, passed: ([.[] | select(.scores.linux_scorer.value.main_task_success=="C")] | length)}' \
  logs/fleet/<fleet_id>/events.jsonl

# Pull a failed .eval for inspection (workers also upload full .eval at batch-end):
aws s3 cp s3://<bucket>/<fleet_id>/<job_id>.eval /tmp/job.eval
uv run inspect view /tmp/job.eval

Debugging

• Bootstrap logs: uploaded to s3://lb-fleet-results-migrated/{fleet_id}/logs/{instance_id}-bootstrap.log when bootstrap exits, with timing at each step
• EC2 console: link printed after launch
• SSH: via EC2 Instance Connect (emergency debugging)
• Stale warning: the controller warns after 10 minutes with no progress
• Hard timeout: the controller terminates the fleet after 3 hours with no progress while workers are silent (or 1.5 × the sample time limit, if longer), and after 6 hours with no completed sample even while workers are active