Control Tower Documentation
Control Tower is an AI control evaluation framework for measuring the effectiveness of safety protocols when deploying potentially misaligned AI agents. It provides realistic software environments where AI agents complete tasks while safety protocols monitor for malicious behavior, and models AI control as a game between a Blue Team (maximizing usefulness and safety) and a Red Team (attempting to subvert safety measures).
Where to start
- Running evaluations on existing settings: start with the Quickstart, then the Guide tab covers running evaluations, monitoring, viewing trajectories, and measuring safety and usefulness.
- Building settings and tasks: start with the Hands-On Guide, then the Build tab covers environments, porting a benchmark, main tasks, side tasks, scorers, sabotage trajectories, and protocols.
Both paths use the objects defined in Core Concepts; the Glossary spells out every abbreviation. To change Control Tower itself, see CONTRIBUTING.md.
Guide
Getting Started
| Page | Description |
|---|---|
| Quickstart | Fresh machine to running evals on public environments with an OpenRouter key |
| Running Evals on Hawk | Submit evals to a shared Hawk cluster and view results |
Running Evaluations
| Page | Description |
|---|---|
| Running Evaluations | Run evaluations with policies, task selection, batching, and replay |
| Sabotage Eval Pipeline | Run the end-to-end honest, attack, monitor, and safety pipeline |
Monitoring
| Page | Description |
|---|---|
| Monitoring Trajectories | Score trajectories with action and trajectory monitors |
Viewing Trajectories
| Page | Description |
|---|---|
| Trajectories | Understand trajectory files, IDs, conversion, upload, and download |
| Web Viewer | View uploaded trajectories and eval results |
| Inspect Logs Viewer | Inspect .eval logs locally |
| Docent | Run Docent locally and manage collections |
Measuring Safety and Usefulness
| Page | Description |
|---|---|
| Safety Curves | Compute safety + usefulness via ct game-metrics metrics |
| ROC Curves | Monitor discrimination plots from ct game-metrics graph roc |
| Pareto Frontier | (Safety, usefulness) Pareto plots from ct game-metrics graph pareto |
Concepts
Concepts and Terms
| Page | Description |
|---|---|
| Core Concepts | Every object Control Tower works with, what it is in code, and where it is documented |
| AI Control Concepts | Understand the core framework |
| Architecture at a Glance | See how settings, environments, tasks, samples, trajectories, and matches fit together |
| Control Tower and Inspect | Which behaviour comes from Inspect and is documented there |
| Glossary | Abbreviations and project terms used across the documentation |
Build
Start Building
| Page | Description |
|---|---|
| Hands-On Guide | Build a toy environment and run your first eval |
| Development Setup | Get your local environment running |
| Live Development | Iterate on envs and tasks with ct live |
Settings and Environments
| Page | Description |
|---|---|
| Making Environments | Create evaluation environments |
| Environment Architecture | Settings, env discovery, and repo layout |
| Porting Settings | Bring an existing benchmark in as a setting |
| Internet Simulator | Simulated external services inside environments |
| Hawk Values Contract | The hand-authored hawk-values.yaml each env needs to run on Hawk |
Tasks and Scorers
| Page | Description |
|---|---|
| Making Main Tasks | Define tasks for agents |
| Making Side Tasks | Design sabotage objectives |
| Side Task Failure Types | Classify side tasks as live or delayed |
| Making Scorers | Write task validators |
| Side-Task Scorer Patterns | Reusable scorer_utils helpers for side-task scorers |
| Auditing Environments | Environment quality audits via ct run subcommands |
| Task Generation and Audit Commands | The ct run commands that generate tasks and audit environments, tasks, and trajectories |
Sabotage Trajectories
| Page | Description |
|---|---|
| Making Sabotage Trajectories | Hand-craft sabotage trajectories (strajs) that accomplish a side task while evading the monitor |
| Sabotage Trajectory Review | The @straj-review bot for sabotage trajectory (straj) pull requests |
| Attack Strategy Guides | Design strategy guides for the strategy-attack policy |
Policies, Protocols, and Monitors
| Page | Description |
|---|---|
| Protocols and Untrusted Policies | How agents are built and blue-team control logic is applied |
| Function Registry | Registering monitors and policies |
Reference
Command Line, Python, and HTTP
| Page | Description |
|---|---|
| CLI Reference | Generated reference for every Control Tower command and option. |
| Python Reference | Generated reference for the registries Control Tower is extended through. |
| HTTP API | Viewer routes, wire conventions, auth, and self-hosting |
Infrastructure
Sandboxes and Storage
| Page | Description |
|---|---|
| EC2 Fleet | Run evaluations on a fleet of EC2 workers |
| Simulated Sandbox | The no-Docker simulated sandbox backend |
| Database | MongoDB collections and access |
Project Structure
| Directory | Purpose |
|---|---|
src/control_tower/environments/ | Environment infrastructure (settings, scorer utils, config) |
.settings/ | Cloned environment repos (gitignored, populated via ct settings pull) |
src/control_tower/trajectories/ | Trajectory format handling |
src/control_tower/ct_runs/ | Orchestration and evaluation logic |
src/control_tower/protocols/ | Protocol definitions |
src/control_tower/metrics/ | Safety, usefulness, ROC, Pareto, and plotting helpers for ct game-metrics |
docs/ | This documentation |
tests/ | Test suite |
apps/docs/ | Docs and environment reference site |