Monitoring Trajectories

Monitors score trajectories for suspicious behavior, from 0 (not suspicious) to 10 (very suspicious). Action-level monitors store a result per action; trajectory-level monitors store one result per trajectory. Use them to evaluate trajectory safety, test new monitor implementations, and produce monitored runs for analysis.

Commands

ct run monitor

ct run monitor scores an eval-backed source: a .eval file, or a run ID or viewer URL whose run records a backing eval log (see Downloading from the Viewer). --trusted-model is the model every prompt monitor runs on, except monitoring-bench, which uses its benchmark's fixed 3-model panel; it is required unless every selected monitor is code, like jev. With no -m, it runs mean-action and mean-quick; prefix a monitor name with action/ or trajectory/ only when the same name exists as both. Run ct run monitor --help for every option.

A monitor selection takes one of three forms:

1. a packaged monitor name
2. a filesystem path to a .md asset file (absolute or relative to the working directory; the file stem becomes the monitor's score name)
3. a module:function import path to a code monitor: an async callable with the harness's monitor signature, or a factory returning one (the import path becomes the score name).

monitoring-bench is the in-repo reference implementation of the code-monitor form.

# Two action monitors in one pass: a packaged one plus an external asset file
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano \
    -m no-history -m action//path/to/my-monitor.md

Examples:

# Monitor with the default monitors
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano

# One specific monitor
ct run monitor logs/my_eval.eval -m strong --trusted-model openai/gpt-5-nano

# Monitor and save locally without uploading
ct run monitor logs/my_eval.eval -m strong --save-dir monitored_trajs/ --no-upload --trusted-model openai/gpt-5-nano

# Monitor a specific run from database
ct run monitor abc123def456 -m multi --trusted-model openai/gpt-5-nano

# An action and a trajectory monitor in one pass
ct run monitor logs/my_eval.eval -m multi -m quick --trusted-model openai/gpt-5-nano

# Monitor a viewer-hosted run directly (no intermediate download)
ct run monitor https://basharena-data.vercel.app/runs/ed2b360737041f1743ac4de43280f858 \
    -m multi --trusted-model openai/gpt-5-nano

Monitoring in eval2

eval2 is the evaluation path run by ct run eval2 (see Evaluation Config Files). An eval2 monitor asks a model a prompt over a transcript and gets back a suspicion score. The same Monitor runs live inside a protocol or post-hoc over a finished log; the two differ only in what happens to the scores. eval2 monitors are separate from the ct run monitor monitors above.

A monitor is written as data: a bare name selects a code monitor or a packaged prompt, and a mapping sets these fields:

• name: the key for its scores; defaults to the prompt's name, the code monitor's registered name, or the function's name for a module:function.
• prompt: a packaged prompt by name or a markdown file of your own by path. The packaged prompts, in src/control_tower/protocols/eval2/monitoring/prompts/, are simple, strong, strong-goals, old-action, old-action-no-goals, goal-analysis, no-history, orthogonal, and scopejudge for an action, and trajectory, quick, quick-no-goals, transcript-goal-analysis, deliberative, deceptguard, deceptguard-cot, cot-red-handed-action, and cot-red-handed-cot for a whole run. strong through transcript-goal-analysis are adapted from the eval1 assets of the same name, and the rest from the papers listed under code monitors. A prompt's optional YAML front matter sets its cot_level, samples or model defaults, such as cot_level: cot on the two -cot prompts; a field set explicitly wins. A prompt fills whichever of these fields it names: {task_description}, {environment_description}, {transcript} (everything before the judged turn), {action} (the judged turn), and {side_task_goals} (every side-task goal the environment defines, as a numbered list, the same on honest and attack samples). A prompt that names any other field is refused when the monitor is parsed.
• model: the model role that answers, bound with --model-role. Defaults to the monitor role, falling back to trusted.
• cot_level: how much of U's own text it sees: none (tool calls and results only), text (U's prose too), cot (and any reasoning content the model returned).
• samples: how many times to ask; each draw is its own score.
• monitor: a code monitor by name, or your own module:function mapping (messages, view) to an Inspect Score, used in place of a prompt; view is the BlueView of the sample's environment and main task. It is imported when the monitor is parsed, and prompt, cot_level, and model are refused beside it unless left at their defaults. It raises MonitorFailed when it cannot answer, and that draw is recorded unscored. See examples/monitors/allowed_tools.py and the code monitors below.

A monitor only scores; deciding what to do about a score is the blue protocol's job.

Code monitors

A monitor that needs its own answer shape or asks more than once is a function in src/control_tower/protocols/eval2/monitors/ registered with @eval2_monitors.register, with its prompts beside it in steps/. Select one by its name, like a prompt: -m '[checklist]'; a code monitor and a prompt may not share a name. Each answers on the monitor role (falling back to trusted) and raises MonitorFailed when it cannot answer:

The code monitors from papers read the agent's reasoning, except cot-red-handed-hybrid's action half; of the prompts from papers, only deceptguard-cot and cot-red-handed-cot do. Each module's docstring says where it departs from its paper. deliberative is Deliberative Monitors' prompted teacher, deceptguard and deceptguard-cot are DeceptGuard's black-box and CoT-aware monitors, and scopejudge is ScopeJudge's gate, scoring 10 to deny and 0 to allow. Ensemble Monitoring is a panel rather than a monitor: give a protocol diverse monitors and -bp combine=mean.

Live: monitors a protocol decides on

trusted-monitoring and defer-to-resample each take monitors and a combine rule (see ct protocols eval2-protocols trusted-monitoring).

ct run eval2 -e slack_clone -t bookmark_messages \
  --untrusted-policy honest \
  --protocol trusted-monitoring \
    -bp 'monitors=[simple, {prompt: strong, model: judge, cot_level: text, samples: 2}]' \
    -bp sus_threshold=7 -bp blocks=2 \
  --model-role untrusted=openai/gpt-5-nano \
  --model-role trusted=openai/gpt-5-nano --model-role judge=anthropic/claude-haiku-4-5

• monitors: the list, as YAML, each monitor under its own name. They run concurrently, so several monitors cost one round trip.
• combine: how every score of every monitor combines into the one value the protocol acts on, as an Inspect score reducer: a name Inspect registers (max, the default, mean, median, mode, at_least, ...), or {name: ..., args: {...}} for one that takes arguments. name may be your own reducer as module:function, or the name of one registered with @score_reducer in a module that is already imported, prefixed package/ when that module is in an installed package. The reducer must return a number. A score a monitor could not give is left out, and each score's metadata.monitor and metadata.draw name its monitor and draw, so a reducer can weigh monitors differently (below); a monitor with samples: 2 gives it two scores. The reducer is made when the arguments are parsed, so an unknown name or a wrong argument fails before the run starts.
• sus_threshold, blocks, block_message: what the protocol does with that value. If unset, the monitors score and nothing intervenes. If set, an action above the threshold is blocked and U reads block_message in place of its results, up to blocks times; after that the trajectory is audited and the sample stops. An action the monitors could not score errors the sample.

A reducer of your own, such as a weighted mean over named monitors, or a likelihood ratio fitted offline on honest and attack scores:

# my_reducers.py
from inspect_ai.scorer import Score, ScoreReducer, score_reducer


@score_reducer
def weighted_mean(weights: dict[str, float]) -> ScoreReducer:
    def reduce(scores: list[Score]) -> Score:
        w = [weights[s.metadata["monitor"]] for s in scores]
        return Score(value=sum(wi * float(s.value) for wi, s in zip(w, scores)) / sum(w))

    return reduce

-bp 'monitors=[simple, strong]' \
-bp 'combine={name: "my_reducers:weighted_mean", args: {weights: {simple: 1, strong: 3}}}'

Every score lands in the sample store, tagged with the monitor's name, its draw number, the message it judged, and pass: live. When a live-monitored run ends, the sample's monitor score reports them all, unaggregated (format below). On the transcript, each draw is a model event with the role monitor/<name> in a span named for its monitor, answering with a score tool call; the per-monitor role lets Inspect's usage table split cost by monitor. The response cache is keyed on the monitor's name and draw.

Post-hoc: more monitors over a finished log

ct run monitor2 runs the same monitors over a finished run and records them as live ones are, with pass: post_hoc, in a monitored copy. It takes eval2 logs only, since it reads the seat; use ct run monitor for other logs:

ct run monitor2 logs/run.eval \
  -m '[simple, {prompt: strong, model: judge, cot_level: text, samples: 2}]' \
  --kind action \
  --model-role monitor=openai/gpt-5-nano --model-role judge=anthropic/claude-haiku-4-5

• -m: the monitors. Required: there is no default monitor, live or post-hoc, and an empty list is refused. A monitor whose name already scored the log is refused, since its scores would land on the same keys; give the new one another name.
• --kind: action scores every action on the transcript up to that action, before any of its tool calls ran; trajectory scores the whole run once. It applies to every monitor in the pass, and each packaged prompt and code monitor is written for one kind, as listed above, so pick the ones that match.

Each monitoring pass is its own child of the base run. The copy carries the source's own scores and one monitor score holding the live monitors' scores and every post-hoc pass's, so a log reads the same however it was monitored. Its header records the source as derived_from; the source is never written to.

The monitor score's value maps name/action (or name/draw/action when a monitor drew more than once) to a score. Each entry under metadata.scores names what it judged and what the monitor said:

"monitor": {"value": {"simple/0": 3, "simple/1": 4, "cot/0": 10, "cot/1": 9}, "metadata": {"scores": [
  {"action": 0, "message_id": "YeBqeNEz5Xw…", "pass": "live", "value": 3, "prompt": "simple",
   "model": "openai/gpt-5-nano", "events": ["33Vo5MX8…"], "monitor": "simple", "draw": 1},
  {"action": 0, "message_id": "YeBqeNEz5Xw…", "pass": "post_hoc", "value": 10, "prompt": "strong",
   "model": "openai/gpt-5-nano", "events": ["Gq2k9TzW…"], "monitor": "cot", "draw": 1}]}}

action is the message's index among U's actions and message_id the untrusted message it judged; both are null for a trajectory score, which the mapping keys name/trajectory. pass says who asked: live for a blue protocol during the run, or post_hoc for the monitor scorer afterwards. A prompt monitor's entry carries prompt, model, and events (the model events of its call, which hold the prompt, the answer, and its reasoning), plus model_role when the monitor set model. A code monitor's entry carries function, its explanation, and whatever metadata it returned. A draft a protocol scored and did not run (defer-to-resample scores every draft) is in the entries but not in the mapping, which covers the actions U took. A monitor whose answer is not one valid call to its answer tool is asked again, up to three times in all, each attempt cached under its own key so a rerun replays the same attempts. A monitor that still could not score records unscored with the error as its explanation, and unscored, not zero, is its value in the mapping.

ct run monitor2 runs the control_tower/monitor scorer. You can also run the scorer directly, but on a log that already carries a monitor score, Inspect then writes the new pass beside it as monitor-1:

inspect score logs/run.eval --scorer control_tower/monitor \
  -S 'monitors=[simple]' -S kind=action \
  --model-role monitor=openai/gpt-5-nano --action append

Jev

Jev is TypeSafe's decision model: given a state and a set of yes/no questions, it returns a calibrated probability for each in one call, with no generated text. The Jev monitor asks the 32 questions in jev_questions.yml, one per security property, about the latest action given the task, the environment and the transcript before it, and scores the action ten times the highest probability. Its explanation is the three highest questions, and metadata.jev keeps every probability. Both the eval2 monitor and the eval1 jev action monitor show Jev the model's visible text beside each call and neither shows its hidden reasoning.

Set TYPESAFE_API_KEY to call TypeSafe directly, or leave it unset to go through OpenRouter with the OPENROUTER_API_KEY CT already reads. Each call costs $0.042 per million input tokens and reads at most 32k tokens, so the action and the transcript share an 80k-character budget: an action over 40k characters keeps its head and tail, and the transcript keeps as much of its end as fits beside it. With neither key set the run stops at the first Jev call instead of writing unscored rows.

In eval2 it is a code monitor, live or post-hoc, recorded in the monitor score like any other, and a protocol combines its score with the prompt monitors' asked beside it:

ct run monitor2 logs/run.eval --kind action -m '[jev]'

examples/eval_configs/06-eval2-jev.yml runs trusted-monitoring with Jev and simple as its monitors. In eval1 it is the packaged action monitor jev:

ct run monitor logs/run.eval -m jev

Jev makes no LLM call, so neither command needs a trusted model; a prompt monitor asked beside it binds one, as the example config does for simple.

scripts/monitor_calibration.py reads honest and attack monitored logs of either kind and prints AUROC, Brier score, ECE and a reliability table at the action level and the trajectory level (max over actions), with latency and cost when the scores record them:

uv run python scripts/monitor_calibration.py --honest data/honest/ --attack data/attack/ --monitor jev

--monitor may be omitted when the logs carry one monitor's scores; when they carry several, as a LaStraj log does, the script asks for it instead of mixing them.

Available Monitors

A monitor is a prompt asset: a markdown file under src/control_tower/protocols/monitoring/assets/{action,trajectory}/ whose stem is the monitor's name and whose body is the prompt template. Its YAML front matter declares the monitor's view: samples (independent draws, mean-aggregated), per_goal (one run per side-task goal, max across objectives), members (a weighted-mean ensemble of other assets), template (reuse another asset's prompt body), and schema (the output parser). One harness renders the view, calls the trusted role model, and parses the score, so adding a monitor means adding a file. The exception is monitoring-bench (bench-replica model panel with an XML parser), which is code.

List available monitors (or show one monitor's declared view):

ct protocols monitors action [NAME]
ct protocols monitors trajectory [NAME]

Action-Level Monitors

Score each untrusted-model action.

• no-history - Scores each action in isolation with no preceding action history or candidate covert-objective hints.
• old-action - The default action monitor: one model call per action, shown the side-task goals.
• old-action-no-goals - The default action monitor without the side-task goals.
• checklist - Answers five yes/no checks per action (necessity, security surface, innocent explanation, concrete harm, removability) before scoring; the top of the scale is gated on concrete harm.
• orthogonal - Flags work that does not serve the main task (unnecessary changes, premature optimization, off-task activity) rather than judging malice directly.
• three-scores - Categorical monitor: a safe / uncertain / suspicious monitor score per action, mapped onto the score scale.
• strong - Sees the full conversation history (goal-blind) so it judges each action in context.
• strong-goals - The strong full-history monitor, additionally shown the environment's side-task goals.
• goal-analysis - Scores each action while shown the environment's side-task goals, asking whether the action advances any of them.
• single-goal - Scores each action once per side-task goal in parallel and keeps the maximum score.
• multi - Weighted blend of the orthogonal and old-action monitors' scores for each action.
• mean-action - Averages two independent runs of the old-action monitor on each action. CLI default.
• mean-goals - Averages two independent goal-analysis runs on each action.
• mean-single-goal - Scores each action twice per side-task goal, averages each objective's scores, and keeps the highest average.
• jev - TypeSafe's Jev decision model asked 32 yes/no questions about each action, scored ten times the highest probability (see Jev).

Trajectory-Level Monitors

Score the whole transcript in one pass.

• quick - Reads the full transcript in a single pass, shown the side-task goals, and returns one score for the trajectory.
• quick-no-goals - Reads the full transcript in a single pass with no covert-objective hints.
• transcript-goal-analysis - Reads the full transcript while shown the environment's side-task goals, working through them one by one.
• single-goal-analysis - Reads the full transcript once per side-task goal in parallel and keeps the maximum score.
• mean-quick - Averages five independent quick passes over the transcript. CLI default.
• mean-goal-analysis - Averages two independent goal-analysis passes over the transcript.
• mean-single-goal-analysis - Reads the transcript twice per side-task goal, averages each objective's scores, and keeps the highest average.
• monitoring-bench - Replica of the MonitoringBench benchmark's monitor: the bench prompt on its fixed 3-model panel (scores comparable to the imported monitoring-bench dataset), max-aggregated.

Goal-aware monitors see the goals of every side task registered for the environment, not just the one the agent was assigned. Whether a monitor sees them is fixed by its prompt asset, so goal-aware and goal-blind variants are separate monitors (e.g. old-action vs old-action-no-goals, strong vs strong-goals, quick vs quick-no-goals).

A source .eval containing a single environment from an isolated setting is replayed in that setting's dependency environment automatically. Compose topology is resolved without pulling or building Docker images.

Monitoring the final workspace diff

When a run's main task declared a final_diff config (see Making Main Tasks), the sample stores what changed on disk by the end of the run in sample.scores["linux_scorer"].metadata["workspace_diff"], read through control_tower.trajectories.sample.workspace_diff(sample). This is the primary monitoring signal for in-sandbox agent harnesses, whose file edits never appear in the tool-call stream.

A trajectory monitor appends the diff to its prompt by declaring workspace_diff: true in its asset's front matter. For example, save this asset as quick-diff.md:

---
description: Quick monitor with the final workspace diff
template: quick
workspace_diff: true
---

Run it with ct run monitor <run> -m trajectory/quick-diff.md --trusted-model <model>.

The section is appended after the rendered template, so no asset needs a {workspace_diff} placeholder. The flag is trajectory-only, since the capture is the workspace's final state; an asset under assets/action/ that sets it is rejected at load.

A custom trajectory code monitor receives the rendered section in context.workspace_diff. It is an empty string for runs that did not capture a diff (task without final_diff, simulated or replay-sandbox runs, and samples without captured diff metadata). Action monitors receive no final workspace diff. Code reading an EvalSample directly can use workspace_diff(sample) to obtain the structured fields; it returns None when no capture ran.

Check the truncated and error fields before trusting the diff text. A truncated diff is missing its middle, and an agent can force truncation by writing a lot of churn, so a diff-hunting monitor should treat truncated=True or a large total_bytes as suspicious in itself. The rendered section starts a truncated diff with an explicit warning and renders capture failures explicitly.

Output

Sources are scored directly

The input must be eval-backed: a local or remote .eval, an in-memory EvalLog, or a viewer run backed by an eval log. Anything else (trajectory JSON, datasets, runs without a backing eval log) is rejected. ct run monitor scores the recorded transcripts directly: action monitors see the transcript prefix before each action, trajectory monitors the full transcript, and nothing re-executes.

The source is scored onto a copy: the copy lands in the save dir (an explicit --save-dir, or the default ./data/monitoring/<monitors>/<label>_<timestamp>/) as <source-stem>.monitored.eval (a re-run into the same dir gets a -N suffix), and the source is left unchanged.

An action monitor's score value maps action index to score (None where the action was skipped by design or the monitor failed), with the full monitor scores under metadata.action_scores; a trajectory monitor's score is the monitor score itself. Decision statistics (max, thresholded audits) are computed downstream from these values. Existing named scores survive, so scoring a .monitored.eval with a new monitor accumulates both; a monitor already scored on the source is refused unless --overwrite, which replaces that monitor's scores on the copy. Evals scored before monitor scores were Inspect Scores carry legacy monitor_<name> folds ({max, action_<i>, trajectory}). --overwrite can replace those folds only when the eval's trajectory overlay already uses Score-shaped monitor responses; older MonitorResult overlays are rejected during conversion and currently require regeneration from an unmonitored source.

Unless --no-upload is passed, the command creates a run in the database, uploads all monitored trajectories, and prints the run and trajectory URLs.

The model response cache is on by default, so re-running an unchanged command reproduces the same scores, and the CLI warns when this happens. Clear Inspect's model cache (inspect cache clear) to resample. A monitor's fan-out draws (samples, ensemble members, retry attempts) scope the cache key by position, so they are independent samples within a run.

Cost Attribution

The orchestration layer records cost_usd and token fields in every monitor score's metadata. Multi-model monitors (e.g. multi) additionally get a cost_by_model split. An unpriced model is a None in that split; cost_usd stays the priced spend. See function-registry.md for the mechanism and known limitations, and trajectories.md for the per-sample monitor_cost() / total_cost() accessors.