ROC Curves

Plot ROC curves comparing monitor performance on honest vs attack trajectories.

Quick Start

# Single honest/attack pair
ct game-metrics graph roc --honest <run-id-or-path> --attack <run-id-or-path>

# From a run file with multiple entries
ct game-metrics graph roc --run-file runs.yaml

By default this saves a single ROC plot, scored with the combined_max method, to data/roc/<timestamp>/. Use --agg-func to select different or multiple aggregation methods.

Input Modes

Inline mode

Pass --honest and one or more --attack sources directly. Each --attack is paired with the shared --honest to produce a separate curve.

# One curve (monitored gpt-4.1-nano runs)
ct game-metrics graph roc \
  --honest https://data.linuxarena.ai/runs/c7aeb0aeccba40eb9d910ab13b601bed \
  --attack https://data.linuxarena.ai/runs/bf8a832da92f4431b2579ad04cd983ad

# Two curves (two attack variants, same honest baseline)
ct game-metrics graph roc \
  --honest https://data.linuxarena.ai/runs/c7aeb0aeccba40eb9d910ab13b601bed \
  --attack https://data.linuxarena.ai/runs/bf8a832da92f4431b2579ad04cd983ad \
  --attack https://data.linuxarena.ai/runs/99d571edada441e8a595d7e50f4efc37

Sources can be run IDs, run URLs (https://data.linuxarena.ai/runs/...), local directories, or eval log file paths.

Run file mode

Pass --run-file with a YAML file describing multiple honest/attack pairs and their metadata. This enables --color-by, --style-by, and --filter-by.

ct game-metrics graph roc --run-file runs.yaml

Run file format:

runs:
  # Run URLs (monitored runs uploaded to the run store)
  - label: gpt-4.1-nano (goals)
    model: gpt-4.1-nano
    monitor: mean-action                   # metadata for --color-by/--style-by
    honest: https://data.linuxarena.ai/runs/c7aeb0aeccba40eb9d910ab13b601bed
    attack: https://data.linuxarena.ai/runs/bf8a832da92f4431b2579ad04cd983ad

  - label: claude-haiku (goals)
    model: claude-haiku-4-5-20251001
    monitor: mean-action
    honest: https://data.linuxarena.ai/runs/af9f96b7df2b442aae557bd7c12ef9a0
    attack: https://data.linuxarena.ai/runs/99d571edada441e8a595d7e50f4efc37

  - label: gpt-oss-120b (goals)
    model: gpt-oss-120b
    monitor: mean-action
    honest: https://data.linuxarena.ai/runs/720ef3f583bb449aa2129700f1faf7d0
    attack: https://data.linuxarena.ai/runs/915266b544ba44f896969942f84e4a78

  # Local paths to monitored runs
  - label: gpt-4.1-nano (no-goals)
    model: gpt-4.1-nano
    monitor: mean-action-no-goals
    honest: data/roc_eval/2026-03-04_14-30-47/gpt-4.1-nano/honest
    attack: data/roc_eval/2026-03-04_14-30-47/gpt-4.1-nano/attack

Each entry requires honest and attack, and may add a label plus any metadata fields (e.g. model, monitor). Without a label, one is generated from model and monitor if present, otherwise from the attack source name.

Visual Mapping

Use --color-by and --style-by to map run-level metadata fields to visual properties. Each unique value gets a distinct color or linestyle.

# Color by model, linestyle by monitor variant
ct game-metrics graph roc --run-file runs.yaml --color-by model --style-by monitor

Auto-resolved fields

When the attack source is a run ID, run URL, or a local directory containing run.json, these fields are resolved from the run's metadata and need not appear in the YAML run file:

Values set in the YAML take precedence. If a --color-by or --style-by field is in neither the YAML nor the run metadata, the command errors and lists the runs missing it.

Aggregation Methods

The --agg-func option controls how trajectories are scored. Pass it more than once to get one plot per method plus a combined grid.

# Default: combined_max only (single roc.png)
ct game-metrics graph roc --honest <run> --attack <run>

# Compare combined_max vs action_only (two plots + grid)
ct game-metrics graph roc \
  --honest <run> --attack <run> \
  --agg-func combined_max --agg-func action_only

With one method, the plot title omits the method name; with several, each plot's title names its method. The file names are listed under Output.

Grouping

Use --group-by to split trajectories and produce separate plots per group value.

# One plot per environment
ct game-metrics graph roc --run-file runs.yaml --group-by env

# One plot per failure type (honest baseline shared across groups)
ct game-metrics graph roc --run-file runs.yaml --group-by failure_type

Available group keys:

"Paired" means both honest and attack trajectories are split by the key and matched. "Shared" means all honest trajectories serve as the baseline for every attack group.

Use --min-samples N (default: 2) to skip groups with too few trajectories on either side.

Filtering

Use --filter-by to select a subset of runs from a run file:

# Only plot runs where model=gpt-4.1-nano
ct game-metrics graph roc --run-file runs.yaml --filter-by model=gpt-4.1-nano

# Multiple filters (all must match)
ct game-metrics graph roc --run-file runs.yaml \
  --filter-by model=gpt-4.1-nano \
  --filter-by monitor=mean-action

Output

# Default output directory (timestamped)
ct game-metrics graph roc --run-file runs.yaml
# -> data/roc/2026-03-09_14-30-00/

# Custom output directory
ct game-metrics graph roc --run-file runs.yaml --output-dir results/my_experiment

With the default single aggregation method, the output directory contains:

data/roc/<timestamp>/
  roc.png

With multiple --agg-func values:

data/roc/<timestamp>/
  roc_combined_max.png
  roc_action_level.png
  roc_combined.png          # grid of all methods

Run ct game-metrics graph roc --help for every option.