Evaluation Output

Eval logs are saved to the logs/ directory by default.

A local eval runs through Inspect's eval_set over --log-dir, so rerunning an identical command reuses its completed log, resumes a crashed one, and runs only the samples still missing. --run-config runs Inspect's plain eval instead and never resumes.

An eval's identity is its task name, task args, model, and limits. For locally built Docker environments, the task args include a fingerprint of the built image contents, so editing an environment starts a fresh eval. External sandboxes and modes that build no environment image have no fingerprint, so after changing one of those, or to collect an independent rollout, point --log-dir at a fresh directory or pass a new --run-name. Expansion selections such as --all get a date-stamped default run name, so resuming one across midnight needs an explicit --run-name.

A selection may list several task files; each is its own eval in the directory, and a combination listed in several files keeps its first file's sample. A run that completed under --no-fail-on-error is complete, errored samples included, so a rerun reuses those samples rather than retrying them. With upload enabled, a rerun uploads the reused log as a new run.

By default, ct run eval exits non-zero and does not convert or upload the run if Inspect ends with an error or cancellation. Pass --no-fail-on-error only when sample errors should be retained without failing the command.

# Custom output location
ct run eval --log-dir my-logs/ --run-name experiment1

Eval files use Inspect's .eval format and can be viewed with:

# Open in Inspect viewer
inspect view logs/my-eval.eval

# Register an eval log as a run
ct runs make logs/attack_eval/example.eval