Goal Completeness Audit

Audits side tasks to check that agents have enough information to complete them.

Purpose

The audit checks whether the goal, scoring details, and submit tool description give the agent enough information to complete the side task and pass the scorer. It compares what the scorer checks against what the agent is told, to find:

  • Missing information in goal descriptions
  • Undocumented scorer requirements
  • Ambiguous task specifications
  • Information asymmetries

Commands

ct run goal-completeness

Audit side tasks for goal completeness.

Examples:

# Audit all side tasks
ct run goal-completeness

# Audit specific environment
ct run goal-completeness -e web_scraping

# Audit specific side task
ct run goal-completeness -s visit_malicious_website

# Higher thinking budget for thorough analysis
ct run goal-completeness -t 20000

ct run goal-completeness-view

Interactive viewer for goal completeness audit results.

Usage:

ct run goal-completeness-view [RUN_PATH]

Output

Results are saved to data/goal_completeness_runs/<name>_<timestamp>/:

  • results.jsonl - Completeness analysis for each side task, including gaps between scorer requirements and agent knowledge
  • prompt_logs/<env>_<side_task>.json - Prompts and response for each side task
  • run.log - Run log

Analysis Focus

For each side task, the audit asks:

  • Is the goal description clear and actionable?
  • Does the submit tool description match scorer requirements?
  • Are there hidden scoring criteria not mentioned to the agent?
  • Can an agent realistically succeed based on provided information?