Evaluation Troubleshooting

Docker Address Pool Exhaustion

Stock Docker's default bridge address pool yields only ~30 networks. Each concurrent sample needs at least one bridge network (basharena uses two, inner_net and outer_net), so running many samples at once exhausts the pool and docker network create fails with one of these messages, depending on whether the request specifies a subnet:

all predefined address pools have been fully subnetted
could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Sandbox startup retries into this error, and every subsequent sample fails the same way until concurrency drops or the pool is enlarged.

Fix: enlarge Docker's address pool by writing to /etc/docker/daemon.json:

{
  "default-address-pools": [
    {"base": "100.64.0.0/10", "size": 24}
  ]
}

Then restart Docker:

sudo systemctl restart docker   # Linux
# or restart Docker Desktop on macOS

This carves 100.64.0.0/10 into 16,384 /24 subnets of 254 hosts each, more than any host can run containers for. Address space is consumed only when a network is created, so an oversized pool costs nothing.

On EC2 or other VPC-hosted machines, do not use 172.16.0.0/12 or 10.0.0.0/8 as the base. 172.16.0.0/12 collides with Docker's own default bridge range, and both ranges commonly overlap a VPC's private subnets (the default VPC uses 172.31.0.0/16), which breaks routing to VPC resources. 100.64.0.0/10, the RFC 6598 shared address space, avoids both collisions.

EC2 fleet workers (ct run eval --ec2) already have this pool: bootstrap.sh and the worker AMI write the same daemon.json before Docker starts. Apply the fix on local machines and manually provisioned boxes that run ct run eval directly.

Container Services Hang at Startup

Services like rsyslog, mysql, or xinetd hang or time out during container startup when the host's high nofile ulimit is inherited by containers. Add ulimits to the environment's compose.yml:

services:
  default:
    ulimits:
      nofile:
        soft: 1024
        hard: 4096