These pages document master, which is unreleased and in development. The Quick Start installs the latest stable release; anything newer than that tag is marked in the text.

Benchmark Harness

Benchmark-agnostic runner for evaluating AI copilots and LLMs on coding tasks under reproducible isolation, with deterministic scoring and SDD-aware run records.

The harness does not author benchmarks — it runs established public benchmarks (Aider Polyglot, SWE-bench Verified, and BigCodeBench ship today; LiveCodeBench is scaffolded but not yet registered) and one-off custom CCT fixtures through a single adapter contract. CCT’s value is the harness, not the fixtures.

See specs/benchmark-harness/spec.md for the full design and specs/benchmark-harness/plan.md for the phased delivery plan.

60-second quickstart

./scripts/bench is the daily driver. It translates a terse CLI into the JSON config the harness consumes — no hand-authored config, no four-env-var incantation, live progress to your terminal, and a per-attempt timeout so a hung model never blocks the run.

First-time check (no LLM call, no spend):

./scripts/bench

Runs an internal stub-vs-stub smoke (proves the wrapper + harness + report pipeline work), prints which LLM endpoints your environment has, and exits — without making any LLM call or touching your Anthropic account.

Compare two models on a coding task:

./scripts/bench sonnet ollama:qwen2.5-coder:7b

sonnet/opus/haiku use your ambient Anthropic auth; ollama:/lmstudio:/openrouter:/vllm: auto-fill the gateway env vars for you (colons in model tags like qwen2.5-coder:7b are parsed correctly). The wrapper prompts for confirmation before any Anthropic-API-bearing run (all-local comparisons never prompt); add --yes for CI / non-interactive use.

Different models, tasks, or run counts:

./scripts/bench --task python/bowling,go/bowling --runs 5 sonnet opus

A curated comparison without thinking about candidate specs:

./scripts/bench --preset local-vs-cloud

Discovery:

./scripts/bench --help
./scripts/bench --list-presets      # anthropic-tour, local-vs-cloud, cross-language-mini
./scripts/bench --list-providers    # what your machine can actually run (Ollama ≥0.14.0, etc.)

Live progress streams to stderr while a run is in flight (stdout stays clean JSON), so a long run is never silently stuck. A single candidate routes to ./scripts/benchmark run; two or more route to compare. Everything below is the underlying machinery ./scripts/bench drives — reach for it only when you need the raw knobs.

Advanced configuration

The ./scripts/bench quickstart above is the recommended entry point. This section documents the raw ./scripts/benchmark compare JSON-config flow it builds on, the four-ANTHROPIC_* gateway incantation, the claude-code: long form, and the attempt_timeout_seconds knob — kept here for users who need them directly.

Compare multiple LLMs (raw JSON config)

You have a benchmark (say, Aider Polyglot) and you want to know which LLM does best on it. The compare subcommand takes a JSON config listing N candidate LLMs and runs them sequentially under one shared run-dir, then aggregates a Markdown report with mean ± stdev per metric and the calibrated winner verdict.

1. Make sure your backend is authenticated

The harness records which provider an LLM run uses; it never sets the provider. Where it cannot establish that, it records an honest absence rather than a guess — for the codex backend the provider is unestablished (provider_id: null, #281), because that backend passes codex no provider selection and codex resolves its configuration in layers. Configuration happens through the backend’s own gateway env vars. For Claude Code (the only copilot backend in the MVP — see issue #33 for Aider/Codex/GH-Copilot CLI):

Anthropic API (default): claude login once, or set ANTHROPIC_API_KEY in your shell.
Local LLM via vLLM: spin up vLLM with its Anthropic-compatible endpoint (vllm serve <model> --enable-anthropic-api) and use the env block in the compare config to point Claude Code at it.
Local LLM via Ollama: same pattern; Ollama exposes an Anthropic-compatible endpoint on http://localhost:11434.
LM Studio, OpenRouter, etc.: same pattern; whatever URL the gateway serves goes into ANTHROPIC_BASE_URL.

See Claude Code’s LLM gateway docs for the full env-var set.

2. Pick a benchmark and fetch its dataset

./scripts/benchmark list
# {
#   "adapters": ["aider-polyglot", "cct-dogfood-memkernel", "stub"],
#   "backends": ["claude-code", "stub"]
# }

# Aider Polyglot needs a one-time clone of the upstream dataset (pinned by SHA):
python3 -m benchmarks.adapters.aider_polyglot.fetch

./scripts/benchmark list --benchmark aider-polyglot
# Lists every (language, exercise) the adapter exposes.

3. Write a compare config

Copy benchmarks/compare-config.example.json as a starting point. Minimal shape:

{
  "benchmark": "aider-polyglot",
  "runs": 3,
  "task": ["python/bowling", "go/bowling", "rust/bowling"],
  "candidates": [
    { "name": "sonnet",   "backend": "claude-code", "model": "sonnet" },
    { "name": "opus",     "backend": "claude-code", "model": "opus"   },
    {
      "name": "llama3-vllm",
      "backend": "claude-code",
      "model": "meta-llama/Llama-3-70B-Instruct",
      "env": {
        "ANTHROPIC_BASE_URL": "http://localhost:8000",
        "ANTHROPIC_AUTH_TOKEN": "dummy"
      }
    }
  ]
}

Field reference (full schema: benchmarks/schema/compare-config.schema.json):

Field	Type	Required	Notes
`benchmark`	string	yes	Adapter id from `./scripts/benchmark list`.
`runs`	integer ≥1	no (default 1)	Repetitions per task per candidate. Use ≥3 for cross-LLM comparisons; the calibrated winner-rule needs stdev to declare a winner over noise.
`task`	string[]	no	Task filter (omit to run every task). Same shape as `./scripts/benchmark run --task`.
`attempt_timeout_seconds`	integer ≥1	no	Per-attempt wall-clock cap. A timed-out attempt is recorded `result="timeout"` / `scores.timeout=true`, counts as a pass-rate failure, and is flagged separately in the report’s `timed_out` tally (it does not alter the winner calculus). Precedence depends on the entry path. Via `./scripts/bench`: `--attempt-timeout` CLI flag > this field > the wrapper’s heuristic (300s cloud, 600s when `ANTHROPIC_BASE_URL` is set). Via a hand-written config run with `./scripts/benchmark compare --config …` (which bypasses the wrapper): this field, else the backend’s own default / `CCT_CLAUDE_TIMEOUT_SECONDS` env override (no 300/600 heuristic — that lives in `bench.py`).
`candidates[]`	array (≥2)	yes	LLMs to compare. Order is preserved in the report.
`candidates[].name`	string	no	Human-readable label; defaults to `<backend>:<model>`. Must be unique.
`candidates[].backend`	string	yes	Backend family — `claude-code` or `stub`. Do not use the combined `claude-code:sonnet` form (rejected).
`candidates[].model`	string	no	Model id passed to the backend.
`candidates[].env`	string→string map	no	Provider routing env vars, applied only for this candidate’s runs and restored after. Values are passed verbatim into `os.environ`; only key names are persisted to the compare manifest (no secret leakage).

Three things the config deliberately does not support:

Per-candidate task overrides — comparison must be apples-to-apples.
Per-candidate runs overrides — same reason.
Parallel execution — candidates run sequentially. Parallel runs would contend on the polyglot cache, the per-attempt worktree provisioning, and most providers’ rate limits.

4. Run the comparison

./scripts/benchmark compare --config my-compare.json
# {
#   "run_dir": "runs/20260513T140000Z-compare-aider-polyglot",
#   "report_md": "runs/20260513T140000Z-compare-aider-polyglot/report.md"
# }

The harness validates the adapter and every candidate backend before doing any work, so a typo in candidates[5] does not waste candidates 0–4’s wall time. Each candidate’s per-attempt artifacts (run-record.json, score.json, stats.json, diff.patch, transcripts) land in its own nested run-dir under the parent.

5. Read the report

cat runs/20260513T140000Z-compare-aider-polyglot/report.md

The report shows:

One group per candidate (labelled by the candidate’s name), with total attempts, pass rate, and mean ± stdev for elapsed seconds. When two candidates share (backend, model) but differ in env routing — e.g. the same Claude Code model through Anthropic API vs. an OpenRouter gateway — the unique candidate names keep them as distinct groups rather than collapsing them.
A per-task table (which tasks each candidate passed).
Pairwise winner verdicts using the calibrated rule: (Δ > 2σ) AND (|Δ| ≥ threshold) per metric. Below those thresholds the report emits directional, no winner declared rather than calling a winner on noise.
backend_metadata.provider_endpoint per candidate, so a comparison that mixes Anthropic API + a local gateway is visibly marked as such.

6. (Optional) skip the auto-report

./scripts/benchmark compare --config my-compare.json --no-report
# Later:
./scripts/benchmark report --run-dir runs/20260513T140000Z-compare-aider-polyglot/

Useful when you want to inspect run-dirs first before producing the comparison.

Common patterns

Three Anthropic models on Python tasks only:

{ "benchmark": "aider-polyglot", "runs": 3,
  "task": ["python/bowling", "python/book-store", "python/forth"],
  "candidates": [
    { "name": "haiku",  "backend": "claude-code", "model": "haiku"  },
    { "name": "sonnet", "backend": "claude-code", "model": "sonnet" },
    { "name": "opus",   "backend": "claude-code", "model": "opus"   }
  ] }

Anthropic API vs. one local LLM:

{ "benchmark": "aider-polyglot", "runs": 3,
  "task": ["python/bowling"],
  "candidates": [
    { "name": "sonnet-cloud", "backend": "claude-code", "model": "sonnet" },
    { "name": "local-qwen", "backend": "claude-code",
      "model": "qwen2.5-coder:32b",
      "env": { "ANTHROPIC_BASE_URL": "http://localhost:11434",
               "ANTHROPIC_AUTH_TOKEN": "dummy" } }
  ] }

Same model, two different providers (give each candidate a unique name since (backend, model) overlaps):

{ "benchmark": "stub", "runs": 1,
  "candidates": [
    { "name": "anthropic-direct",
      "backend": "claude-code", "model": "claude-sonnet-4-6" },
    { "name": "openrouter-proxy",
      "backend": "claude-code", "model": "claude-sonnet-4-6",
      "env": { "ANTHROPIC_BASE_URL": "https://openrouter.ai/api/v1" } }
  ] }

Status

Phase	Ships	State
0	Contracts, CLI skeleton, schemas, tests	done
1	Stub adapter + stub backend + run orchestration + CI smoke test + report skeleton	done
2	Aider Polyglot adapter, `worktree+venv` tier	done
3	Claude Code (`claude -p`) backend with provider-routing recording	done
4a	Calibrated winner-declaration rule, report verdicts, dogfood subcommand	done
4b	Gate 1 (Polyglot liveness) + Gate 2 (memkernel#3 spec-first verdict-correctness) dogfood execution	pending — maintainer-driven on user’s machine
#33	SWE-bench Verified + BigCodeBench adapters, `docker` isolation tier, codex backend	done
#36	`./scripts/bench` driver — terse `provider:model[@endpoint]` specs, presets, live stderr progress, per-attempt timeout + skip, safe stub-smoke default	done
#41	Aider backend + recorded-transcript verification + apples-to-apples leaderboard procedure	done
#34 (A–C)	Calibrated LLM-judge scoring (`judge`), calibration corpus + validation (`calibration-corpus`, `calibrate`; Spearman ρ gate), rich reports (`report --html --csv` + static-SVG charts)	done; epic #34 has remaining sub-issues
#260	Routing-quality measurement substrate (E1 of #109): hybrid scenario through the unmodified supervisor, control arms + outcome matrix + reuse fingerprint, `quality_fn: v1` report with the control-set gate, write-time redaction	done

Comparison driver (./scripts/benchmark compare) shipped 2026-05-13; the ./scripts/bench terse wrapper shipped 2026-05-18.

CLI

# Daily driver (terse wrapper — see the 60-second quickstart):
./scripts/bench                                                # safe stub smoke + env detection
./scripts/bench sonnet ollama:qwen2.5-coder:7b                 # compare two models
./scripts/bench --preset local-vs-cloud --runs 5               # curated preset, run-count override
./scripts/bench --yes --attempt-timeout 600 sonnet opus        # CI: no prompt, 600s/attempt cap
./scripts/bench --help                                         # usage
./scripts/bench --list-presets                                 # available presets
./scripts/bench --list-providers                               # detected backends/providers

# Underlying harness (what the wrapper builds on):
./scripts/benchmark list                                       # adapters + backends
./scripts/benchmark list --benchmark aider-polyglot            # tasks for an adapter
./scripts/benchmark run --benchmark aider-polyglot \
    --backend claude-code --model sonnet --runs 3              # single (backend, model) run
./scripts/benchmark compare --config my-compare.json           # multi-LLM comparison
./scripts/benchmark report --run-dir runs/<UTC-ts>/            # aggregate (Markdown + JSON)
./scripts/benchmark report --run-dir runs/<UTC-ts>/ --html --csv  # + report.html, SVG charts, CSV exports
./scripts/benchmark dogfood --backend claude-code --model sonnet  # Gate 1 dogfood

# Calibrated LLM-judge subsystem (issue #34 — secondary signal, never overrides deterministic):
./scripts/benchmark judge --run-dir runs/<UTC-ts>/ \
    --judge claude-code:sonnet [--rubric default-v1]           # rate attempts → judge.json per run
./scripts/benchmark calibration-corpus --target-n 50 \
    --axes model,adapter --name set1                           # select runs for human labeling
./scripts/benchmark calibrate --labels set1.labels.jsonl \
    --judge claude-code:sonnet --name set1 [--threshold 0.6]   # Spearman ρ vs human labels

Backend and model are separate flags. The combined --backend claude-code:sonnet form is rejected (see spec.md § v3 correction for why the abstractions were split).

Exit codes (stable across the MVP):

0 — success.
2 — usage error (argparse), unknown adapter / backend, or invalid compare-config.
3 — runtime failure during run / report / compare.
8 — subcommand or feature not yet implemented (Phase N stub).

Adapter contract

A benchmark adapter is a Python module exposing the BenchmarkAdapter protocol from scripts/benchmark_runner/contracts.py:

class BenchmarkAdapter(Protocol):
    benchmark_id: str
    isolation_default: Literal["worktree", "worktree+venv", "docker"]

    def list_tasks(self) -> list[TaskSpec]: ...
    def prepare_task(self, task: TaskSpec, worktree: Path) -> None: ...
    def prompt_for(self, task: TaskSpec, attempt: int, prior: VerifyResult | None) -> str: ...
    def verify(self, task: TaskSpec, worktree: Path) -> VerifyResult: ...
    def golden_patch(self, task: TaskSpec) -> Path: ...
    def max_attempts(self) -> int: ...

The adapter owns “what is a task in this benchmark and how do I check it.” The harness owns isolation, run records, scoring, and reports.

max_attempts() returns 1 for single-shot adapters (SWE-bench-style) and 2 for Aider-style two-shot retry, where prompt_for(attempt=2, prior=...) includes the failed first-attempt test output. Adapters with no golden patch raise NotImplementedError from golden_patch — the runner refuses to run those tasks under the stub backend.

How to add a new adapter

Create benchmarks/adapters/<your-id>/adapter.py implementing BenchmarkAdapter.
Pin any external dataset by SHA in benchmarks/adapters/<your-id>/REVISION.
Expose a module-level register() function that calls register_adapter(<id>, AdapterClass). Do not call register_adapter at module import time — Python imports each module only once per process, so an import-time side-effect would break test isolation when the registry is reset between cases. See benchmarks/adapters/stub/adapter.py for the worked example.
Add the new register() call to scripts/benchmark_runner/_register.py:register_all — the single place where the production set of adapters is wired up.
Add at least one task that the stub backend can satisfy via golden_patch.
Add adapter-conformance tests under scripts/benchmark_runner/tests/.

Backend contract

class Backend(Protocol):
    backend_id: str
    def run(self, prompt: str, ctx: RunContext) -> BackendResult: ...

The MVP ships these backends:

Backend	Phase	Description
`stub`	1	Copies `golden_patch` into the worktree; CI smoke test only.
`claude-code`	3	Spawns `claude -p` headless; parses transcript usage. Provider routing via `ANTHROPIC_BASE_URL` (vLLM, Ollama, LM Studio).
`codex`	#33	Spawns `codex exec --json --sandbox workspace-write --skip-git-repo-check [--model <m>] -` (prompt on stdin), parses the JSONL transcript. Provider routing: the OpenAI Codex CLI selects a provider via its own configuration — `$CODEX_HOME/config.toml` (else `~/.codex/config.toml`) and the layers above it — using `[model_providers.<id>]` blocks (OpenAI cloud, or a local `base_url` for Ollama/vLLM). CCT records the base user config.toml path — `$CODEX_HOME/config.toml`, else `~/.codex/config.toml` — in `backend_metadata` (never secrets), and records the provider as unestablished (`provider_id: null`). It records no selected provider because it selects none: this backend passes codex neither `-c model_provider=<id>` nor `--profile`, so codex chooses from its own layered configuration and the harness has no signal about which provider answered (corrected in #281 — the value had been the first key under `[model_providers]`, in arbitrary dict order). It does not set or parse the config. Pinned & verified: `codex-cli 0.130.0` — see `specs/benchmark-harness/verification/codex.md`.
`aider`	#41	Spawns `aider --yes-always --no-auto-commits --no-dirty-commits --no-gitignore --no-git --no-check-update --no-stream --message-file <attempt_dir>/aider-message.txt [--model <m>] [--edit-format <fmt>]`; captures plain-text transcript. Provider routing: Aider reads credentials from env (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_BASE`); CCT records only presence booleans in `backend_metadata.provider_env_present`, never values, and never sets them. Model string is Aider-native `<provider>/<model>` (e.g. `anthropic/claude-sonnet-4-5`). Pinned & verified: `aider 0.86.2` — see `specs/benchmark-harness/verification/aider.md`. See `### Aider backend` below for addressing, env knobs, and the apples-to-apples procedure.

Not a backend: vLLM, Ollama, LM Studio, OpenRouter — these are providers (LLM HTTP endpoints) that backends route to via the backend’s own gateway env vars. CCT records what it can establish about which provider a run used, and records an honest absence where it cannot — for codex that is provider_id: null (#281). It does not set the routing.

How to add a new backend

Create scripts/benchmark_runner/backends/<family>.py implementing Backend. Export a factory(model: str) -> Backend callable.
Add the register_backend(<family>, factory) call to scripts/benchmark_runner/_register.py:register_all. The <family> is what the user types before the colon in --backend <family>:<model> (or alone when there is no model variant — see stub).
Document any required env vars (e.g. CCT_VLLM_ENDPOINT) in this README.
Add a backend-conformance test that drives run() against a recorded transcript or HTTP fixture (no live network calls in the unit tests).

Aider backend

Aider is a backend, not a bench provider. ./scripts/bench whitelists providers (sonnet, ollama:…, vllm:…) that all resolve to backend=claude-code; it has no backend concept and is not modified. Like codex, aider is addressed only via the lower-level harness (scripts/benchmark run|compare = many backends, each with its own model-string convention; ./scripts/bench = one backend / many providers).

Run a single task:

./scripts/benchmark run --benchmark aider-polyglot \
    --backend aider --model anthropic/claude-sonnet-4-5 \
    --task python/bowling --runs 3

Compare-config candidate shape:

{ "name": "aider-sonnet", "backend": "aider",
  "model": "anthropic/claude-sonnet-4-5" }

Model string format: Aider-native <provider>/<model>, passed verbatim to --model. Examples: anthropic/claude-sonnet-4-5, openai/gpt-4o, openrouter/qwen/qwen-2.5-coder-32b.

Provider credentials: Aider reads them from env (ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, OPENAI_API_BASE). The harness records only presence booleans in backend_metadata.provider_env_present — never the values — and never sets them.

Env knobs:

Variable	Effect
`CCT_AIDER_TIMEOUT_SECONDS`	Per-attempt wall-clock cap (overrides the default 600 s).
`CCT_AIDER_EDIT_FORMAT`	Force a specific edit format (e.g. `diff`, `udiff`, `whole`). When unset, Aider uses its per-model default — the methodology-fidelity choice that keeps numbers comparable to Aider’s published leaderboard. Setting this records `edit_format_forced=true` in `backend_metadata`.

Pinned invocation contract and recorded headless transcript: specs/benchmark-harness/verification/aider.md (aider 0.86.2, captured 2026-05-19; same structure as specs/benchmark-harness/verification/codex.md).

Aider-vs-Aider Polyglot apples-to-apples (maintainer procedure)

This procedure is NOT executed in CI or in this PR. Running the full 225-exercise Polyglot pool requires real API spend and multiple hours of wall time — it is maintainer-scale, like the existing dogfood Gate. The PR ships the documented invariants and exact command; it does not run the leaderboard.

The 9 comparability invariants that make CCT-Aider numbers directly comparable to https://aider.chat/docs/leaderboards/:

Same 225-task Polyglot pool. The aider-polyglot adapter exposes the same 225 (language, exercise) pairs as Aider’s published benchmark. Satisfied by the existing adapter’s list_tasks().
2 attempts per exercise. The adapter’s max_attempts() returns 2. The harness calls run() up to twice per task.
Attempt 2 receives attempt 1 test output. The adapter’s prompt_for(attempt=2, prior=...) appends prior.tests_output to the second-attempt prompt — same information Aider’s own harness passes on retry. Satisfied by the existing adapter unchanged.
pass@2 primary, pass@1 secondary. A task counts as passed if either attempt exits 0 (pass@2); pass@1 is the subset where attempt 1 already exits 0. These are derived by the aggregator from the per-attempt score.json files (which record each attempt’s result) — there are no explicit pass@1/pass@2 fields in the schema; do not look for them there.
Per-model default edit format. CCT_AIDER_EDIT_FORMAT is left unset for leaderboard runs. The harness omits --edit-format from the argv, so Aider selects its per-model default — exactly the behavior Aider’s own leaderboard uses. The resolved format is recorded in backend_metadata.edit_format_resolved for audit.
Resolved edit format recorded per run. backend_metadata.edit_format_resolved captures the format Aider echoed in its output, parsed from the Model: … with <fmt> edit format line (a substring of the Model: line — B3 confirmed Aider 0.86.2 does NOT emit a standalone Edit format: <fmt> line), or None if not echoed. This satisfies the audit requirement without forcing a value.
Model and params recorded. backend_metadata carries model, aider_version, chat_mode, edit_format_resolved, edit_format_forced, and map_tokens_effective. No temperature field — Aider has no CLI temperature flag (Aider-internal via litellm; neither set nor observable by the harness).
T=0 is not truly deterministic. Aider exposes no --temperature CLI flag; temperature is litellm-internal. The harness neither sets it nor can observe it. Run variance from LLM nondeterminism and per-run repo-map variation is accepted — same class as all other backends. The report’s calibrated winner-rule accounts for this.
Scoring = unit-test exit 0. A task is scored passed when the adapter’s verify() exits 0 (language-specific test runner: pytest for Python, go test for Go, etc.). No LLM judge. Identical to Aider’s own leaderboard scoring criterion.

Exact leaderboard-faithful command (run over the full pool, not just a subset; --runs 1 because pass@2 is the adapter’s 2-attempt loop, not --runs 2):

./scripts/benchmark run --benchmark aider-polyglot \
    --backend aider --model <provider/model> --runs 1

Dogfood-subset caveat: benchmarks/adapters/aider_polyglot/dogfood-subset.txt references */leap task ids that are absent from the pinned snapshot (known pre-existing stale data, NOT fixed in this PR — scope discipline). Maintainers running a smoke subset should use --task python/bowling (verified present) or tasks confirmed against the local cache, not the dogfood-subset file.

Known comparability nuance (recorded, not modified): Aider’s own polyglot harness truncates attempt-2 test output to the first 50 lines before passing it to the model. The CCT aider_polyglot adapter appends the full prior.tests_output. This is a recorded, known nuance — the adapter is out of scope for this PR.

--no-git apples-to-apples caveat (tracked): the pinned argv includes --no-git to keep the harness worktree clean for _write_diff (real Aider creates .git/ in a non-git dir, polluting the scored diff). Aider’s published leaderboard runs each exercise inside a git repo, so the repo-map (Repo-map: disabled in our recorded transcript) may degrade on multi-file tasks. Tracked for empirical evaluation in gosha70/code-copilot-team#46 (git-with-cleanup pattern: switch to running Aider with git enabled + a backend finalizer that removes .git/ if the multi-file delta exceeds 5% across 5+ reference tasks).

Isolation tiers

isolation:
  tier: worktree                 # cheap; clean per-attempt directory
  # or:
  tier: worktree+venv
  python: "python3"              # interpreter to use for `python -m venv`
  install_command: "pip install -q pytest"
  # or:
  tier: docker
  image: <prebuilt image ref>     # e.g. swebench/sweb.eval.<arch>.<id>:latest
  container_mount: /testbed       # bind-mount the worktree over the
                                  # image's repo dir (default /workspace)
  dockerfile: <path>              # (build-from-Dockerfile variant)
  build_args: {}

The runner provisions one worktree per attempt under runs/<ts>-<benchmark>-<backend>-NNN/<task>/<attempt>/worktree/. For the worktree+venv tier (Phase 2), it also creates a .venv/ inside the worktree, runs the configured install_command with the venv’s bin/ at the front of PATH, and the verify path looks for worktree/.venv/bin/python and worktree/.venv/bin/pytest before falling back to the host toolchain.

The docker tier (issue #33) provisions a long-lived container with the host worktree bind-mounted at IsolationConfig.container_mount (default /workspace; the SWE-bench Verified adapter sets /testbed, where its prebuilt image keeps the repo + editable-installed deps). prepare_task + the backend edit the host worktree; those edits are live in the container; verify runs the test sets in-container via isolation.run_in_worktree; teardown (release_worktree) is called by the runner in a finally. docker is local-only — never in CI (images are multi-GB); a missing/misconfigured Docker daemon is reported as an environment prerequisite, never a silent skip. The SWE-bench Verified adapter (swe-bench-verified, REVISION-pinned via the stdlib HF rows-API fetch.py; image ref derived at runtime as swebench/sweb.eval.<host-arch>.<instance_id with __→_1776_>:latest; single-shot) is the first real docker-tier consumer; verify applies the instance test_patch then runs FAIL_TO_PASS/PASS_TO_PASS in the image’s testbed conda env. Update procedure: edit REVISION, run python3 -m benchmarks.adapters.swe_bench_verified.fetch.

Per-task isolation is declared by the adapter’s isolation_for(task) -> IsolationConfig. Adapters that don’t vary per task return IsolationConfig(tier=self.isolation_default). The Aider Polyglot adapter overrides this to use worktree+venv for Python tasks (so pytest lives inside the worktree, not on the host) and worktree for the other five languages (which assume the host has go, cargo, gradle, npm, and cmake/make/a C++ compiler installed; this is documented as a host-toolchain requirement).

Run-record layout

runs/<UTC-ts>-<benchmark>-<backend>/
  <task-id-slug>/
    <attempt>/
      run-record.json         # see benchmarks/schema/run-record.schema.json
      score.json              # see benchmarks/schema/score.schema.json
      stats.json              # see benchmarks/schema/stats.schema.json
      prompt.md               # canonical prompt the harness handed the backend
                              #   (output of adapter.prompt_for); sha256 in run-record.json
      effective-prompt.md     # optional: post-wrap prompt the backend actually sent
                              #   (Claude Code's system prompt + user prompt, etc.);
                              #   present only when BackendResult.prompt_path is set
      model-output.txt        # optional: model's raw text response
                              #   (path also recorded in run-record.json)
      transcript.jsonl        # optional, backend-specific structured log
      diff.patch              # post-attempt minus pre-attempt

The prompt.md artifact is always written by the runner before invoking the backend; its path + sha256 are required fields in run-record.json. This means every recorded run is audit-traceable — a sha256 mismatch across runs flags prompt drift before it confounds backend comparisons.

Schemas under benchmarks/schema/ are the single source of truth. Examples live alongside the schema tests at scripts/benchmark_runner/tests/fixtures/schema/.

Determinism

backend_invocation:
  temperature: 0       # default
  seed: <int | null>   # optional; recorded in stats.json

Backends that don’t support seeding record seed: null. The report flags such comparisons “higher-variance,” and the winner-declaration rule (Phase 4) protects against false positives.

Routing-quality evaluation (E1 of #109, issue #260)

A measurement substrate for CCT’s routing arc (#109 increments A–D: registry, Tier-1 failover, Tier-2 delegation/reconciliation, probe-verified recovery). It answers “does the router earn its keep?” with controlled evidence instead of a single uncontrolled number. It is measurement-only (plan decision 10): no new key the router reads, no runtime authority, everything downstream of execution — and that boundary is executable, not documentary (the diff guard in test_routing_eval_injection.TestNoProductionRoutingFileTouched fails the suite if routing-eval work touches a production routing file).

The scenario. presets/hybrid-routing.json declares Aider-shaped tasks, trials with pinned seeds, a deterministic injected event stream (e.g. quota exhaustion at a task boundary), the operator registry, and two special task classes: tier1_only_tasks (negative controls — any Tier-2 execution evidence anywhere in their records fails them as contaminated) and delegate_tasks (driven through the real --delegate/--reconcile packet flow). The scenario executes through the unmodified production supervisor and tick CLI (benchmark_runner.routing_eval.scenario.run_hybrid_scenario + SupervisorRunner); provider behaviour is injected only through the documented test seams, and the #109 §12 arc (initial preference → failover → Tier-2 provisional → probe-verified recovery → independent Tier-1 reconciliation) is proven per trial by verify_arc from harvested routing-run records alone — never from a driver-maintained checklist. Scenario configs are validated by benchmark_runner.routing_eval.scenario_config; benchmark compare refuses them by design (a candidate comparison and a routing-scenario comparison are different contracts).

The arms. A routing result is meaningless without its controls, so the report REFUSES (hard error, not a warning) to emit a cct_router figure without all three, computed for the same preset from the same outcome matrix:

Arm	What it is
`cct_router`	the real supervised run, with failover/delegation/recovery live
`always_best`	per task, the strongest declared profile (tier, then declared priority)
`always_cheapest`	per task, the cheapest profile under the declared cost basis — every trial priced, or the task is `insufficient_evidence`
`oracle`	per (task, trial), the best outcome actually observed in the matrix — the quality ceiling (it bounds quality, not cost)
`oracle_budget`	optional: the oracle under a per-cell cost ceiling

Derived arms are selected from the outcome matrix (schema/outcome-matrix.schema.json): the exhaustive task × profile × trial sweep, integrity-checked for exact Cartesian coverage and bound to a five-component reuse fingerprint (registry digest, preset digest, execution identity, task-set revision, toolchain digest). The router evidence must agree with the matrix on every fingerprint component durably carried by both sides before any comparative figure exists — and the supplied control selections are verified twice over: every selected cell must be identically an eligible cell of the declared matrix, and each selection must equal the reporting boundary’s own recomputation of its declared selector (partial, mis-profiled, or non-optimal selections of genuine cells are refused, not reported). Selector authority is DERIVED inside the reporting boundary, never accepted: build_report takes the registry path and validated config themselves and builds the SelectorContext internally (selector_context_from_registry), so there is nothing a caller could fabricate. The registry passes the production validator itself (rc_validate — grammar AND semantic violations refuse the context), profile tiers/priorities/roles come from its own declarations, the eligibility predicate is the production selector’s (tier membership in the task class’s tier_order — ordinary work routes tier1_only, declared delegate tasks route tier2_preferred — AND the execution role: build for ordinary work, bounded-build for delegation; persisted matrix flags are bound to the predicate, never trusted bare), and the oracle_budget ceiling comes from the validated config. The derived registry and preset digests must match the matrix fingerprint — any other declarations authorize nothing.

The artifacts. Routing-run records (schema/routing-run.schema.json) are harvested from the supervisor’s durable outputs and published by the production entrypoint itself: run_hybrid_scenario resolves the literal credential set from the executed registry’s credential_env references and writes through routing_eval.redaction.write_run_records — the single gate that scrubs at write time (known credential values literally first, then pattern-based defense in depth), refuses any scrub that would alter measurement semantics or fingerprints, re-validates the scrubbed record against the schema, relativizes evidence references against the artifact root, and writes canonical, byte-reproducible JSONL. write_run_records does no secret resolution of its own — its secret_values parameter is required, so a non-production caller passing () is making an explicit declaration, never falling into a default.

quality_fn: v1 is the declared reporting projection — fixed weights, reported BESIDE the full metric vector, never in place of it:

Component	Weight
verifier pass rate (primary)	0.50
lint / typecheck / coverage / security regression	0.075 each
scope violation	0.10
repeated repair cycles	0.05
human intervention	0.05

One global component mask is computed over the complete matrix before any selection: a component unevaluable in ANY executed cell is dropped everywhere with the remaining weights renormalized; a missing primary outcome withholds Q for the whole comparison. Sequence- dependent measures (Tier-2 accepted-unchanged, reconciliation rework ratio, rollbacks) exist only along the router’s stateful run and are not_applicable for derived arms.

Reading the report (routing_quality.build_report): it names its quality_fn version and included components, then per arm gives Q, the full metric vector, cost under the comparison’s single declared cost_basis, and an insufficient map. insufficient_evidence is first-class and contagious — never rendered as zero, never silently dropped: an unpriced trial withholds the arm’s cost, any Q or basis violation withholds the Pareto frontier whole (never partially drawn), and an insufficient required control refuses the report outright. There is no AIQ scalar — a single operating point traces no curve. Routing-eval cost lives in E1’s own artifacts under its own provenance rules (measured from transcript total_cost_usd, or estimated@<price-table>); the harness’s no-dollar-cost rule for backend metadata and scores stands untouched.

Judge & calibration

Deterministic scoring answers “did it build, test, and lint clean?” It cannot rate quality differences inside a passing run — idiomaticity, error handling, test thoughtfulness, security hygiene. Issue #34 adds an LLM judge for those dimensions, but only as a secondary signal: it never overrides the deterministic verdict, and a dimension’s scores are excluded from winner-declaration until the judge is proven to agree with human reviewers on that dimension.

The judge lives in scripts/benchmark_runner/judge/. It mirrors the backend contract: a Judge protocol, a structured invocation record, and a structured-output schema. The invocation record captures the determinism controls the judge backend actually exposes — it does not assume temperature 0 / fixed seed are available. The initial claude-code:<model> judge records temperature_control: "unsupported" and seed_control: "unsupported" (with temperature/seed both null), because the local claude CLI exposes neither knob; future judge backends that can pin those knobs record "supported" instead. Re-run stability is therefore an empirical property — surfaced and quantified by the calibration step below — not a guarantee inherited from a pinned temperature or seed. The judge is driven by a fixed rubric prompt (benchmarks/calibration/rubric-<name>.md, default default-v1) and can be routed at a local gateway via the same provider env vars as any backend — use the hosted_vllm/ LiteLLM provider for vLLM endpoints.

The workflow is gated on calibration so the judge is never “just another opinion”:

# 1. Rate every attempt under a run-dir. Writes judge.json adjacent to
#    each score.json; never touches score.json.
./scripts/benchmark judge --run-dir runs/<ts>/ --judge claude-code:sonnet

# 2. Select a calibration corpus (>=50 task-runs spanning >=2 axes of
#    variation: model / adapter / backend / repeated-runs) for humans to label.
./scripts/benchmark calibration-corpus --target-n 50 --axes model,adapter --name set1
#    -> benchmarks/calibration/set1.corpus.jsonl + set1.meta.json

# 3. A human reviewer labels each row 1-5 per rubric dimension, producing
#    set1.labels.jsonl (one {run_path, dimension, rating, notes} per line).

# 4. Validate the judge against the human labels: per-dimension Spearman ρ
#    + exact-match rate. Dimensions with ρ >= threshold (default 0.6) are
#    "calibrated"; the rest are reported but excluded from winner math.
./scripts/benchmark calibrate --labels set1.labels.jsonl \
    --judge claude-code:sonnet --name set1 --threshold 0.6
#    -> set1.calibration-report.md + set1.calibrated-dimensions.json

Calibration is empirical, not assumed: if a dimension’s judge-vs-human correlation falls below the threshold, that dimension ships as uncalibrated and never declares a winner. Activate calibrated-judge verdicts in a report by passing the calibration output:

./scripts/benchmark report --run-dir runs/<ts>/ --html --csv \
    --calibrated-dimensions set1.calibrated-dimensions.json

Rich reports

report emits Markdown + JSON by default. Two additive flags produce richer output (no JavaScript, no new dependencies):

--html → report.html plus static-SVG charts: chart-pass-rate.svg (always), and chart-judge-histogram.svg + chart-verdict-forest.svg (only when judge data is present). Deterministic and judge scores render in clearly separated sections.
--csv → report-by-model.csv and report-per-task.csv for spreadsheet analysis.

The existing Markdown + JSON reports are never removed — HTML/CSV are strictly additive.

What this harness deliberately does not do

No dollar-cost reporting. The cost_reporting.enabled field in stats.json is permanently false. Cross-provider billing correlation is not solved; estimates would mislead. See specs/benchmark-harness/spec.md § Constraints.
Judge scoring never overrides deterministic scoring. Calibrated LLM-judge scoring has shipped (issue #34 — see § Judge & calibration), but it is strictly secondary. A run that fails its deterministic tests can never win on judge-only criteria, and any rubric dimension that fails calibration (Spearman ρ below the threshold) is flagged “uncalibrated” and excluded from winner-declaration math. The deterministic verdict is always primary.
No custom application fixtures as the foundation. The MVP’s first public adapter is Aider Polyglot precisely so we ride a published leaderboard for sanity. Custom CCT fixtures are one adapter among several in issue #33, never the centerpiece. Carve-out: cct-dogfood-memkernel ships in this issue as Gate-2 verdict-correctness calibration infra against memkernel#3 (a fresh forward-looking spec-first task) — never invoked outside maintainer-driven dogfood, never in CI, never in cross-backend leaderboard reports. See specs/benchmark-harness/spec.md § Constraints.

Running the tests

PYTHONPATH=scripts:. python3 -m unittest discover -s scripts/benchmark_runner/tests -v

CI runs the same discovery on every PR matching the smoke workflow’s path filter (Phase 1+). The hermetic suite stays green on a fresh checkout with stdlib python3 only — no network, no per-language toolchains, no pytest, no Anthropic auth.

Skipped tests and when to re-run them

Skipped tests cover environment-dependent paths that the hermetic CI can’t validate. They drift toward never-running unless explicitly exercised on a documented schedule. Each skipped test below has explicit re-run criteria — run the listed command at the listed trigger.

Test	Skip reason	Re-run trigger	Re-run command
`test_polyglot_adapter.py:test_python_verify_passes_with_example_solution`	Requires `pytest` on host	Pre-merge of any PR touching `benchmarks/adapters/aider_polyglot/` or `scripts/benchmark_runner/backends/claude_code.py`	`pip install pytest && PYTHONPATH=scripts:. python3 -m unittest test_polyglot_adapter -v`
`test_polyglot_adapter.py:test_python_verify_fails_with_starter`	Requires `pytest` on host	Same as above	Same as above
`test_isolation.py:test_real_pip_install_pytest`	Network + pip required	Pre-merge of PR touching `scripts/benchmark_runner/isolation.py`	`CCT_BENCHMARK_INTEGRATION=1 PYTHONPATH=scripts:. python3 -m unittest test_isolation -v`
`test_polyglot_dogfood_subset.py:TestDogfoodSubsetResolvesAgainstRealCache`	Requires real upstream cache	Pre-merge of PR touching `benchmarks/adapters/aider_polyglot/` (any file)	`python3 -m benchmarks.adapters.aider_polyglot.fetch && PYTHONPATH=scripts:. python3 -m unittest test_polyglot_dogfood_subset -v`

Maintainer responsibility: when one of the trigger paths changes, the PR description must include a paste of the re-run output (pass/fail summary). If the PR adds a new skipped test, this table is updated in the same PR — a skipped test without an entry here is a review finding.

Local manual smoke

For exercising the real claude CLI against the Polyglot fixture (local-only, not in CI — see spec.md § “Dogfood gate”):

# Pull the upstream Polyglot dataset (one-time):
python3 -m benchmarks.adapters.aider_polyglot.fetch

# Run one Python task with Claude Code (default = Anthropic API):
./scripts/benchmark run --benchmark aider-polyglot \
    --backend claude-code --model sonnet --runs 1 --task python/bowling

# Same task, routed through a local vLLM gateway:
export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_DEFAULT_SONNET_MODEL=<served-model-name>
./scripts/benchmark run --benchmark aider-polyglot \
    --backend claude-code --model <served-model-name> --runs 1 --task python/bowling

Either run produces a complete record; the second path’s backend_metadata.provider_endpoint reflects the local URL.

Tip: ./scripts/bench sonnet vllm:<served-model>@<endpoint> does the three env exports above for you and probes the endpoint first (Anthropic-shape proxy → used directly; raw OpenAI-only vLLM → an ephemeral LiteLLM proxy is started in front and torn down on exit).

What the harness is for

Moved here from the project README in #214 Phase 3.2.

code-copilot-team ships a benchmark-agnostic harness for evaluating AI copilots and LLMs on real coding tasks under reproducible isolation — so you can answer “which copilot/model is actually better on this kind of work?” with a controlled run record instead of a vibe.

It does not author benchmarks; it runs established public ones (Aider Polyglot, SWE-bench Verified, BigCodeBench) and custom CCT fixtures through one adapter contract. There are two entry points — a terse daily-driver wrapper and the underlying harness CLI:

# Daily driver — safe by default (no-arg run is a free stub smoke + env detection)
./scripts/bench                                          # prove the plumbing, no LLM call, no spend
./scripts/bench sonnet ollama:qwen2.5-coder:7b           # compare two models on a coding task
./scripts/bench --preset local-vs-cloud --runs 5         # curated comparison preset
./scripts/bench --list-presets                           # discovery: available presets
./scripts/bench --list-providers                         # discovery: detected backends/providers

# Underlying harness
./scripts/benchmark list                                 # adapters + backends + judges
./scripts/benchmark run --benchmark aider-polyglot \
    --backend claude-code --model sonnet --runs 3        # one (backend, model) run
./scripts/benchmark compare --config my-compare.json     # multi-LLM comparison
./scripts/benchmark report --run-dir runs/<ts>/ --html --csv  # rich report (HTML + SVG charts + CSV)

What it measures. Deterministic scoring is the primary signal — build/test/lint pass, required files present, elapsed time, token usage — with a calibrated winner-declaration rule (Δ > 2σ AND ≥ threshold) that refuses to call a winner on noise. A calibrated LLM judge (issue #34) adds a secondary quality signal (idiomaticity, error handling, test thoughtfulness, security hygiene), but only after it’s proven to correlate with human reviewers (Spearman ρ ≥ threshold per dimension); it never overrides the deterministic verdict, and a run that fails its tests can never win on judge-only criteria. No dollar-cost estimates are ever reported.

Backends (the agent driving the task): claude-code, codex, aider, plus a deterministic stub for CI. Local models (vLLM, Ollama, LM Studio) are reached as providers through the gateway env vars — ./scripts/bench sonnet vllm:<model>@<endpoint> probes the endpoint and spawns an ephemeral Anthropic↔OpenAI proxy when needed.

Operator docs

Full harness guide, CLI reference, adapter/backend/judge contracts: benchmarks/README.md.
60-second quickstart: benchmarks/README.md § 60-second quickstart.
Routing-quality evaluation (measuring CCT’s router against control arms, #109 E1): benchmarks/README.md § Routing-quality evaluation.
Shadow-mode routing analysis (consuming E1 evidence sets through session analytics + Studio, #109 E2): scripts/session_analytics/README.md § Routing evidence.
Calibration gates + shadow kNN (the #109 §12 promotion conditions made executable, #109 E3): scripts/session_analytics/README.md § Calibration gates.
Design rationale: specs/benchmark-harness/spec.md and the per-feature spec bundles under specs/.