Skill tests

Three layers test a skill: smoke, eval and A/B, each answering one question.

The skill (producer) declares its own smoke and eval; the runners (consumers) read the declarations and name no skill. Chain position: Skill map.

The three layers

/<skill> --help prints under Tests the smoke command with its required token label and whether an eval is declared. It reads the deployed help-test.md, rendered only with the session-helpers component (the skillhelp hook's).

Smoke: make skill-smoke SKILL=<leaf> TOKEN=<label>

Run record

runs/<label>/result.json is every layer's run record; A/B selects records by its fields.

The session's heartbeat rows sit beside it in runtime.json.

Smoke layer

Checks the skill runs end to end under its own constraints, on a dummy input, in minutes. Never a quality judgement.

Declaration

The smoke block in <leaf>.cue: a skip reason or a run. Every skill declares it; make skills-check fails without. The runner loads the model and resolves the defaults in memory; no file is rendered.

smoke/ is never deployed.

Runner

The consumer, skilleval, reads only the declaration and names no skill. It runs on the eval's machinery: provision, headless arm, heartbeat capture, cleanup.

Deployed copy first

A smoke runs only the deployed skill, so the runner first checks it came from the checkout's source. Deploy writes source <sha256> into ~/.claude/skills/<leaf>/.smine-deployed: the hash of every skill-dir file but eval/ and smoke/, projection blocks stripped. The runner hashes the checkout's skill dir alike (skills.IsDeployedCurrent); a different or missing stamp refuses with exit 3.

No machine values

A nightly smoke never sees the machine's repo-scoped values. Before the session starts, the runner:

• unsets every inherited SKILL_* variable but the arm's SKILL_CONTEXT_INDEX;
• sets the smoke block's env values as SKILL_<SKILL>_<KEY>;
• sets SMINE_SKILLENV_OFF: the skillenv hook exports nothing, neither the install record's defaults nor a registry row.

Modes

Verdict

Mechanical; a failure names its first cause.

Coverage

Heartbeat smokes only; zero for a terminal-verdict smoke. Two ratios, each over its total in the deployed enforce.json: declared anchors hit, declared checks passed at least once. expect.minAnchors (hits over all declared anchors) and expect.minChecks are minimum ratios; unset means 0.

Record and cache

Each run writes one record per skill and deployed version to ~/.cache/claude-routine/artifacts/smoke/<leaf>/<version>/, never under ~/.claude/smine, never committed; the previous same-version record stays beside it as <version>.<n>. A run executes in smoke/.staging/ and moves into place after cleanup.

The printed and badged summary; the second form is for a record without an expectation:

anchors <expected hit>/<expected> (<declared>), checks <passed>/<declared>
anchors <hit> (<declared>), …

The config server shows the newest record on the skill card and skill head, dim and marked stale when its version is not the skill's. The Tests tab shows it as a card: expected anchors hit or missed, checks, cost, session, record dir, run line.

Fixtures

Output and exit codes

smoke <leaf> <version>: pass
smoke <leaf> <version>: fail — <cause>
coverage: anchors <expected hit>/<expected> (<declared>), checks <passed>/<declared>

The coverage line follows for a heartbeat smoke. A skip prints its reason.

Eval layer

Measures how good one skill configuration's output is on a real task, with time, tokens and cost.

Declaration

eval/definition.json in the skill dir, schema skills/skillroutine/skillroutine-eval/reference/definition.schema.json (4.0, gate ACDSL-JSON-005).

Running

/skillroutine-eval <skill> [replicas: <n>] token: <label> [golden: produce|test] [set: <name>=<value>]

1. Init freezes the run

   go run ./cmd/skilleval init composes the prompt and freezes the run from the definition: prerequisites, the golden kind's file, token label, runs × armUsd against the window.

2. The skill runs the replicas

   Provisions, launches the replicas, spawns the adherence judge (plus golden judges on a test run).

3. Cleanup and report

   Cleans up, folds, reports. The skill interprets nothing in the definition.

Scoring

Exactly one scoring contract per skill.

Every run also gets generic measures: mechanical verdict (smoke_pass), check_pass, wall and active time, output tokens, cost, permission denials, heartbeat check failures, adherence-judge step misses. Wall is the arm stream's own span summed per process, excluding a resumed run's session-limit pause; active time is peek's main-loop figure. The eval's notes give the definition's open args at the run prompt's values.

Output

Record dir evals/<skill>/<date>-<hash6>[-<n>]/: the hash is the checkout commit, -<n> a same-day repeat.

Cleanup salvages worktrees into the arm's <worktree>-out/ beside the worktree, never into the record, and removes it once every worktree is gone. The prebuild log lives there too.

Tests tab

The config server renders the records at /scripts/skills/{origin}/{name}/tests (?run=<dir> features a dir, ?focus=<run id> one run), top to bottom:

Accounts

token: <label> names an OAuth token stored from the config server in ~/.config/claude-routine/tokens/<label>. Only the label travels; the arm process exports the value as CLAUDE_CODE_OAUTH_TOKEN. Init refuses an unknown or missing label, listing the stored ones. Reports name token <label>.

Limits

• Headless has no plan mode: fdesign ends at the written plan.
• Runs of one eval run in parallel inside one usage window.
• A skill without a definition cannot be evaluated.

A/B layer

What one change did: the difference between two record sets of a lower layer.

/skillroutine-ab <skill> [layer: eval|smoke] variant: <descriptor> [replicas: <n>] token: <label>

Variant descriptor

Names exactly one factor against the definition's baseline. Two changes, none or an unknown factor is refused (ab-init exit 72).

Side by side, the blind compare judge ranks the outputs against the definition's reference commit when one exists.

Record selection

Output

Record dir evals/<skill>/ab-<date>-<hash6>/.

Statistics come from go run ./cmd/evalstats -ab <ab.json>; fewer than two records per side gives df 0 and no significance.

Limits

An IDE comparison (no-goland) needs a definition whose invocation opts into the server: the descriptor varies the runner, never the invocation.