LLM-Judged Specs

Codify persistent assertions about the codebase that an LLM judge can check by reading. Specs defend a class of properties that are hard to check with code but easy to check by reading: documentation freshness, design-convention adherence, accuracy of cross-references, skill correctness. Mechanical checks still belong in *.test.js; probabilistic, distribution-level properties belong in evals.

Spec files live alongside what they describe (*.spec.md), just like unit tests and evals.

Vocabulary

• Spec — the *.spec.md file.
• Assertion — a single statement of truth about the system, checked by a judge.
• Judge — an LLM that reads the relevant artifact and renders a verdict against an assertion.
• Runner — the tool (spec.js) that reads a spec, runs each assertion through a judge, and reports the verdicts.

How it works

You write a spec file. The runner does the rest: discover specs, parse them into assertions, judge each assertion, and report the verdicts.

node spec.js skills/testing/specs.spec.md

Orchestration is plain code; only the judging is an LLM. That keeps the runner deterministic, testable, debuggable, predictable in cost (one judge call per assertion), and CI-friendly. This matches how LLM-as-judge frameworks work in general: code orchestrates, the LLM judges.

1. Discover

By default the runner globs **/*.spec.md from the repo root. Pass explicit paths or globs to narrow the run; a bare directory means "every spec beneath it." node_modules is always excluded.

2. Parse

Each spec is a markdown document. Every level-2 heading (##) is one assertion, stated as a single descriptive sentence; the prose beneath it (up to the next ##) is the context the judge uses. Content above the first ## (title, intro) is not an assertion. The runner derives a slug from each heading when it needs a stable ID.

3. Judge

For each assertion, the runner spawns a judge — an LLM with read access to the codebase — and hands it a single assertion plus a surrounding prompt. The runner does not resolve or supply target files; the judge reads whatever the assertion implicates and renders a verdict.

spec.js → JUDGE: "Here is one assertion. Read what it implicates, then rule."
       ← { "reasoning": "...", "verdict": "pass" | "fail" | "refused" }

The judge returns a structured verdict, not freeform prose:

{
  "reasoning": "string — the judge's rationale, generated first",
  "verdict": "pass" | "fail" | "refused"
}

reasoning comes first and is always required — for pass, fail, and refusal alike — so the judge reasons before it rules and every outcome carries an audit trail. A missing or null verdict is not a refusal; it is a malformed response the runner treats as an error and retries.

4. Report

The runner reports the three verdict buckets — passed, failed, and refused — plus an errored bucket for assertions whose judge never returned a valid verdict (a malformed response that survived its retries, or an unreachable model). An error is operational, not a verdict: it means the runner couldn't get an answer, so the assertion is neither confirmed nor refuted. One errored assertion does not abort the run; the other verdicts still complete and report.

The runner prints a markdown report to stdout and, with --report/--json, writes it to disk. The process exits non-zero when any assertion failed or errored; refusals are surfaced but, on their own, do not change the exit code.

The three verdicts

Verdict	Meaning	What to fix
pass	Judged, and the assertion holds	nothing
fail	Judged, and the assertion does not hold	the code or the docs
refused	Could not be judged as written	the assertion

Folding refusal into fail is wrong. A failed assertion means "judged and false" (real drift); a refusal means "the assertion couldn't be judged" (the spec is underspecified). They point the author at different fixes.

The judge's strictness is author-controlled. The assertion's prose sets the bar, and the judge follows it literally rather than importing its own disposition. When an assertion is too ambiguous for the judge to find the pass/fail line in the prose, that is an authoring defect: the judge refuses rather than guessing. The fix is to sharpen the assertion, not to tune the judge.

One judge call per assertion

The runner makes one judge call per assertion, never one per file. Isolating each assertion prevents cross-contamination between verdicts (a hard failure on one assertion souring the read of the next), eliminates order and anchoring effects, contains errors and retries to a single assertion, and yields one focused explanation per verdict. Per-assertion calls fan out and run concurrently, so the cost is mostly token overhead, not wall-clock latency.

Do not batch multiple assertions into one judge call to save tokens. If two assertions seem to require shared judgment, that is a sign they should be a single assertion, or the spec should be re-factored.

Consensus

By default the runner makes a single judge call per assertion. Consensus is a run-level flag, off by default: --consensus N samples the judge N times per assertion and settles on the majority verdict. --agree K sets how many of those samples must agree; the default is a strict majority. When no verdict reaches the threshold, the assertion is reported as refused — the judges could not confidently settle it. Whether consensus should be configurable per-assertion is deferred.

Naming and placement

• *.spec.md is the file extension; the filename prefix names the concern (deployment, merge, skill).
• When a spec describes a specific file or folder, name it after that artifact (src/router.js → src/router.spec.md; skills/tldr/ → skills/tldr.spec.md).
• Always co-locate a spec with what it describes. Do not collect specs under a top-level specs/ directory.
• A spec may hold many assertions, organized as a test file would be.

project/
  src/
    router.js
    router.spec.md          # assertions about router.js
  skills/
    tldr/
    tldr.spec.md            # assertions about the tldr skill

Find all specs: glob **/*.spec.md.

Writing assertions

Each assertion is a level-2 heading stating the claim as one descriptive sentence, followed by prose the judge uses as context. The heading is both the human-readable description and the machine identifier.

# Skill documentation spec

## Body is internally consistent with the description

The SKILL.md body must not contradict claims made in the frontmatter
description, and must accurately reflect the skill's stated purpose.

## Examples illustrate the description

Each example in the body should serve as a clear illustration of when the
skill applies, not contradict it.

An assertion is well-authored when its prose makes the pass/fail line unambiguous to a competent reader. If the judge can't tell where that line is, it refuses — sharpen the prose.

See specs.spec.md for a worked example that checks this documentation and the runner against each other.

Running specs

# Judge every *.spec.md in the repo
node spec.js

# Narrow to a path, directory, or glob
node spec.js skills/testing/specs.spec.md
node spec.js skills/testing

# Sample the judge three times per assertion, majority wins
node spec.js skills/testing/specs.spec.md --consensus 3

# Write the report and full results to disk
node spec.js --report spec-report.md --json spec-results.json

CLI reference

Flag	Purpose	Default
[paths/globs]	Spec files, directories, or globs	**/*.spec.md
--consensus N	Judge calls per assertion; majority wins	1
--agree K	Votes required to settle under consensus	majority of N
--concurrency N	Max judge calls in flight at once	8
--retries N	Retries on a malformed judge response	2
--report PATH	Also write the markdown report here	stdout only
--json PATH	Write full results as JSON	none
--help	Show usage and exit	—

When to use a spec vs a test vs an eval

• Spec (*.spec.md) — an LLM judge can give a single-pass pass/fail verdict on a property of a fixed artifact (a document is internally consistent; examples illustrate the description).
• Code test (*.test.js, shell) — the check is deterministic and code-expressible.
• Eval (*.eval.md) — the property is probabilistic, measured across many trials against a distribution of inputs (does a skill trigger in the right contexts most of the time).

Each spec assertion runs an LLM call, which is slow and noisy compared to a code test. Don't use a spec for a property with a clean code-test or eval form; that makes the runner expensive enough that the gate gets disabled within a quarter, which is worse than no gate at all.