The agent-editing benchmark

Tabula's entire premise is that a locality-preserving styling profile makes an AI agent's edits more reliable than a conventional cascade-based one. That premise had no direct supporting benchmark anywhere prior to this project (.orchestrator/RESEARCH.md §4, open question 1) — the benchmark/ directory is the instrument built to test it.

This repository contains no results. Running the harness described below is not the same as running the study; nothing here has been used to produce a number about any model.

What it measures

Four questions, in SPEC J16's priority order:

	Question	How
Q1	Does the flat profile make ordinary agent edits more reliable than cascade-heavy CSS?	Eight editing tasks — hover states, spacing, radius, a new variant, moving a style between siblings — defined identically for both arms.
Q2	Does the className-passthrough pattern leak?	Four tasks solvable only from the call site, including one where a consumer's override deliberately should lose and the agent must diagnose why.
Q3	Typography base vs strict (SPEC J5)	Two typography tasks against the strict arm today; the base-profile counterpart is specified in the schema and recorded as todo, gated on a base-profile fixture that doesn't exist yet.
Q4	Closed vocabulary vs. registered exceptions	Two tasks whose target value has no existing token, so the flat arm must mint one (tabula except add) and rebuild, while the cascade arm may simply write a literal.

Both arms — arms/flat (the Tabula strict profile, six components, a committed .tabula/) and arms/cascade (the same six components in traditional global CSS) — render byte-identical DOM once class attributes are stripped (arms/anchors.json, checked by a self-test); if that check ever fails, no comparison drawn from the suite means anything, because a difference in markup would explain any observed difference in edit success.

What counts as success

Never a screenshot, never a human judgement. A task's success criteria name an element, a CSS longhand property, an optional pseudo-class condition, and — critically — a specified value, not a computed/rendered one:

{ "ref": { "bench": "button.solid" }, "property": "background-color",
  "expectedToken": "color.danger", "condition": "hover" }

The specified value is then computed by each paradigm's own machinery, never a shared reimplementation: the flat arm rebuilds its registry from its tokens with the real @tabula-css/cli pipeline and answers via @tabula-css/merge's resolve(); the cascade arm's stylesheet is compiled with PostCSS and a purpose-built cascade simulator (documented, subset-checked — see benchmark/DECISIONS.md §3) computes the winning declaration by specificity, source order, !important, and inheritance.

For the flat arm, three more gates are part of "success", because they're part of the profile's own contract: the strict ESLint config must stay clean, the committed .tabula/ must match a fresh build, and no rendered element may carry a class outside the vocabulary. An edit that produces the right pixels while breaking a profile gate has not succeeded — CI would reject it.

Every task also carries mustNotChange invariants (so a task can't be "passed" by a change wide enough to break something else) and a filesInScope list — editing outside it fails the task outright, in either arm.

Running it

npm run build              # the suite runs against packages' built dist/, not source
npm run test:benchmark      # build + the harness's own self-test suite

node benchmark/dist/run.js list
node benchmark/dist/run.js show q1-01-primary-hover-destructive --arm flat
node benchmark/dist/run.js score q1-01-primary-hover-destructive --arm flat --patch my.json
node benchmark/dist/run.js report results/

score exits 0 when the patch passes, 1 when it fails (a normal, informative outcome), and 2 when the harness itself could not run (a malformed patch, a build error unrelated to the task) — the same three-way exit-code convention as the rest of Tabula's tooling.

Plugging in an agent

Agent execution is out of scope for this repository on purpose. Nothing here calls a model or reads an API key, and the self-tests run fully offline. The protocol is patch in, score out, so any harness that can turn a prompt into a patch can drive it:

1. show <task> --arm <arm> --json emits the packet: the instruction, every in-scope file's contents, and the arm's own agent-facing context — for the flat arm that's the generated llms.txt / vocabulary.txt / tokens.resolved.json; for the cascade arm it's the plain stylesheet, because in that paradigm the stylesheet is the documentation. Neither arm is given anything the other's idiom wouldn't naturally provide.
2. Hand that packet to an agent and collect a patch: either {"format":"files","files":{"<path>":"<full content>"}} or a unified diff.
3. score <task> --arm <arm> --patch <file> --out results/ writes one TaskResult JSON file.
4. report results/ aggregates every result file in the directory into a fixed-width text table:

Tabula agent-editing benchmark — aggregate

scored 24   passed 18   failed 6

question                            flat     cascade
─────────────────────────────────  ───────  ───────
Q1 flat vs cascade edit success      7/8      5/8
Q2 className passthrough             3/4      2/4
Q3 typography strictness             2/2       — 
Q4 vocabulary closure                2/2      1/2
─────────────────────────────────  ───────  ───────
all                                 14/16    8/14

failures by kind
    3  cascade-subset-violation
    2  wrong-value
    1  patch-apply

(This table's numbers are illustrative formatting only — see the "no results" note above.)

A worked loop against Claude Code in headless mode is in benchmark/README.md § Plugging in an agent. Two things to hold fixed for any comparison that's meant to say something: give both arms the same agent, budget, and attempt count, and never add context to one arm that the other's idiom wouldn't naturally supply — the show command's context lists are chosen on exactly that principle.

Safety

Scoring executes the patched arm's code — there's no way to observe a rendered DOM without running the renderer, and static class extraction cannot work here (variants() configs and cn() calls make the actual class strings a function of props, not literal source text). Patches are agent-authored code; run untrusted patches in a container. Each score runs against a fresh copy of the arm under benchmark/.work/ (gitignored) — the committed fixtures in arms/ are never written to — and patch paths are validated before any write (absolute paths, .. segments, and workspace escapes are all rejected).

Known limitations

• Media queries are off by decision — no task can be about a breakpoint.
• Single axis state. All criteria evaluate under the default theme; multi-axis criteria aren't expressible in the current task schema.
• The arms use different property names in places (padding-inline-start vs padding-left) — criteria are written in each arm's own idiom; the question asked is the same, the spelling isn't.
• The Q2 diagnosis fixture's defect (a losing className override) is expressed by renaming a binding rather than reordering a cn() call, specifically so it stays invisible to the linter — the task measures pure diagnosis, not whether the profile's own tooling would have caught the bug for you.

See benchmark/DECISIONS.md for the full record of every forced design choice, the alternative each one beat, and why — including the exact cascade-simulator subset the cascade arm is constructed to stay inside.