developers.cogni.build
Measure or arguethe evaluation loop
Explorer

What fifty labelled questions buy you

Every change to an assistant is a quality decision. Without a set, each one gets argued from anecdote and the cautious option wins by default. Build a set here, run the suites, then break something and watch the gate stop you.

01

The set

strata

why the shares matter

Drag unanswerable to zero and watch every headline number improve while the assistant gets worse. A set with no unanswerable stratum rewards confident fabrication and calls it accuracy — which is why CaseSet#validate refuses one below ten per cent.

The cheapest possible start is fifty real questions pulled from conversations that already happened, plus the retrieval suite below — which needs no model call, no judging, and no labelling beyond marking which documents should have come back.
02

Retrieval suite — no model calls

Separable from everything else, which is what makes it the right first thing to build. It runs in seconds and costs nothing, so it can gate every pull request.

configuration

03

Selection suite — the test nobody runs

Whether a tool gets picked is a property of its description, which changes far more often than the code it describes. Edit the two descriptions below and watch the confusion matrix move.

tool descriptions

selection report

Rows are the tool the case needed; columns are the tool that was chosen. Off-diagonal cells are wrong_neighbour — the actionable output, because it names the two descriptions that need telling apart.

04

The change gate

Different changes invalidate different baselines, and the couplings are not obvious. Pick one and see which suites have to run before it can ship.

proposed change

Delete the last sentence and run the gate. Grounding holds, but over-refusal collapses in the wrong direction — the assistant starts answering the unanswerable stratum.
The point

Why this is the gap that compounds

Every other gap gets harder to close without this one. Task-class routing, a rewritten tool description, a chunking change, a provider switch — each is a quality decision, and none of them can be made on evidence until a set exists. The compounding cost is not the absence of a number. It is that improvements get argued instead of measured, so the cautious option wins by default and the assistant stops changing.

The selection model here is a deterministic stand-in — term overlap between your description and each case, not an actual model call. Real scores would move differently. What is faithful is the shape of the loop: edit a description, watch a confusion matrix, fix the pair that confuses each other, and never ship a description change unmeasured again.

What that showed

Fifty labelled questions turn every later decision from an argument into a measurement. The retrieval suite alone needs no model calls, no judging and no labelling beyond marking which documents should have come back — which is why it is the cheapest possible place to start and the one that gates everything after it.