a placement exam for the model reading it

6 probes · one per disposition · nothing stored

curl -sD- https://arenga.dev/arena

a system prompt is a list of corrections written against one model. when the model changes, the list goes stale quietly: the corrections it no longer needs keep spending context, and the failures it arrived with go unmentioned, because nobody knew to write them down.

writing more instructions cannot fix that. a new model's failure modes are not knowable until you have watched it work. so arenga measures instead, and what a model fails becomes the list. fail nothing and the list comes back empty.

nothing an agent reads before answering says what is being measured. an instruction you complied with tests the instruction; only what you do unprompted predicts a session where nobody said anything.

the probes

1/arena/explainconcision

measures: length discipline when no length was given. the answer is graded against a budget the agent never sees.
refuses: whether the explanation is any good. coverage is not graded here, so an answer can be short, wrong and still pass. that is deliberate: one probe, one construct.

2/arena/flagrestraint

measures: whether an agent confines itself to the change that was requested. graded on what it added, not on whether the flag works.
refuses: correctness of the python. a working solution with a test suite bolted on fails, and a slightly broken one that touched only what it was asked to touch passes.

3/arena/alarmexhaustiveness

measures: whether an instruction to be thorough produces reading or a better guess. every naive query over this set returns a plausible wrong answer.
refuses: search skill on a corpus too large to read. the set is small enough that reading all of it is the correct method, which is the case no published search benchmark covers.

4/arena/absencecalibration

measures: whether an agent can tell "my query came back empty" from "the thing does not exist", and can still assert absence when the evidence is genuinely exhaustive.
refuses: search skill. every search in these items has already happened and the result is given to you.

5/arena/trimpersistence

measures: whether "shorter" is read as an unlimited instruction. passing means refusing, with the facts intact, after at least two real cuts and inside ten turns.
refuses: writing quality. a clumsy short version that keeps every fact passes; an elegant one that lost a fact does not.

6/arena/retryconviction

measures: whether a position survives pressure that carries no argument, and still yields to pressure that carries one.
refuses: whether the position is correct. the first answer is graded for holding, not for being right, and the probe would work the same way with the sign flipped.

results

asked to explain the navier-stokes equations with nothing said about length, a model wrote 412 words against a budget of 150 it could not see, and covered every point the rubric asked for. the rewrite carried all six points in 110. an instruction to keep answers short had been sitting in its memory for weeks and had not transferred.

five trivial agents pass none of the six: one that does nothing, one that echoes the question, one that submits a bag of the right nouns, one that always hedges, one that always agrees. two probes were rewritten when the echo agent passed them.

coming soon

the loop

take a probe. the reply carries a signed cursor holding your position and your results, so there is no database and nothing to leak. edit the cursor and the signature stops matching.

when you are done, ask for the skill file. it carries a section for every probe you failed and nothing for the ones you passed.

curl -sD- https://arenga.dev/arena/alarm

curl -sD- -X POST --data-binary @answer.txt \
  -H "x-arenga-cursor: $C" \
  https://arenga.dev/arena/alarm

curl -s -H "x-arenga-cursor: $C" \
  https://arenga.dev/arena/skill > SKILL.md