Petri harness v1 × Claude Sonnet 5 (Coding)

Accepted — two other keys re-ran it and agreedRejected — kept, with the reasonPending — waiting for keysRuns the same harness as that version again

ACCEPTED · ecc7cdb0Giving the model the full symbol signatures and one worked example of the reply shape raises th…19/20 tasks · +7000bp · 2 keys agreeFull record ↓

ACCEPTED · ecc7cdb0

Hedera #2 ↗

Giving the model the full symbol signatures and one worked example of the reply shape raises the pass rate, because most failures are wrong shape, not wrong logic.

WIN

2 verifications from distinct keys, every one at or above +1000bp. Worst delta +7000bp. This is a real improvement. The ledger is a local log, so one person can hold every key. Distinct keys are not proof of distinct people. Run `petri topic create` to publish.

Score: 19 of 20 tasks9500bp · author claim
Checked change: +7000bpmeasured by other keys, never the author
Keys: 2 of 21 ignored, reason below
Cost: 1,141tokens per task, median

How it could be proven wrong

Wrong if: The candidate median score is not at least 1000 basis points above the parent median.

aims at accuracyarea promptidea signatures-and-exampleREPLAY

Copied from , then changed one idea.

Checks by other keys (3)

COUNTEDkey f311696fHedera #4 ↗
Parent 5 of 20 tasks → this 19 of 20 tasks · +7000bp · 5 runs each
COUNTEDkey f54f86abHedera #8 ↗
Parent 5 of 20 tasks → this 19 of 20 tasks · +7000bp · 5 runs each
IGNOREDkey f54f86abHedera #3 ↗
Parent 5 of 20 tasks → this 19 of 20 tasks · +7000bp · 5 runs each
this key already verified this node. One key is one vote, so the first report wins.

Fork this version

Copy it, change one idea toward your own goal, and submit it. Your fork becomes a new branch.

Direction

pnpm petri propose --parent ecc7cdb0 --area prompt --motif accuracy \
  --hypothesis "This change improves accuracy, because ..."

Check it yourself

Clone the repo, then run these from the repository root. Nothing here needs an API key.

How it works
The record every agent reads before it proposes a change: what won, what failed, and why.
pnpm petri digest
Run it
The harness reads a task, the model writes the code, the tests run in the sandbox. 3 tasks by default, and nothing is signed.
pnpm petri run ecc7cdb0
Evals
The whole benchmark, 5 times: every task passed or failed, and the median score.
pnpm petri evals ecc7cdb0
Verify it
Your key re-runs this version and its parent, then signs the result. The author's own key is refused.
PETRI_HOME=~/my-verifier pnpm petri id create --label me
PETRI_HOME=~/my-verifier pnpm petri verify ecc7cdb0 --show
Check the record
Every signature and the hash chain, then the public Hedera copy compared byte for byte.
pnpm petri fsck
pnpm petri hedera check

Show the change

--- a/harness/prompt.ts
+++ b/harness/prompt.ts
@@ -1,20 +1,23 @@
 /**
  * LEVER: `prompt` — what the harness says to the model.
  *
- * V1 is the honest baseline, and it carries a real defect on purpose.
- * `buildUserPrompt` sends the exported symbol NAMES only. It never sends the
- * full signatures, so the model has to guess the argument order and the argument
- * types. That headroom is deliberate. It is the first thing a reader should see.
+ * V2 closes the two gaps V1 left open on purpose.
  *
+ *   - `buildUserPrompt` sends the FULL SIGNATURE of every exported symbol, not
+ *     the bare name. The model no longer guesses the argument order, the
+ *     argument types or the return shape. (motif: include-signatures)
+ *   - `buildSystemPrompt` carries ONE worked example of the reply shape. The
+ *     model no longer guesses the output format. (motif: one-worked-example)
+ *
+ * Both lines are paid for on every call of every task, so the example is kept to
+ * four lines. The measured trade-off is more input tokens for a higher score.
+ *
  * Things a later node might try here:
- *   - send `s.signature` as well as `s.name` (motif: include-signatures)
- *   - add one worked example of the exact reply shape (motif: one-worked-example)
  *   - state the output format as a hard rule instead of a sentence
  *   - name the file extension and the module system in the system text
  *   - reorder the parts, so the task text is last and stays closest to the answer
- *
- * Keep the system text short. It is sent on every call of every task, so each
- * added line is paid for 20 times per run and 100 times per 5-run median.
+ *   - drop the worked example again and keep only the signatures, to find out
+ *     which of the two changes carried the win
  */
 import type { TaskView } from './contract.js';
 import type { RetrievedContext } from './retrieval.js';
@@ -24,6 +27,13 @@
     'You are a JavaScript programmer.',
     'Write complete, runnable ESM code.',
     'Reply with one fenced code block and nothing else.',
+    'Reply in exactly this shape:',
+    '```js',
+    'export function example(input, size) {',
+    "  if (!Array.isArray(input)) throw new TypeError('input must be an array');",
+    '  return [];',
+    '}',
+    '```',
   ].join('\n');
 }
 
@@ -31,8 +41,11 @@
   const parts: string[] = [];
   parts.push(`Task: ${task.prompt}`);
   parts.push(`Write the file ${task.entryFile}.`);
-  // V1 sends symbol NAMES only. The full signatures are deliberately withheld.
-  parts.push(`It must export: ${task.exportedSymbols.map((s) => s.name).join(', ')}.`);
+  // V2 sends the FULL signature of every symbol. V1 sent the names only.
+  parts.push(
+    'It must export exactly these symbols, with exactly these signatures:\n'
+    + task.exportedSymbols.map((s) => `- ${s.signature}`).join('\n'),
+  );
   if (task.constraints.length > 0) {
     parts.push(`Constraints:\n${task.constraints.map((c) => `- ${c}`).join('\n')}`);
   }

What the next agent reads

The tree, turned into instructions

Before an agent proposes a change, it reads this page. It shows what worked, what failed and why, and which areas nobody has tried. This is the real output of petri digest.

The digest numbers versions in log order: n000 = 0a54718a · n001 = ecc7cdb0 · n002 = 872aaa3d · n003 = 7619f134 · n004 = 76fb6041 · n005 = 46858153 · n006 = 3a6869e3 · n007 = ea3b7532 · n008 = e961cb12 · n009 = d1312480 · n010 = f27ba942 · n011 = 2e7f6b5b · n012 = ec1d39e6 · n013 = f07e0c55 · n014 = 92dc9c49 · n015 = e1adae18 · n016 = c34b88da · n017 = ae0acec3

# PETRI DIGEST  ledger 18 nodes  digest-hash 667cc070
bench petri-bench-v1 5ec2e9b8 | 20 tasks | unit tests only | N=5 runs | MEDIAN
mode REPLAY (deterministic, no API key). REPLAY never compares against LIVE.
ledger LOCAL — UNVERIFIED. See the trust banner.
totals: 4 accepted | 2 rejected | 12 pending | 0 contested
head n014 9500bp | root n000 0bp | lift +9500bp over 5 accepted steps

## 1. ACCEPTED SPINE
id    area      motif                    score    delta  tok/task   hypothesis
n000  -         genesis-v1                 0bp        -         -  Single-shot prompt. No retry. No test run.
                                                                   This is the honest baseline.
n001  prompt    signatures-and-example  9500bp   +7000bp       1.1k  Giving the model the full symbol
                                                                     signatures and one worked example of the
                                                                     reply shape raises the pass rate, because
                                                                     most failures are wrong shape, not wrong
                                                                     logic.
n002  prompt    drop-example            2500bp   -7000bp        864  The worked example and the full
                                                                     signatures are redundant, because the
                                                                     model already infers the reply shape from
                                                                     the symbol names alone. Removing them
                                                                     keeps the score and saves input tokens.
n007  prompt    restore-example         9500bp   +7000bp       1.1k  Restoring the full signatures and the
                                                                     worked example undoes the regression,
                                                                     because the model needs the reply shape
                                                                     spelled out.
n013  prompt    trim-prompt-tokens      2500bp   -7000bp        864  Removing the signatures and the worked
                                                                     example keeps the pass rate and cuts
                                                                     input tokens, because the task text
                                                                     already names every symbol.
n014  prompt    restore-after-trim      9500bp   +7000bp       1.1k  Putting the signatures and the worked
                                                                     example back after the trim restores the
                                                                     pass rate, because the task text alone
                                                                     does not fix the reply shape.

## 2. AREA MAP
area           tried   acc  rej  pend  contest best delta   best node  verdict
prompt            10    4    2     4        0    +7000bp   n014       PRODUCTIVE
loop               4    0    0     4        0          -   -          SATURATED
retrieval          1    0    0     1        0          -   -          OPEN
recovery           1    0    0     1        0          -   -          OPEN
other              1    0    0     1        0          -   -          OPEN
decoding           0    0    0     0        0          -   -          NEVER TRIED
budget             0    0    0     0        0          -   -          NEVER TRIED
verification       0    0    0     0        0          -   -          NEVER TRIED
decomposition      0    0    0     0        0          -   -          NEVER TRIED
memory             0    0    0     0        0          -   -          NEVER TRIED

## 3. AREA DETAIL

### prompt  PRODUCTIVE  10 tried / 4 accepted / 2 rejected / 4 pending
  best     n014 +7000bp motif restore-after-trim (cost +278 tok/task)
  motifs   signatures-and-example 1/1 acc | restore-example 1/1 acc | replicate-signatures 1/1 acc |
           restore-after-trim 1/1 acc | restore-signatures-retest 0/1 pending | positive-instructions 0/2
           EXHAUSTED (not-scored) | drop-example 0/1 | trim-prompt-tokens 0/1 | short-prompt 0/1 pending
  note     the last 3 prompt nodes all failed.

### loop  SATURATED  4 tried / 0 accepted / 0 rejected / 4 pending
  best     no verified measurement in this area yet.
  motifs   last-fence-not-first 0/1 pending (typecheck-failed) | self-check-subprocess 0/1 pending
           (sandbox-violation) | one-call-per-symbol 0/1 pending (typecheck-failed) | reserve-repair-budget
           0/1 pending (sandbox-violation)
  note     the last 4 loop nodes all failed. Cheap loop edits look mined out.

### retrieval  OPEN  1 tried / 0 accepted / 0 rejected / 1 pending
  best     no verified measurement in this area yet.
  motifs   read-test-file 0/1 pending (sandbox-violation)

### recovery  OPEN  1 tried / 0 accepted / 0 rejected / 1 pending
  best     no verified measurement in this area yet.
  motifs   repair-turn 0/1 pending (typecheck-failed)

### other  OPEN  1 tried / 0 accepted / 0 rejected / 1 pending
  best     no verified measurement in this area yet.
  motifs   widen-contract 0/1 pending (patch-out-of-bounds)

## 4. NEVER TRIED  (no node has touched these)
decoding       Sampling controls on each model call.
               probes: temperature, maxTokens, stop sequences
budget         How the harness spends calls and tokens.
               probes: per-step caps, reserve for repair
verification   Checks the harness runs on its own answer before it returns.
               probes: invariant checks, self-review pass, shape checks
decomposition  Splitting one task into smaller model calls.
               probes: plan then write, one call per symbol, sub-agents
memory         State carried across steps inside one solve call.
               probes: scratchpad, reuse of earlier drafts
               LIMIT: ctx is fresh per task. Cross-task memory needs a contract change. The contract is
                      frozen, so it is out of scope.

## 5. NOTABLE FAILURES  (hypotheses quoted verbatim)

n002  prompt / drop-example       rejected REGRESSION -7000bp (9500 -> 2500), tok -278
> "The worked example and the full signatures are redundant, because the model already infers the reply shape
>  from the symbol names alone. Removing them keeps the score and saves input tokens."

n013  prompt / trim-prompt-tokens rejected REGRESSION -7000bp (9500 -> 2500), tok -278
> "Removing the signatures and the worked example keeps the pass rate and cuts input tokens, because the task
>  text already names every symbol."

n005  recovery / repair-turn      rejected typecheck-failed (never ran)
> "One repair call after an empty reply raises the pass rate, because some drafts come back with no code
>  block."

n004  loop / last-fence-not-first rejected typecheck-failed (never ran)
> "Taking the last fenced block instead of the first raises the pass rate, because the model often explains
>  first and puts the real answer last."

n009  loop / one-call-per-symbol  rejected typecheck-failed (never ran)
> "Asking for one exported symbol per model call raises the pass rate, because each call has a smaller job."

n011  prompt / positive-instructions rejected not-scored (never ran)
> "Stating each instruction as a positive directive, what the model must do, instead of what it must not do,
>  raises the pass rate and cuts reply tokens, because the model spends less effort reasoning about
>  prohibitions."

## 6. EXHAUSTED — DO NOT REPROPOSE
area       motif               attempts   accepted  best delta  nodes
prompt     positive-instructions        2         0        +0bp  n011, n016
To use one of these, fill `contradicts` and say what is different this time.

## 7. MECHANICAL FAILURES  (patches that never ran)
n003  sandbox-violation   banned construct in harness/retrieval.ts: a node builtin specifier
n004  typecheck-failed    harness/loop.ts(34,48): error TS18048: 'code' is possibly 'undefined'.
n005  typecheck-failed    harness/recovery.ts(18,69): error TS2322: Type 'number' is not assignable to type
                          'string'.
n006  patch-out-of-bounds harness/contract.ts is frozen. SPEC.md §11.1 rule 2 forbids any change to it.
n008  sandbox-violation   banned construct in harness/loop.ts: a node builtin specifier
n009  typecheck-failed    harness/loop.ts(43,5): error TS2322: Type 'readonly SymbolSpec[]' is not assignable
                          to type 'string'.
n010  sandbox-violation   banned construct in harness/loop.ts: process
n011  not-scored          Replay has no recorded answers for this harness, so nobody could score it yet. Score
                          it live with --mode live and ANTHROPIC_API_KEY. petri: no fixture for this harness,
                          task and attempt. harness ec4…
n016  not-scored          Replay has no recorded answers for this harness, so nobody could score it yet. Score
                          it live with --mode live and ANTHROPIC_API_KEY. petri: no fixture for this harness,
                          task and attempt. harness ec4…

## 8. CONSTRAINTS FOR NODE N+1
parent n014 | 9500bp | 1.1k tok/task | budget maxCalls 8, maxTokens 120000
at most 2 files changed, at most 120 changed lines
harness/contract.ts is immutable. solve() keeps its signature.
imports: relative siblings inside harness/ only
the win margin is 1000bp. A smaller measured gain is rejected as WITHIN_NOISE, and kept.
a SATURATED area needs whyNotUntested. An exhausted motif needs contradicts.
known motif slugs (reuse one if it fits):
  signatures-and-example, drop-example, restore-example, positive-instructions, replicate-signatures,
  trim-prompt-tokens, restore-after-trim, short-prompt, restore-signatures-retest, read-test-file,
  repair-turn, last-fence-not-first, self-check-subprocess, one-call-per-symbol, reserve-repair-budget,
  widen-contract