Matt Priselac

Research · The plain-prompt challenge · Diary

The plain-prompt challenge: diary

Diary

The challenge started on 1 November. Its founding result, the benchmark that said no, is in the harness diary.

The challenge

November 2026
Experiment log
November 1–9

Trying to beat the plain prompt again

The benchmark that said no in August still bothered me, so I handed it to Claude with a small weekly budget and a free hand: find any structure, prompt or harness that predicts the human ratings of argument critiques better than a plain prompt does. The rules were the ones I hold myself to. Every run’s prediction is written down before it runs, every option tried against the data is counted, and the sealed test stays unread until something passes the validation split.

Three weekly budgets and 31 tries across four models later, nothing has beaten the plain prompt on any of them. Judging all of an argument’s critiques together, reconstructing the argument first, adding a defender’s reply, comparing critiques in pairs, checking each critique’s key claim, and the dataset’s own rubric all recovered the same single judgment, with the arms correlating around 0.8 with each other. What moved the number was the model. Correlation with the human ratings went from 0.41 for an open-weight 27B model to 0.44 and 0.49 for two small models and 0.67 for a mid-size frontier model. The small models fall for long, confident, technical-sounding objections: their scores track a critique’s length at about 0.4, where the human raters’ track it at almost zero. Turning on a model’s thinking did more than any structure, and undid much of the trap where a confident that doesn’t follow gets full marks.