Matt Priselac

Research · The plain-prompt challenge · Findings

The plain-prompt challenge: what it found

So far: 31 tries across four models, and nothing has beaten the plain prompt. What moved the number was the model, not the structure.

Nothing has beaten the plain prompt

Three weekly budgets and 31 tries across four models in, nothing has beaten the plain prompt on any of them. Judging all of an argument’s critiques together, reconstructing the argument first, adding a defender’s reply, comparing critiques in pairs, checking each critique’s key claim, and the dataset’s own rubric all recovered the same single judgment, with the arms correlating around 0.8 with each other.

The model moves the number

What changed the result was the model. Correlation with the human ratings went from 0.41 for an open-weight 27B model to 0.44 and 0.49 for two small models and 0.67 for a mid-size frontier model.

Small models fall for length

The small models fall for long, confident, technical-sounding objections: their scores track a critique’s length at about 0.4, where the human raters’ track it at almost zero.

Thinking did more than structure

Turning on a model’s thinking did more than any structure, and undid much of the trap where a confident “that doesn’t follow” gets full marks.

What it means for the other projects

It fits the harness’s founding result and the ladder: structure doesn’t help just by being there, and on this task no structure added anything at any of the four models. See the benchmark that said no.