Matt Priselac

Research · The plain-prompt challenge

The plain-prompt challenge: can structure beat a model that is simply asked?

So far: 31 tries across four models, and nothing has beaten the plain prompt. What moved the number was the model, not the structure.

The question

In August I tested whether reconstructing an argument as a graph, and attaching each critique to the part it targets, would predict how people rated critiques of arguments better than simply asking a model. On held-out arguments it didn’t: the plain prompt won. It became the founding negative result of the harness, and it kept bothering me.

So in November I handed the question to Claude with a small weekly budget and a free hand: find any structure, prompt or harness that predicts the human ratings better than a plain prompt does. Anything goes, as long as it runs under the same rules I hold myself to.

Why it matters

Most of the other projects add structure around a model: a harness that keeps judgments local, a store of commitments, rules stated in context. Each of those bets that structure supplies something the model lacks. This project is the check on that bet. If nothing beats a model that is simply asked, the structure isn’t earning its keep on this task, and the question becomes where, if anywhere, it does.

Where to go next

  • Method. The weekly budget, predictions written before each run, every option counted, and a sealed test.
  • Diary. The first three weeks.
  • Findings. What moved the number, and what didn’t.