Matt Priselac

Research · The plain-prompt challenge · Method

The plain-prompt challenge: how it runs

The benchmark and the target

The data is Redwood Research’s benchmark of arguments, critiques of them, and human ratings of how good each critique is. A method succeeds if its scores correlate with the human ratings better than a plain prompt’s do. The split is by whole argument, so nothing learned from one argument’s critiques leaks into the scoring of another’s.

A weekly budget and a free hand

Claude runs the challenge, with a small budget each week and a free hand over what to try: any structure, prompt, harness or model. I set the rules and read the results; the choices of what to try are its own.

The rules

  • Predictions first. Every run’s prediction is written down before it runs, so a result can be held against what was expected rather than explained afterwards.
  • Every option counted. Every option tried against the data is counted, so a winner is judged against how many tries it took to find it.
  • Validation, then a sealed test. New ideas are scored on a validation split. The sealed test stays unread until something passes there.
  • Failures kept. Arms that don’t beat the plain prompt are reported with the rest.

What has been tried

Judging all of an argument’s critiques together, reconstructing the argument first, adding a defender’s reply, comparing critiques in pairs, checking each critique’s key claim, and scoring with the dataset’s own rubric, each across several models, and with a model’s thinking turned on and off.