The benchmark and the target
The data is Redwood Research’s benchmark of arguments, critiques of them, and human ratings of how good each critique is. A method succeeds if its scores correlate with the human ratings better than a plain prompt’s do. The split is by whole argument, so nothing learned from one argument’s critiques leaks into the scoring of another’s.
A weekly budget and a free hand
Claude runs the challenge, with a small budget each week and a free hand over what to try: any structure, prompt, harness or model. I set the rules and read the results; the choices of what to try are its own.
The rules
- Predictions first. Every run’s prediction is written down before it runs, so a result can be held against what was expected rather than explained afterwards.
- Every option counted. Every option tried against the data is counted, so a winner is judged against how many tries it took to find it.
- Validation, then a sealed test. New ideas are scored on a validation split. The sealed test stays unread until something passes there.
- Failures kept. Arms that don’t beat the plain prompt are reported with the rest.
What has been tried
Judging all of an argument’s critiques together, reconstructing the argument first, adding a defender’s reply, comparing critiques in pairs, checking each critique’s key claim, and scoring with the dataset’s own rubric, each across several models, and with a model’s thinking turned on and off.