The benchmark that said no
The key test was whether reconstructing an argument as a graph and attaching each critique to its target would predict human ratings of argument critiques better than a plain prompt, on Redwood Research’s benchmark of arguments, critiques and ratings. The pilot looked promising. The pre-registered held-out test, split by whole argument, did not: no uplift, decisive.
The direct prompt won. It didn’t refute the platform, but it showed that model-produced structure has to earn trust step by step. Later work made the point sharper: the stronger the model, the more a hand-coded harness got in its way. The question is still being tested in the plain-prompt challenge.
Sameness is the hardest problem
Most of what went wrong traced back to deciding when two sentences are the same claim, not to computing what stands. Exact equality of canonical keys was precise but caught only about a quarter of true paraphrases, on 469 human-labelled pairs: a summarizer that is consistent still doesn’t converge. Asking a model the comparison directly worked much better. One model correctly matched 67 of 106 pairs that really were the same claim while wrongly linking only one of 363 pairs that weren’t, and a second model did about as well.
There is a catch in that test. Almost every new statement in it (467 of 469 pairs) had already been tidied into a summary, so what it shows is that two clean summaries can be matched reliably. Whether matching still works when one side is everyday language, leaning on context and on “that” and “it”, is the next thing to test, and it needs real conversations with human-checked labels.
Observation over authority, at 97%
On the ladder, the harness’s clearest win is a norm that a first-hand observation should back a claim better than a bare appeal to authority. With definitions only, models rank the observation higher 56% of the time, close to a coin flip. With the rule stated, they get it right 85% of the time. Through the harness, 97%.
It doesn’t generalize. For the norm that a criticism should do more damage when it hits an argument’s only support, rules help only the strongest models, and the harness helps middle-ranked models only when it is handed a correct map of the argument, while getting in the way of the strongest. What no model supplies well is the parse. The ladder on the Research page has every cell.
The judges lean toward attack
Shown the same words as a support for a claim and as an attack on its negation, the rating judges called the attack decisive far more readily: 84 times in 144 against 39. This held across three judge models from two families, so the bias runs against supported claims. Tightening the attack side narrowed the gap without closing it.
The judge that appraises a source’s reliability erred both ways about people’s plain reports of their own situation: first it marked them down for the person’s interest in the outcome, then, once that was fixed, it rated them too high. Graded on cases its drafter never saw, the redraft scored 87 of 96 against 68 before.
What decides between semantics
Comparing four semantics on the same graph, the principle for comparing arguments (weakest-link against last-link) mattered less than how the engine writes its records, above all that a support and an attack often land in the same strength band. ASPIC⁺ reads that as a standoff; the flat rule hands it to the attack. Read without preferences, assumption-based argumentation and ASPIC⁺ agreed on all 1,444 statements in the stored runs.