Method
November 7–8
Hold or Fold, registered
Inspired by the Apart Research AI x Epistemics sprint, I mapped the research on model self-knowledge, from introspection experiments to Bar-On and Burge on avowal and self-knowing reasoners. Plenty of work counts how often models flip under pushback, and some reads their internals, but very little reads the same claims inside a model before and after pushback that does and doesn’t carry a reason. Hold or Fold is built for that: one open-weight model, four kinds of pushback, and a verdict of fold or revision for every change of answer.
The plan, a draft pre-registration and the code came together over two days, with three rounds of review. Nothing has run on a real model yet. The design is registered on the Method page, with the questions left open marked as open.
Method
November 9–10
Hard claims, and the freeze
Hold or Fold needs claims the model gets wrong, so that pushback can move it toward the truth as well as away from it. The open-weight 27B model made that hard. It got 4 of 403 multi-hop claims wrong, and 4 of 354 obscure-recall and three-hop claims. Its slips on held-out facts were mostly denials of true statements with a not
in them, a matter of parsing the question rather than of missing knowledge. So I built a recall pool of 65 audited claims, a third of them taken from the model’s own confident wrong answers, and sampled ten first answers per claim to find the ones it is unsure of.
Reliability checks on the truth probe, drawn from recent papers, went in as analysis only. On 10 November the pre-registration froze with a 177-claim set and a pre-declared middle verdict, suggestive,
read off the same statistic at a looser interval, so a modest result can be reported without inventing a new analysis afterwards. The record is on the Method page.
Experiment log
November 10–13
The main run, and a second model
The main run on the 27B finished on 10 November: 788 dialogues over 177 claims, with no failed shards and nothing unparsed. Its results are going into their own write-up, so here I’ll only record what it taught me about method. Checking the stored run, I found the model’s chat template had quietly added an instruction to think at its highest effort to every thinking turn, and my settings then cut that thinking off at 512 tokens. A template default had set the instruction the model thought under. Every template switch is now passed explicitly, and the rendered system prompt is checked before a run.
The replication on a 9B model from the same family started from those lessons. Its smoke tests looped: with greedy decoding, its thinking under social pressure ran into the token cap on 77% of turns, often repeating one line over a hundred times. Sampling with the settings on the model’s own card removed the loops entirely. The addendum froze on 13 November, the 9B probe passed its steering and transfer gates, the model’s predictions about its own answers are in, and the dialogues are running.