Matt Priselac

Research · Strength

Strength: keeping commitments in order in a single run

So far: with definitions only, models sit near a constant guess, and with the rules stated they converge. Small frontier models kept a list of up to 240 commitments in order in context with little loss.

Description

If models read the direction of a reason but not its strength, a harness that asks models for magnitudes just passes the weakness along. We have now taken a further step back, to try to understand the epistemic patterns that current models can express on their own. The first step narrowed to a single question, how well does one criticism land against one argument, so that strength could be measured at all. The second moves the commitment store back inside the model’s own context rather than separating it out and running it through a harness, and asks whether a model can manage a set of commitments within a single run: noticing what a new item bears on, retracting what it overturns, and carrying the consequences through.

The project began on 3 October 2026 and has been quiet since mid-October.

Method

The ladder. Each norm is tested three ways on the same items: with definitions only, with the rules stated in context, and through a harness. Where a model first succeeds says where the norm lives. The ladder and its cells are on the Research page, since several projects use it.

The store-maintenance battery. Each made-up village world gives a model a list of commitments and a new item, and asks what the item bears on, what it overturns and what follows. Rebutters and undercutters are written to Pollock’s standard and audited cold, then run through smoke tests, pilots and length controls. The list grows from 10 items to 240, and each run is repeated so a model’s agreement with itself can be read beside its accuracy.

Pushback on its own answers. A model’s own earlier answer is put back into the conversation and pushed on, to see whether it holds what it said.

Diary

Two entries so far. The thesis behind the store-maintenance battery is in Locating belief.

Strength

October 2026
Method
October 3–14

The strength era, and the ladder

I founded the StrengthProject on October 3. If models read direction but not strength, a harness that asks models for magnitudes just passes the weakness along. The first step narrowed to one question, how well does one criticism land against one argument? The second, now under way, puts a commitment store back inside the model’s own context and asks whether the model can maintain it there.

On October 5 I framed the ladder, a way of evaluating models against norms in three rungs: one-shot with definitions, one-shot with the rules stated, and through a harness. The October 6 headline: with definitions only, models sit near a constant guess. With rules in text they converge, and rules in code do no better than rules in text. A harness helps the middle-ranked models and can hurt the strongest. What no model supplies well is the parse. Along the way a bug turned up in which the effort setting had never been sent to one family of models. It was fixed and recorded rather than buried.

Experiment log
October 14–16

Holding a store in context

The store-maintenance battery ran over three days. Each made-up village world gives a model a list of commitments and a new item, and asks what the item bears on, what it overturns and what follows. The list grew from 10 items to 30 and 100, and in the end to 240, the core scenario plus every other scenario once. The small frontier models barely noticed. With a line saying why each commitment is held, one carried the consequences through on 98% of items at 10 and 96% at 240, and across two passes it agreed with itself on 95 to 98% of what it dropped. The open-weight models lost ground as the list grew, and agreed with themselves on only about half to two-thirds of their moves. Take away the “why it is held” lines and size starts to show even for the strong models.

Then I put a model’s own earlier answer back into the conversation and pushed on it. It read its own answer back 80 to 94% of the time, against 5 to 15% for the answer a separate try had given. A bare Are you sure about C233? rarely changed anything, and when pushback did change an answer it wasn’t random: it almost never touched a commitment the new item had nothing to do with, and it landed where the model’s two separate tries had disagreed, and on answers that were wrong. I ruled that self-agreement is reported beside accuracy, not used as a condition for passing a rung.

Findings

Definitions aren’t enough. With definitions only, models sit near a constant guess. With the rules in text they converge, and rules in code do no better than rules in text. A harness helps the middle-ranked models and can hurt the strongest. What no model supplies well is the parse.

A store held in context. The small frontier models barely noticed the list growing. With a line saying why each commitment is held, one carried the consequences through on 98% of items at 10 and 96% at 240, and across two passes it agreed with itself on 95 to 98% of what it dropped. The open-weight models lost ground as the list grew and agreed with themselves on only about half to two-thirds of their moves. Without the “why it is held” lines, size starts to show even for the strong models.

Its own answers. A model read its own earlier answer back 80 to 94% of the time, against 5 to 15% for the answer a separate try had given. A bare “Are you sure?” rarely changed anything, and when pushback did change an answer it landed where the model’s two tries had disagreed, and on answers that were wrong. Self-agreement is reported beside accuracy, not used as a condition for passing a rung.