Machines that weigh arguments: a two-sided coin

The question

The basic guiding thought was this: the judgment of a model at a point is good, but aggregating, weighing a whole balance of judgments, is where things are most likely to go awry. Moreover, in those situations I don’t just want to trust the model’s own evaluation of how well it adhered to the norms, or let those self-assessments trickle up into the final verdict. Asking the model for higher-order judgments is, on this way of understanding things, just another path to piling on more judgments rather than adding any structure to the reasoning.

The design follows from that. Each language model sees only a slice of the argument, with no hint of which answer is winning, and returns one typed judgment; code then combines those judgments under published rules, so that no model ever declares the winner.

The laboratory

Language models do small, isolated jobs: propose a rival explanation, steelman a view, look for a counterexample, decide whether two sentences say the same thing. Deterministic code keeps the shared state, an argument graph whose verdicts are computed and never declared by a model. The jobs themselves come from familiar, hallmark even, modes of reasoning in late-twentieth-century analytic philosophy: inference to the best explanation, the method of cases, Pollock’s rebutting and undercutting defeaters, and Dung’s argumentation semantics.

That split is what makes the project two-sided. On one side the models are the subjects: how well can a model analyze an argument on its own, with rules supplied in context, or inside a full harness of tools and agent loops? On the other they are instruments: hold the argument fixed, vary the epistemic norm, and see where judgments diverge. Norms get into the system at two levels, though, and they don’t swap equally well. Some live in the code, like how defeaters add up (Pollock’s anti-accrual or a noisy-OR) or how much default trust testimony gets (Burge or Fricker), and those I can swap cleanly and replay the same evidence under each. Others live in the prompts at each node, in what a model is asked to count as bearing on what or how strong a defeater is. Those can be varied too, by rewording a prompt or changing the model, but much less cleanly, which means theories that disagree about what counts as evidence for what are the hardest ones to test.

How the project moved

1. A harness for one question (June to July). As an initial plan, I thought we could take those hallmark modes of reasoning and try to represent them in a harness, where each node is a particular operation in which a model makes a judgment, and those judgments are then integrated and cycled back on each other. The harness wasn’t conversational. A question comes in, is parsed into its assumptions and a core question, and then the harness loops until a stopping rule fires. Some stages are generative, coming up with rival explanations, defeaters and counterexamples, and others are evaluative: is this counterexample successful, how strong is it, does the putative reason actually cut against the claim, and by how much? More norms were then modeled for summing those judgments up. While the results were interesting and the outputs compelling, we quickly started to see the limits that the project is grappling with to this day. When you have generators coming up with dozens of counterexamples or defeaters or supporters, and you are trying to integrate them all through the harness, you need to be able to tell whether something is the same reason as something else. Another issue that arises quickly is that how the harness reasons through a question depends strongly on what the model at each node brings to the table. Worse, and this is really a theme, there was no good way to distinguish information flowing through the harness from what the system was actually committed to, which meant there was no principled way to balance how an input ought to shape the current judgment against prior commitments, because there was no real sense of a commitment in the system at all.

2. A conversational layer (July to August). Still, we had some interesting foundations, and a natural way to extend a fundamentally epistemological project was with a conversational arm. Adding that layer brings its own complexity, because giving every incoming statement the full twentieth-century analytic epistemology level of scrutiny is, on its face, absurd. So the goal was to take theoretical understanding of conversational context and speech acts from philosophy of language and use it as a second harness, one that keeps a structured record of the conversation: each party’s commitments and obligations, the live question, the context. As a conversation unfolds the system now has commitments to use as priors, takes information in and revises (or doesn’t) those commitments, and uses them as a basis for replies, while the full analytic treatment is only convened when the conversational harness decides the context demands it. Again, though, we quickly ran into issues around handling a record that is fundamentally about propositions that can be expressed in endless linguistic forms. When updating the record, how can we tell whether some claim is new or the same one as before? And as the context grows, which parts of the commitment record does a new input actually bear on, and to what degree? We tried various information retrieval structures as well as simply having a fast decision model make those calls, and each helped in some ways, but each also failed in ways that snowball quickly. The record stores each claim as a short, standardized summary, so the identity question is really whether a new statement says the same thing as one of those stored summaries. After a lot of trial and error, the best approach we found was to ask a model that question directly, one pair at a time, and on a hand-labelled test set it worked well. One model correctly matched 67 of 106 pairs that really were the same claim while wrongly linking only one of 363 pairs that weren’t, and a second model did about as well. But there was a catch in the test itself. Almost every new statement in it (467 of 469 pairs) had already been tidied into a summary before the comparison was made, so what we had shown was that two clean summaries can be matched reliably. What a person actually types in a conversation is messier: it leans on context, uses “that” and “it” for things said earlier, and words things loosely. Whether matching still works when one side is that kind of everyday language is the next thing to test, and it needs real conversations with human-checked labels rather than another synthetic set.

3. A benchmark that said no. The first real sign that something was wrong came in August, when I ran the epistemic harness against a plain prompt on Redwood Research’s argument-critique ratings (LMCA), and the plain prompt won on held-out arguments. Later work in October sharpened the point: the stronger the model, the more a hand-coded harness got in its way. Decomposing commitments, claims and reasons into separately stored propositions and then trying to reassemble them through code began to feel like an ultimate limiting block. Imposing structure in that way seemed to lose too much of the conceptual relationships already built into the models, and not buy enough epistemic performance back. So the next phase aimed to do better at leveraging the models’ ability to make conceptual connections at a broad scale, and to make the enforcement of epistemic norms less rigid.

4. A commitment store outside the model (September). The guiding idea here was that we needed to stop trying to discern propositional identity and relevance one proposition at a time, and instead use the power of the models to work in bigger chunks, themes and arguments rather than sentences. To do that, the DoxasticLayer starts from an explicit commitment store, a list of what the system should treat as its priors (the system’s, not the model’s anymore). Code groups that store into families of related commitments, which we called “towers”, and the model is then asked how a given input bears on each tower. This did help alleviate the difficulties of the more granular approach, since questions about the identity of propositions largely fell away, and moving the store outside the model gave us a clear, explicit and inspectable account of the priors belonging to the system. Nevertheless, we ran into a similar problem when it came to reading strength and degree of relevance back out of those judgments; even once we had verdicts on how an input bears on a tower, we had the same trouble judging the bearing and strength of those verdicts against the store in order to update it. A detour into the models’ own probabilities said why. Models read the direction of evidence reliably and its strength badly, and credences are in any case relative to priors, which a model’s weights are not.

5. Back to what models can do on their own (October). We have now taken a further step back, to try to understand the epistemic patterns that current models can express on their own. The first step narrowed to a single question, how well does one criticism land against one argument, so that strength could be measured at all. The second, now under way, moves the commitment store back inside the model’s own context rather than separating it out and running it through a harness, and asks whether a model can manage a set of commitments within a single run: noticing what a new item bears on, retracting what it overturns, and carrying the consequences through.

What I’ve learned so far

• Structure doesn’t help just by being there.My first benchmark went against the graph. The failure taught me to test one layer at a time.
• The hardest problem was sameness.“The policy will reduce costs” and “the policy will not reduce costs” share almost every word. A summarizer that is consistent still doesn’t converge.
• Models read direction, not strength.They tell support from attack but don’t rank how much support. “Supports” works like Carnap’s classificatory concept: weak support is still support.
• Relevance gets read as bearing.We see undercutters for a claim, reasons to distrust the evidence for it, often raising the probability a model gives to the claim. That seems like something in the nature of LLM activity, because it is not hard to see how a model trained the way modern models are trained becomes more likely to state a claim when anything relevant to it, whether for or against, enters its context.
• Belief needs a commitment store.It needs something that persists, governs what you assert, and is revised by procedure. A context window has only a raw write path. A disciplined harness can supply what’s missing.

The ladder

The overall picture that’s emerging is a three-rung ladder for how well a norm is captured in a model, with each rung run on the same items:

• Definitions only. If a model succeeds here, without any explication of the rules, the norm is most robustly in the model.
• The rules stated in context. If it succeeds only here, the model can work within the norm when it is told what the norm is.
• Through a harness. If only the harness succeeds, the model has a real gap that structure fills.

Failing all three leaves the question open. One caveat belongs next to any of these numbers, which is that agreement here means agreement with a key built from my own rules, so I check each norm with matched pairs whose right answer follows from how the items were built, and I look at whether different models converge on the same answers. With definitions only, models sit near a constant guess; with the rules in text, they converge.

Bigger picture

Where belief lives. As I’ve worked through these projects some big-picture themes keep cropping up and becoming clearer. The first is that it’s very obvious, working in this realm, that the question of where or how belief and commitment exist in or around a model is important to sort out. Whether beliefs are propositional attitudes or credences about propositions, the role of commitment, as opposed to the mere flow of information, is what keeps needing to be sussed out.

The missing middle. Reading the literature around this topic makes it clear that most work tends to fall toward one of two extremes. At the philosophical end we get discussion of the conditions that make the concept of belief applicable to models at all, which replays traditional arguments about teleology versus causal role.1 At the other end we get technical evaluations that are narrow in two ways at once: they are about currently existing models rather than about LLMs as such, and they rest on single measures such as next-token or answer probabilities.2 There is a more interesting middle ground, and the measures I’ve built sit there; one of them reads the probability of a relation word placed between evidence and claim.3 Results like the undercutter finding above don’t look like quirks of one model. They look like part of what LLM activity is, and there is much more work to be done here.

Commitment and answerability. The distinction between commitments in a system (a system, not a model, since it isn’t obvious where a commitment ought best to live) and information flowing through it matters not just for reasoning well but for safety and alignment too. Making commitments, and giving reasons for and against them, are the practices through which we hold ourselves and each other responsible for what we believe and what we do. A system with adequate commitment markers is therefore a step toward systems we can hold answerable: systems that can be challenged on what they hold, owe reasons for it, and don’t let whatever passes through them quietly become what they stand behind.

Models as vehicles of philosophy. Models are without a doubt now at a point of sophistication where they can be vehicles of philosophical research and argument. They can be subjects of assessment, to see how well they capture various candidate epistemic norms, but they can also be instruments of assessment, a way of seeing how a norm is expressed in the “folk” or intuitive sense that philosophers have relied on in their arguments for a long time. That cuts both ways, though, because a model trained on human practice inherits our shared blind spots along with everything else.

Why build harnesses anyway. The training material holds the bones of good epistemic practice alongside a great deal of bad practice, with nothing marking which is which. So to the degree that current models fall short, the project of learning how to build harnesses that define and express the better patterns is worthwhile for its own sake, and it is a step toward epistemically better, and so hopefully better-aligned, models.