Matt Priselac

Research · The harness · Diary

The harness: diary

Diary

Reconstructed from the git history and the dated design documents. Short commit subjects are quoted where they capture a moment well. Commits per month across the whole research repository:

Jun · 60Jul · 136Aug · 409Sep · 463Oct · 162

Building the harness

June – July 2026
Dev diary
June 23–24, 2026

A whole harness in two days

My first commit, Initial commit: defeasible reasoning harness, came with four design documents I’d written beforehand: an epistemic constitution (the norms), context-management principles (critics stay independent), an orchestration spec, and a build brief. The core idea was there from the start. LLM “leaves” each do one job on a status-blinded slice of an argument graph, and a deterministic Dung grounded-semantics engine computes which claims stand.

The first two days had 47 commits. The operations went in one at a time: steelmanning, counterexamples, reframing, source appraisal and evidence-seeking, distinction-drawing, surfacing hidden assumptions, and finally an explicit stopping rule. By that evening the log reads O10 termination judgment — the explicit stopping rule (all 12 ops complete). A graded layer (DF-QuAD) went alongside the binary verdicts. Then came a self-contained HTML viewer that replays a run move by move, and a plain-language narrator.

One early commit set the tone for the whole project: docs: correct O2 run observation — the reframe was operationally inert. I fixed it in the next commit.

Dev diary
Late June – early July

Making it usable

Multi-provider models with per-operation model and effort choices, parallel LLM calls, a chat-to-graph intake (the “front door”) and a local web app. On July 1 a structured review of the reasoning structure was implemented in six stages. Among the changes: per-claim case review, and counterexamples grounded in evidence instead of free invention.

Dev diary
July 6–14

The conversational harness

A sibling built from philosophy of language rather than epistemology. The Conversational Constitution (norms K1–K8, operations T1–T14) grew to about 124 KB. The build ran in steps: a score spine updated only by typed moves, an interpretation ladder, felicity validation with a “reject, don’t guess” rule, then four model leaves on top. Two adversarial reviews in the first day each found real defects. July 7–13 was a full calibration phase, with sweeps, a human-tier adjudication, and a ruling that one judge “was the defect.” Then came the “conversation x-ray”, an observer-only web front end.

Dev diary
July 13–27

Fusion, and learning from live sessions

The reasoning and conversational harnesses were joined so that a conversation’s claims become a “doxastic web” that the deliberation engine settles. The case-law ledger ran to 54 entries over two weeks. Each entry reviewed a live session, named the defects, ruled fixes and smoke-tested them, usually all on the same day. The subjects trace the system growing up: verdict always ships, the narrator is starved behind the high-stakes refusal storm, best session yet; turn 5 walled on one false contrary, first wall-free session. A documentation site went up on July 25–27.

The benchmark that said no, and the identity problem

August 2026
Experiment log
August 15–17

The benchmark that said no

On August 15 I restated the charter: the system is an instrument for exploring doxastic norms, not a solver of their hard problems. Experiment compute moved to Google Cloud Batch.

The key test was E6, built on a Redwood Research benchmark of arguments, critiques and human ratings. The question was whether reconstructing the argument as a graph and attaching each critique to its target would predict human ratings better than a plain prompt. The pilot looked promising. The pre-registered held-out test, split by whole argument, did not: E6 phase 2 — no uplift, decisive; signals not constants; realization chain indicted. The direct prompt won. This became my founding negative result. It didn’t refute the platform, but it showed that model-produced structure has to earn trust step by step. Looking back, this was the first sign of something that the October ladder results would make much sharper, which is that decomposing an argument and rebuilding it in code can end up costing a strong model more than it gives back.

Several smaller experiments ran alongside it. A local 4B model beat a frontier small model on one judgment at prompt parity, ~25x cheaper. A synthetic study showed that the standard way of aggregating defeaters double-counts repeated reasons, so I adopted “ground-aware” accrual, which counts one underlying reason once.

Experiment log
August 18–25

The identity problem takes over

The first realistic “generation window” ran ten conversation set-ups, four replicates each, across mixed model families, and captured 40,344 boundary pairs. It first surfaced nine engineering defects that could have quietly corrupted the evidence. One was every model call now carries a timeout — a dead socket hung a run for eighteen minutes. Another was the wave cap was the 429 storm: 8 was the exact width that broke, measured.

The bigger finding was that the weak point wasn’t computing standing. It was deciding when two sentences are the same claim. On August 23 I hand-labelled 191 identity pairs myself (the identity error is nearly all on one side). The fixes came in quick succession. Retired claims stop answering lookups. The judge is shown concise claim heads instead of whole paragraphs. A denial keeps its “not” when displayed. A user’s argument is stored as premises, a conclusion, and a separate claim that the one supports the other. Window 2 confirmed most of these and found that the failure “moved upstream.”

Experiment log
August 27–30

Canonicalization gets demoted

Window 3 (nine of nine runs clean) produced K06 UNIFIES for the first time: a test case that had failed for a week now succeeded. It was followed by the correction that this was a p~0.1 tail draw. An audit on 469 human-labelled pairs showed that exact equality of canonical keys was precise but caught only about a quarter of true paraphrases. The lesson: consistency is not convergence. Key equality stopped counting as proof of identity, and “link-first” identity, which asks the comparison question directly, replaced it. Its first gold run looked like a failure until the failure was traced to the experiment itself: arriving denials had been shown without their “not.” With that one wiring fix, and the prompt unchanged, both model families passed.

A software-engineering pass ran alongside: all-or-nothing turn boundaries with rollback, observers scoped to a single run, hardened saved state, and a plain-language system walkthrough. On August 30 I wrote From a Conceptual Benchmark to Real Work, a narrative of the whole experimental arc so far.

Opening the lab, and a new default

October – November 2026
Launch
October 27–28

Opening the lab

I published a curated snapshot of the harness as a public repository, structured-reasoning-harness, with fresh history. It has front-door documents: a findings page that starts from four results (the benchmark that said no, anchored confidence bands, the ladder, and direction without strength), an authorship note, a glossary, and an account of what was left out and why, such as restricted dataset details, a third-party model’s name, bulk model output and infrastructure identifiers. Tests that need unpublished data are skipped with their reasons stated, and CI runs the rest. Its first run failed two suites, one because a name check read the Mac’s system dictionary, which Linux doesn’t have; the fix was a frozen list of the twenty words it had turned down.

Since the snapshot, work has continued on building the attachment judge into the commitment store and testing whole turns end to end.

Dev diary
November 4–7

One graph, four semantics

The store has always decided what stands with one rule, Dung’s grounded semantics over a flat graph of supports and attacks. This week I built three alternatives on the same graph, run in shadow so each could be compared without changing a result: ASPIC⁺, which separates premises from rules and lets strength decide which attack wins; a graded semantics (DF-QuAD) that gives every statement a number instead of a label; and assumption-based argumentation.

The surprise was where the disagreements came from. The principle for comparing arguments (weakest-link against last-link) mattered less than how the engine writes its records, above all that a support and an attack often land in the same strength band, which ASPIC⁺ reads as a standoff and the flat rule hands to the attack. With those choices settled, last-link ASPIC⁺ became the default on 5 November, and I ruled that the givens a run starts from stay revisable unless a run says otherwise. A check against the theory came back clean: read without preferences, assumption-based argumentation and ASPIC⁺ agreed on all 1,444 statements in the stored runs. A live deliberation under the new defaults converged on the true view, and a run that ends with its rivals tied now says so. A standoff is contested, not absent.

Experiment log
November 9

Prompts that rate what the record does

An audit of the rating prompts against the project’s prompting standards led to a rewrite: each judge now rates what the record in front of it actually does, and attacks the system writes itself are scored by their structure. Replaying the stored runs showed why it mattered. The judge that appraises a source’s reliability had raised 50 doubts, and 39 of them had been rated too weak to reach the board.

Two findings were less comfortable. Shown the same words as a support for a claim and as an attack on its negation, the judges called the attack decisive far more readily, 84 times in 144 against 39, and this held across three judge models from two families, so the bias runs against supported claims. Tightening the attack side narrowed the gap without closing it. And the reliability judge erred in both directions about people’s plain reports of their own situation: first it marked them down for the person’s interest in the outcome, then, once that was fixed, it rated them too high. A person’s account now tops out at medium, and graded on cases its drafter never saw, the redraft scored 87 of 96 against 68 before.