Skip to main content

evidence-not-confidence

Why It Exists​

A common pattern: have a model extract or infer a fact, ask it for a "confidence" number, and gate behavior on that number. The number looks like corroboration. It usually is not: a model grading its own output is one source scoring itself, and it will be wrong in the direction that makes it look useful.

This skill exists because two opposite mistakes were made for real, on real systems, in the same week. One system had several genuinely independent sources and needed a proper scoring design. The other had exactly one source and had been given decorative scoring machinery that implied a check which never happened. The skill is how to tell which situation you are in, and what to build in each.

What It Does​

It routes on a single question: does the fact you are scoring have more than one genuinely independent source, or exactly one?

  • Several independent sources: build a priced evidence ledger. A closed set of evidence kinds, one fixed weight per kind, a combination function, and a write gate. Combine with noisy-OR (1 - Π(1 - weight)) so two weak independent sources agreeing can beat either alone, with an explicit contradiction kind that caps the result. Bands have floors, and the top band requires both a high score and at least one primary source, so a pile of weak evidence can never reach it. Only the top band auto-applies; everything else becomes a suggestion a person confirms. Tools may report a kind they observed, never a score.
  • Exactly one source: do not build a ledger. A weighted combination over one input is theater. Define a small set of trust classes for how the fact was produced (for example self-graded, corroborated by an independent source, human-confirmed), default to the most conservative class the system can actually prove, and put that in the type system. The stronger classes are typed but documented as not yet constructible, naming what would have to exist for each. A fact the model scored 0.98 is still self-graded.

How To Use It​

Work through the decision checklist in order, and stop at the first "no": a persistent entity? two or more genuinely independent sources (traced upstream, not just by name)? Then build the ledger; otherwise build the trust class. For the trust-class path, also record per write path whether attaching a class is even structurally possible today. That is the kind of specific, falsifiable finding the exercise should produce, rather than "we added confidence scoring".

Gotchas & Invariants​

A source that looks independent but is not​

The obvious second source (a library the extracted facts could be checked against) turned out, once the write paths were traced, to sometimes be downstream of the same generation as the fact being checked: a document is rewritten by a model from library content, facts are re-extracted from the rewritten document, and "corroboration" against the library is the model's own earlier output being graded as verified. A closed loop, not two sources.

Before counting anything as a second source, trace it upstream. Could it have been produced, even indirectly, from the same generation event? If so it is the same source under a different table name. Any pipeline where model output can be written back into data it also reads from is a candidate.

One rule on either path​

A confidence value is never an input a model fills in about its own output. A tool schema, expected JSON shape or API payload with a confidence field the model populates is the anti-pattern, whichever path you are on. Confidence is only ever a computed output of a scoring system over evidence the system itself priced, or it does not exist and you use an honest label.

Red flags​

  • A model-written "confidence" with no independent second source feeding anything: single-source, dressed as multi-source.
  • A second source whose generation traces back through the same pipeline as the first.
  • Weights, bands or a combination formula built before confirming a second real evidence kind exists.
  • A high-trust word ("verified", "confirmed") in UI copy or prompts for data that is really single-source. Search the templates and strings directly; the overclaim usually lives in the copy, not the scoring logic.