Falsification Is Asymmetric

One counterexample can break an "always" claim, but no finite number of successes proves a "never." So: a single probe disproves "always," not "never" — and before asserting a categorical negative, sample across the spectrum rather than generalizing from one case.

Explanation

The source (Working Heuristics — Data, Matching, Product, Org, Tooling) carries two epistemic rules that share one logic:

• A single probe disproves "always," not "never." Finding one case where X fails is enough to kill "X always holds." But finding one case where X is absent does not establish "X never happens" — you've sampled a single point of a whole space. Sample across the spectrum before claiming a categorical negative.
• Test a category claim against the other side's actual artifact. Don't assert "they can't do Y" from reputation or assumption — go look at their real artifact. If it turns out they can, find the narrower axis where you're genuinely distinct, rather than defending a categorical claim the evidence just broke.

Both are the same asymmetry: disconfirming a universal is cheap (one case); confirming a universal negative is expensive (the whole space). Calibrate your confidence and your sampling accordingly.

Why it matters / where it shows up

• It guards against lazy positioning. "We're the only ones who do X" is a category claim that one rival artifact can falsify on the spot — embarrassing if asserted publicly. Check the artifact first; retreat to the true narrow distinction.
• It disciplines testing and QA. "It never happens" deserves a spread of probes; "it always works" can be refuted by a single adversarial case — so spend test effort where a single counterexample is decisive.
• It pairs with shrinking the problem. Like Shrink the Candidate Space, it's about structuring inquiry — sampling the spectrum — instead of over-investing in one point and over-generalizing from it.

Open questions

• "Sample across the spectrum" needs a defined spectrum. How do you know your sample frame covers the space, rather than confirming a negative within blinders?

Cognitive backdrop

Willingham lists "only seeking evidence that confirms one's beliefs" and "ignoring countervailing evidence" among the pitfalls critical thinking avoids. He notes that even physicians discount patient data that conflict with a diagnosis they have in mind. This heuristic is a metacognitive strategy against that pitfall. Willingham also adds a limit: you can only recognize a result as anomalous (a disconfirming probe) if you had an expectation to violate, and that expectation is domain knowledge (Critical Thinking Is Domain-Bound). "Sample across the spectrum" is also the move that makes deep structure visible.

Connections

• Metacognitive Strategies — this heuristic is one; Willingham sets its limits.
• Critical Thinking Is Domain-Bound — spotting the disconfirming case needs domain expectations.
• Build It as an Extension — the competitive-positioning heuristic ("find the narrower axis where you're distinct") lives next to product/extension decisions.
• Shrink the Candidate Space — both restructure inquiry rather than brute-forcing a single point.
• Anthropic — Investigating Vault Particles with Claude — see also (not grounding): a clean instance. Claude argued from priors that size-exclusion chromatography would fail (misfolded clumps elute in the same void volume); the team ran it once and the objection was dead. A model's confident "no" is a hypothesis, and one cheap probe outranks it — which is also why knowing the domain well enough to design the probe matters.
• Provable Inference — see also (not grounding): Anthropic proposes a "Proof-of-Failure": cryptographic proof that a model spent at least X tokens on a patch and found no bug (Anthropic — Progress on Provable Inference for Large Language Models). The asymmetry decides what it can certify: the effort is provable, the "never" isn't. One bug found later still beats a million clean tokens.
• Distributions over Best Fits — the same asymmetry in scientific modeling. Hodgkin and Huxley's four-particle hypothesis "only carried weight because of what they had ruled out," and the authors of Anthropic — Teaching AI to Build Scientific Models say the most useful experiments are those that "rule out the greatest number of remaining models." It also offers one answer to this page's open question about the sample frame: an explicit distribution of candidate models is the defined spectrum.
• System of Record — "check the actual artifact, don't assert what you can't see" is the same evidentiary discipline as verifying authority before trusting a source.

Sources

• Working Heuristics — Data, Matching, Product, Org, Tooling — "a single probe disproves always, not never"; "test a category claim against the other side's actual artifact."
• Willingham — Critical Thinking, Why Is It So Hard to Teach — confirmation-seeking as a core thinking pitfall; anomalies need expectations.
• Anthropic — Teaching AI to Build Scientific Models — interpretive weight comes from eliminated alternatives; choose experiments that eliminate the most models.