concept · concepts/Human-in-the-Loop by Exception.md

Human-in-the-Loop by Exception

status: active · created 2026-06-26 · updated 2026-11-08 · 1 source · #data-engineering #matching #human-in-the-loop #automation #workflow #craft

How to design the seam between automation and people: automate the structured bulk, reserve humans for true exceptions and confirmation, pay each human decision only once, and make every confirmation teach the system. Humans are the scarce, high-judgment resource — spend them only where judgment is actually required.

Explanation

The source (Working Heuristics — Data, Matching, Product, Org, Tooling) gives three rules that together describe a well-designed human/automation boundary in a data pipeline:

Automate the structured bulk; reserve humans for true exceptions and confirmation. Never make someone re-type what a system already emits. If the data is already structured, machine-handle it; route only the genuinely ambiguous cases to a person.
Persist human decisions so each is paid once, not re-derived every cycle. A human adjudication is expensive; capture it so the next run reuses it instead of re-asking. (The "do it once at the boundary" principle, applied to human judgment.)
Make the manual step productive. Every human confirmation should teach the system something it didn't have — a label, a rule, a corrected mapping — so the pool of cases requiring a human shrinks over time rather than recurring.

Why it matters / where it shows up

It's the data-pipeline cousin of The Human Sandwich. Both reserve the human for the judgment slice and let the machine do the collapsible bulk; both treat human attention as the constraint to optimize around.
The residual after blocking is where humans go. Candidate generation removes the easy non-matches; the ambiguous remainder is exactly the "true exception" worth a confirmation — and that confirmation, persisted and fed back, improves the matcher.
Confirmation as a learning signal turns a cost center into a flywheel: the same click that resolves one case becomes training data that prevents the next.

Open questions

When does "make the manual step teach the system" tip into over-automating away the human judgment you actually wanted to keep in the loop?

Connections

Communication and Control Are Intertwined — see also: Ashby's regulator R acts only on the disturbances the table T does not already block (§11/9); reserving humans for exceptions is that division.
AI Use Tracks AI Capability — see also (cross-domain, not grounding): AI is most capable where success is easy to verify (RL suitability r = 0.78). This suggests a mechanism for where the "true exceptions" sit: in hard-to-verify work. Anthropic — How Scientists Use Claude also argues for keeping humans in "key research decisions," with "deliberate checkpoints" in risky domains such as biology. That is the by-exception stance applied to research governance.
Verification Requires Domain Knowledge — see also (cross-domain, not grounding): the precondition this heuristic assumes. Routing the exception to a human only pays if that human can adjudicate it. Anthropic — Investigating Vault Particles with Claude shows the failure mode: errors in the team's weak area ran for two weeks.
Do It Once, at the Right Place — "persist a human decision so it's paid once" is the human-decision face of that meta-principle.
Placement and Judgment Over Raw Execution — Move 2 ("reserve human judgment for the exception") of that cross-domain synthesis.
The Human Sandwich — the AI-collaboration analogue: human frames and judges, machine collapses the bulk in between.
Agents as Employees vs Human-Agent Collaboration — "by exception" is a stance on how much to delegate vs. supervise.
Shrink the Candidate Space — blocking decides which cases even reach a human.
Data Contract — "pay it once" is the human-decision version of "derive once at the boundary."

Sources

Working Heuristics — Data, Matching, Product, Org, Tooling — automate the bulk / reserve humans for exceptions; persist human decisions; make the manual step teach.

Linked from 16

Agents as Employees vs Human-Agent Collaboration

Human-in-the-Loop by Exception — cross-domain: the same automate-the-bulk / human-for-exceptions split, framed as data-pipeline design rather than agent collaboration.

AI-Accelerated Scienceauto-ingested

Deliberate checkpoints may be warranted in biology, accepting slower progress in exchange (Human-in-the-Loop by Exception).

AI Use Tracks AI Capabilityauto-ingested

This is a mechanism-level reason why human judgment concentrates on the hard-to-verify exceptions.

Anthropic — How Scientists Use Claudeauto-ingested

Human-in-the-Loop by Exception — deliberate checkpoints at key research decisions.

Ashby — An Introduction to Cybernetics

Human-in-the-Loop by Exception — the regulator R acting only on disturbances that T does not already block (§11/9).

Communication and Control Are Intertwined

Human-in-the-Loop by Exception — the regulator acts only on the variety the table does not already block (Ashby §11/9).

Data Contract

This is an instance of the broader Do It Once, at the Right Place principle — the same logic that puts compute where the data lives and persists a human decision so it's paid once.

Do It Once, at the Right Place

On the human decision → Human-in-the-Loop by Exception.

Dreaming (Offline Memory Consolidation)auto-ingested

The Human-in-the-Loop by Exception shape: expensive checking is done in a batch, offline, rather than inline on every action.

Overview

Hubs: Data Contract, One Fact Per Grain, Pipeline Legibility, Shrink the Candidate Space, System of Record, Human-in-the-Loop by Exception, Data Gravity, Build With, Not Against.

Placement and Judgment Over Raw Execution

Data craft: derive once at the contract; compute where the data has gravity; persist a human decision so it's paid once.

Provable Inferenceauto-ingested

The Human Sandwich · Human-in-the-Loop by Exception — human review is one kind of verifier; PI shows what it looks like when the verifier is a small trusted program instead.

Shrink the Candidate Space

Human-in-the-Loop by Exception — the residual ambiguous pairs after blocking are exactly the "true exceptions" worth a human's confirmation.

The Human Sandwich

Human-in-the-Loop by Exception — cross-domain twin: the data-pipeline version of the same pattern — automate the collapsible bulk, reserve the human for the judgment slice (and bank that judgment so it's paid once).

Verification Requires Domain Knowledgeauto-ingested

Human-in-the-Loop by Exception and The Human Sandwich both reserve judgment for a human.

Working Heuristics — Data, Matching, Product, Org, Tooling

Derive at the boundary (Data Contract), compute where the data lives (Data Gravity), persist a human decision so it's paid once (Human-in-the-Loop by Exception).

Links to 12

In the log 3

2026-11-09 ingest

Ashby — An Introduction to Cybernetics (W. Ross Ashby, 1956; open-access PDF, agent-fetched during the human-requested daily pass; chapters 1 and 11 read, full text in raw/) — 1 source page (Key claims verbatim). Updated Verification Requires Domain Knowledge (the claim in cybernetic form: variety can destroy variety), Method Reflection (T3 given a measure: control bounded by the review channel's capacity), Norbert Wiener, Human-in-the-Loop by Exception (see-also), Overview (new cybernetics thread), index. No contradictions.
2026-11-08 auto-ingest

Anthropic — Investigating Vault Particles with Claude (Anthropic Life Sciences, 2026-11-04; subscribed-feed capture) — unattended; 1 source + 1 concept (Verification Requires Domain Knowledge — you can only catch AI errors where you already have expertise), both tagged auto-ingested. Deliberately lean per T5 salience. The wiki's first worked case of AI-accelerated science: prokaryotic vault structure (phage L05, 96 copies, 9.5 MDa) in <4 months by a team outside its expertise; purification 3 days → 90 min, first model in 34 h; Claude's errors documented (2 weeks lost to a bad protocol; mirror-image cryo-EM map; "we told it to stop improvising"). Updated AI-Accelerated Science (case study + 1 contradiction: physical bench time treated as the cap, but a method substitution cut a physical step ~48× with no lab automation), AI Use Tracks AI Capability (verifiability has two halves — ground truth exists vs operator can read it), The Bottleneck Shift (Claude supplied method ideation + a coenzyme-A hypothesis; also argued against the method that worked), The Human Sandwich (the pattern in a wet lab; the bread must be load-bearing), Agent Autonomy Spectrum (internal multi-agent harness; the one unsupervised improvisation), AI Commoditizes Expertise (borrowed expertise and its bill), Critical Thinking Is Domain-Bound (first field evidence on its open question), Human-in-the-Loop by Exception + Falsification Is Asymmetric (see-also, no source-count bump), Anthropic (in-house biology lab), What's Scarce When Execution Is Cheap (narrows the scarce contribution from judgment toward verification), Overview, index. Method Reflection: T3 sharpened again — review only catches what the reviewer knows, which lands directly on T4's PR-review checkpoint; 4 candidate counterweights recorded. Seed Match Process Weight to Stage and Stakes: third promotion-candidate callout (verdict: analogy only — fourth near-miss; needs a direct source). 3 needs-your-call (1 contradiction, 1 promotion candidate, 1 method question). qmd search still broken (Node ABI) — searched with grep/index.
2026-11-08 auto-ingest

Anthropic — How Scientists Use Claude (Anthropic Research / Economics, 2026-11-05; subscribed-feed capture) — unattended; 1 source, 2 concepts opening the AI & science thread (AI Use Tracks AI Capability, AI-Accelerated Science), all tagged auto-ingested. Updated The Bottleneck Shift (science usage evidence + 1 contradiction: is the what-to-do bottleneck durable? 72% expect AI to match top researchers at ideation by 2030), Individual vs Organizational AI Gains (adoption lags capability; J-curve), Agent Autonomy Spectrum (measured autonomy 23.7% / 64.1% / >70%), Token Shortage (AI cost caps acceleration ~16×), AI and Employment — Replacement vs Expansion (task-level evidence), Human-in-the-Loop by Exception (see-also, not grounding), What's Scarce When Execution Is Cheap, Anthropic (Claude Science, Economics team, Cowork), Claude Code (science usage; Cowork is a separate surface), Dario Amodei ("Machines of Loving Grace"), Overview, index. Method Reflection: sharpened T3 (AI takes analysis + writing, humans keep ideation; that split may not last). Seed Match Process Weight to Stage and Stakes: second promotion-candidate callout (verdict: analogy only). 3 needs-your-call (1 contradiction, 1 promotion candidate, ops). qmd search still broken (Node ABI) — searched with grep.