Most teams treat "human in the loop" as a comforting phrase. It sounds safe. It sounds responsible. It also often means nobody has decided what the human actually does.
A human can approve an action, review a sample, handle exceptions, audit after the fact, train the workflow with corrections, or own the business decision. Those are different patterns. They have different costs, failure modes, and staffing needs.
This article gives you a practical decision model for choosing the right pattern.
Human review is a control, not decoration. If the reviewer has no clear authority, no time budget, no checklist, and no stop rule, the workflow is still effectively automated.
Start with consequence
Do not start with the model. Start with the consequence of a wrong output.
Ask five questions:
- Can this affect a customer, employee, supplier, or regulator?
- Can it send, publish, delete, charge, refund, or change a record?
- Can it expose personal, confidential, financial, legal, or health-related data?
- Would a wrong answer be hard to detect later?
- Would a mistake damage trust even if it is technically reversible?
The more yes answers you have, the more explicit the human role must be.
Pattern 1: Human approves every action
Use this when the action is external, destructive, financial, legal, HR-related, or customer-visible.
Examples:
- sending a customer email,
- publishing a public article,
- issuing a refund,
- deleting records,
- changing a contract clause,
- making an employment recommendation.
The model prepares a draft or recommendation. The human approves, edits, or rejects it. The final action does not happen until approval is recorded.
Good approval design includes:
- a clear diff or preview,
- the source evidence used,
- the model confidence or risk flags if available,
- a one-click reject path,
- a required reason for override in high-risk flows,
- an audit log with reviewer, timestamp, and final action.
This is the most expensive pattern, but it is the right default for consequential actions.
Pattern 2: Human reviews exceptions
Use this when most cases are routine but some are ambiguous or risky.
Examples:
- support tickets that mention cancellation, legal threats, security, or billing,
- invoice extraction with low confidence or missing fields,
- lead qualification where the company size or intent is unclear,
- document classification when multiple categories match.
The workflow handles normal cases and routes exceptions to a queue.
Exception routing needs specific rules. "Low confidence" alone is usually too vague. Better triggers:
- missing required fields,
- conflicting extracted values,
- unsupported language,
- unrecognized document type,
- customer sentiment above a risk threshold,
- account tier is enterprise,
- action would cross a money or data threshold,
- source data is stale.
Exception queues need ownership and service levels. If nobody checks the queue daily, the system has not reduced work; it has hidden work.
Pattern 3: Human samples outputs
Use this when the workflow is low consequence but quality drift matters.
Examples:
- internal summaries,
- content tagging,
- meeting action item extraction,
- enrichment of non-sensitive CRM fields,
- suggested knowledge-base links.
The workflow runs automatically. A human reviews a sample: maybe 5 percent of outputs, 20 random cases per week, or every output from a newly changed prompt version.
Sampling works only when corrections feed the system:
- record what was wrong,
- classify the error type,
- update prompt, retrieval, schema, or tool rules,
- add examples to evals,
- track error rate over time.
Sampling is a quality system. It is not a launch approval gate.
Pattern 4: Human audits after the fact
Use this when the workflow is low-risk, reversible, and high-volume.
Examples:
- internal tagging,
- duplicate detection,
- draft-only knowledge-base suggestions,
- cost routing between models,
- non-customer-visible formatting cleanup.
The workflow runs. Logs, dashboards, and periodic audits detect problems.
This pattern is acceptable only when:
- actions are reversible,
- the workflow has a kill switch,
- logs are detailed enough to reconstruct decisions,
- the cost of a missed error is low,
- users know how to report a bad output.
Do not use after-the-fact audit for customer-visible commitments, sensitive data, payments, or regulated decisions.
Pattern 5: Human owns the decision
Use this when AI assists analysis but should not make the decision.
Examples:
- hiring,
- credit or eligibility screening,
- legal strategy,
- medical advice,
- security incident severity,
- vendor selection,
- major purchasing decisions.
The model can summarize evidence, list tradeoffs, generate questions, or compare options. The human decision owner signs off on the final judgment.
The workflow should make that explicit:
- "AI-generated analysis, not a decision."
- "Decision owner: name or role."
- "Evidence reviewed: sources."
- "Known limitations."
- "Final rationale."
This prevents a common failure: the model's fluent recommendation becomes the decision by default.
A simple approval matrix
Use this as a starting point:
Workflow consequence | Default human pattern |
|---|---|
Internal, reversible, low visibility | Audit after the fact |
Internal, repeated, quality-sensitive | Sampling review |
Ambiguous cases in otherwise routine flow | Exception review |
Customer-visible or external action | Approve every action |
Destructive, financial, legal, HR, regulated | Human owns final decision |
The matrix is not law. It is a forcing function. If you choose a lighter pattern, write down why.
Design the review screen
A good review screen reduces reviewer fatigue.
Show:
- what the system proposes,
- what evidence it used,
- what changed from the current state,
- why the item was routed to review,
- the allowed actions,
- the risk flags,
- the deadline if there is one.
Avoid:
- dumping the full prompt,
- asking reviewers to inspect raw logs,
- hiding source documents,
- giving only "approve" and "reject" when "edit" is needed,
- making reviewers re-open five systems to verify one case.
If review is slow, people will bypass it. If review is unclear, people will rubber-stamp it.
Define stop rules
Every human-in-the-loop workflow needs stop rules.
Examples:
- More than 3 percent of sampled outputs fail the checklist.
- Any cross-customer data exposure is detected.
- More than five high-risk exceptions remain unreviewed for 24 hours.
- A prompt or model update increases rejection rate by 50 percent.
- The workflow generates an external action that should have required approval.
A stop rule should say who pauses the workflow and what happens next.
Common mistakes
Putting humans too late. If the reviewer only sees the final polished output, they may miss bad source data. Show evidence and intermediate extraction when needed.
Approving batches blindly. Batch approval is useful, but only after filters and sampling prove the batch is uniform.
No reviewer training. Reviewers need examples of good, bad, and borderline cases.
No feedback loop. If corrections do not improve prompts, retrieval, schemas, or source data, review becomes permanent manual labor.
No capacity planning. A 10 percent exception rate at 1,000 cases a day is 100 human tasks. That is a team, not a footnote.
The takeaway
Human-in-the-loop design is not one pattern. It is a set of controls matched to consequence.
Use:
- approval for high-consequence actions,
- exception review for ambiguous cases,
- sampling for quality drift,
- audit for low-risk reversible work,
- human ownership for real business decisions.
The practical test is simple: if the model is wrong, who notices, who can stop it, and what exactly do they do? If you cannot answer that, the workflow is not ready.
