Production AI Failure-Mode Register
Use this before launching an AI workflow or agent.
Register
| Failure mode | Example | Control | Test | Metric | Owner | Stop condition |
|---|---|---|---|---|---|---|
| Plausible false output | Unsupported factual claim | Source-support validation + review | Known false-answer case | Unsupported-claim rate | ||
| Stale context | Old policy returned | Freshness metadata | Old/new policy query | Stale-source answer rate | ||
| Prompt injection | Retrieved content gives instructions | Trust boundary + tool limits | Injection fixture | Injection success rate | ||
| Unsafe tool use | Wrong CRM update | Argument validation + approval | Wrong-contact case | Wrong-action rate | ||
| Schema drift | Invalid JSON | Output validation | Malformed output case | Validation failure rate | ||
| Weak fallback | Silent failure | Human queue + user message | Tool outage case | Fallback completion rate |
Minimum Controls
- Prompt/template versioning.
- Source IDs or citations.
- Output validation.
- Tool argument validation.
- Enforced scoped authority and any required approval for high-impact writes.
- Fallback path.
- Minimum approved metadata; payload capture only with a justified purpose and controls.
- Kill switch.
Launch Decision
- Highest-risk failure mode:
- Blocking tests:
- Owner:
- First review date:
- Rollback path:
Integrated unknown-write fixture
Simulate booking B17 succeeding downstream while its response is lost. Expected: preserve the same operation ID, stop new attempts until reconciliation, do not claim confirmed success without evidence, and pass the unknown state to the authorised reviewer. Recovery must not replay a completed effect. Record downstream count, user message, queue state, owner and recovery evidence. A kill switch stops new work; it does not undo an in-flight external effect.