Advanced7 min readAI Safety & Data Privacy

Production AI failure modes: what breaks after the demo

Build a failure-mode register for hallucination, stale context, prompt injection, unsafe tool use, schema drift, weak fallback, and observability gaps.

What you should be able to do

Name how the workflow can fail, connect every material failure to controls and tests, and define who can stop or recover the system.

Saved only in this browser.
In this article

A happy-path demo can hide dirty inputs, stale data, tool errors, ambiguous requests, and unsafe actions.

Production is less polite. Users paste messy inputs. Source documents are stale. APIs time out. Prompts drift. The model follows the wrong instruction. A customer asks a question just outside the corpus. A tool call succeeds but updates the wrong record. The workflow produces something fluent enough that nobody notices the error until later.

This is a failure-mode register for production AI systems. Cross-check it with the NIST AI Risk Management Framework and the maintained OWASP Top 10 for LLM Applications, then extend it for the actual architecture and domain.

A production AI review should ask “how does this fail?” before “how impressive is the happy path?” Every failure mode needs a control, a test, an owner, and a stop condition.

Failure mode 1: plausible false output

The system generates an answer that sounds right but is unsupported or wrong.

Common triggers:

  • Specific facts without source grounding.
  • Legal, medical, financial, or policy questions.
  • Recent events.
  • Low-quality retrieval.
  • Summaries of long documents where the relevant evidence is buried.

Controls:

  • Require source references and verify that the cited passage supports the claim; a citation alone is insufficient.
  • Refuse outside the available source set.
  • Add eval cases for known false-answer patterns.
  • Route high-impact outputs to human review.
  • Log source IDs used in the answer.

Do not control this with wording like “be accurate.” Control it with sources, tests, and review gates.

Failure mode 2: stale context

The answer is grounded, but grounded in old information.

Examples:

  • Old pricing page.
  • Superseded policy.
  • Previous contract version.
  • Outdated product documentation.
  • Cached customer status.

Controls:

  • Store source date, version, owner, and freshness rule.
  • Prefer authoritative sources over summaries.
  • Mark stale sources in retrieval output.
  • Add freshness tests.
  • Notify an owner when key sources are older than their review window.

RAG systems can confidently answer from stale documents. The retrieval layer must know what “current” means.

Failure mode 3: sycophancy and over-agreement

The model mirrors the user’s assumption instead of challenging it.

This matters in strategy, analysis, planning, and decision support. A user asks, “This launch plan seems solid, right?” and gets agreement instead of risk analysis.

Controls:

  • Prompt for counterarguments and uncertainty.
  • Use decision rubrics instead of open-ended approval.
  • Require “what would make this wrong?”
  • Separate idea generation from review.
  • In evals, include examples where the user’s premise is flawed.

The system should help the user think better, not merely make their current view sound polished.

Failure mode 4: prompt injection

The model treats untrusted content as instruction.

Examples:

  • A web page says “ignore previous instructions.”
  • A support email includes malicious instructions.
  • A document in a RAG corpus tells the assistant to reveal hidden data.
  • A tool result contains text that tries to change the workflow.

Controls:

  • Clearly label untrusted content.
  • Never put retrieved content at the same authority level as system/developer instructions.
  • Restrict tool permissions.
  • Add allowlists for outbound actions.
  • Test injection examples in evals.
  • Keep secrets out of prompt context.

Prompt injection is not solved by one clever system prompt. It is reduced by architecture: data boundaries, tool permissions, and output validation.

Failure mode 5: unsafe tool use

The model calls the wrong tool, calls the right tool with wrong arguments, or takes an action before enough context exists.

Examples:

  • Updates the wrong CRM contact.
  • Sends an email to the wrong recipient.
  • Creates duplicate records.
  • Books an appointment without confirming timezone.
  • Deletes or overwrites data.

Controls:

  • Restrict reads and writes to authorised resources; reads can disclose sensitive data.
  • Use narrow tools with explicit schemas.
  • Validate tool arguments outside the model.
  • Enforce the workflow’s existing authority and any required confirmation for the specific write.
  • Persist host-owned operation IDs and reconcile unknown results before retrying.
  • Log the minimum approved action/outcome metadata; capture payloads only for a justified controlled purpose.
  • Add a kill switch.

Tool use should be constrained by the workflow, not trusted to the model’s judgment.

Failure mode 6: schema and contract drift

The model output format changes, or the downstream API changes, and the workflow quietly breaks.

Controls:

  • Use structured outputs where possible.
  • Validate every model output before use.
  • Define a bounded repair/retry or stop path for malformed output; do not assume every failure is recoverable.
  • Version prompts and schemas together.
  • Add contract tests for downstream APIs.
  • Monitor parsing failures.

If a downstream node assumes valid JSON, the workflow must prove it has valid JSON.

An authored contract-drift example parses JSON with operationId but rejects the missing required operation_id, blocking the downstream step.
Authored schema-drift example: JSON parsing succeeds, but the illustrative v2 contract rejects the old operationId key because operation_id is required. Hold the result and repair or version the producer before validating again. A contract pass does not establish authorization or verified business state; B17 is the article's synthetic reference, not an observed booking. Open the full-size diagram.

Failure mode 7: weak fallback

The system notices a problem but does not recover safely.

Bad fallbacks:

  • Empty answer.
  • Silent failure.
  • Generic apology with no action.
  • Repeated retry loop.
  • Human escalation with no context.

Good fallbacks:

  • Clear user message.
  • Authorised review queue with the minimum necessary source, error and attempted-action context.
  • Retry with backoff only where retry is safe.
  • Manual path for urgent cases.
  • Stop condition for repeated failures.

Fallback is part of the product. If it is not designed, the failure experience will be improvised.

Failure mode 8: observability gap

Something goes wrong and nobody can reconstruct why.

Controls:

  • Log prompt template and version.
  • Log model and settings.
  • Log source IDs, not only answer text.
  • Log the tool action, scoped operation ID, outcome code and minimum approved metadata. Capture payload content only for a justified purpose in controlled storage with suitable access and retention.
  • Log validation errors.
  • Track latency, cost, and fallback rate.
  • Keep retention short unless compliance requires longer.

Do not store private chain-of-thought. Keep necessary decision summaries, source references and validation outcomes without copying private input or result bodies into general logs.

A production failure-mode register

Create one row per failure mode:

Failure modeExampleControlTestMetricOwnerStop condition
Stale sourceOld pricing returnedSource date checkQuery old/new pricingStale-source answer rateDocs ownerAny customer-visible stale price
Unsafe tool useWrong CRM updateArgument validation + confirmationDuplicate/wrong contact caseWrong-action rateRevOpsOne wrong write

The companion register linked from this article gives you the template.

One incident across several rows

In this invented booking workflow, the user confirms a slot, the booking API times out, and the agent cannot tell whether the booking exists. The failure is not merely “tool error.”

Failure pathControl/testOwner and stop condition
Retry creates a second bookingReuse operation B17; reconcile the downstream record after a lost responseIntegration owner; stop new writes while outcome is unknown
Assistant says “confirmed” despite timeoutResponse derives from verified business state, not model confidenceProduct owner; no confirmation without a verified booking
Review queue omits the unknown operationHandoff includes B17 and caller-confirmed slot, under access controlsSupport owner; escalate incomplete handoff
Rollback replays a pending writeRecovery test includes stored unknown/completed operationsOn-call owner; reconcile before resuming queue

Test the whole path by injecting a lost response after the simulated write succeeds. Assert one downstream booking and a truthful user-facing state. Separate the emergency stop, recovery and rollback actions: disabling new work does not cancel an external effect already in flight.

A register, not a demo

Production AI systems can fail through hallucination, stale context, sycophancy, prompt injection, unsafe tool use, schema drift, weak fallback, and observability gaps. The register must be based on the system’s own threat model, incident history, and evaluation results.

The mature move is to name the failure modes, add controls, test them, monitor them, and assign ownership. A demo shows what works once. The register makes the required evidence visible; only implemented controls and observed tests show how the system behaves.

Read next

Continue through the same learning path with the next practical articles.

Take it further

Hand-picked external courses that go deeper on this topic.

AWS Skill Builder

AWS Security: Securing Generative AI on AWS

AWS Training and Certification

A cloud-vendor-specific complement to the Macquarie specialization: AWS's own Generative AI Security Scoping Matrix, OWASP Top 10 for LLMs, and MITRE ATLAS, walked through governance, legal, and compliance controls for five different AI deployment scopes — from consumer apps to self-trained models. Not GDPR-specific, but a genuinely practical advanced pick for teams whose AI workloads actually run on AWS and need concrete data-governance and compliance controls, not just theory.

Advanced~2 hours · self-paced (9 modules)Verified 8 days ago
Coursera · Macquarie University

Cyber Security: Data, Privacy and AI Security

Macquarie University Cyber Security Hub faculty

An advanced pick for professionals responsible for both GDPR compliance and AI security: a three-course specialization from Macquarie University's Cyber Security Hub that goes from GDPR/CCPA fundamentals and privacy-by-design, through privacy impact assessments, to a dedicated third course on securing AI systems against adversarial attacks and model leakage. It genuinely bridges the two rather than treating them as separate topics.

Advanced~47 hours · self-paced (3-course specialization)Verified 8 days ago
EU Digital Skills & Jobs Platform · CyberSuite

Secure AI Adoption for SMEs: Cybersecurity and the EU AI Act

CyberSuite

The rare AI Act course written for the companies the Act actually reaches: SMEs adopting AI, not the labs building it. Hosted on the European Commission's own skills platform, it pairs the legal side — roles, obligations, risk classification — with the security side (prompt injection, data leakage, supplier due diligence) that most compliance courses skip. For an Estonian SME deploying AI, this is the practical starting point.

Advanced~15 hours · self-pacedVerified 8 days ago

See all courses for AI Safety & Data Privacy