Skip to content

Lesson · Module 2 · GenAI System Design

Containing failure: checks, fallbacks and human review

Plan for wrong, unsafe and unavailable output: layered checks, graceful degradation, where a person belongs in the loop, and how to roll back a model or prompt change.

Level: IntermediateDifficulty: Medium7 chapters1 table1 diagram

Builds on

A language model will sometimes produce output that is wrong, unsafe, malformed or simply unavailable, and no amount of prompt polishing eliminates these failures. The design question is therefore not how to make the model perfect. It is what the rest of the system does in the moment the model is not, and how much damage one bad output can do before something stops it.

That question has four parts, and keeping them apart makes designs easier to reason about:

  • Prevent. Make a bad output less likely: clearer instructions, better retrieved context, constrained output formats.
  • Detect. Notice a bad output that happened anyway, before or soon after it reaches a user.
  • Contain. Limit what a bad output can do, whether or not anyone notices it.
  • Recover. Return to a good state: a fallback for the current request, and a rollback for the change that caused the problem.

Prevention is probabilistic, and detection is imperfect, which is why containment and recovery carry more weight than they first seem to. Most of this lesson is about the last three.

One scenario, with assumptions labelled#

Everything below uses a hypothetical support assistant for a meal-kit subscription company. A customer writes in, the assistant answers using the company's policies and the customer's account, and it can issue a delivery credit through a tool. All figures are assumptions chosen for readable arithmetic, not observations from a real system.

  • 50,000 requests per day.
  • Policy answers are the bulk of traffic. A smaller share asks for a credit after a missed or damaged delivery.
  • Suppose 2 percent of the model's raw outputs are bad in a way that matters: a misquoted policy, an invented fee, a credit that should not be granted, a leak of another customer's details, or output the downstream code cannot parse.

Different failures call for different defenses, so it helps to name them before choosing any:

FailureExample hereBest detectorTypical response
MalformedThe credit request is not valid JSON, or names a plan that does not existParser and schema validationRetry once, then fall back
Wrong but fluentStates a 14-day window when the policy says 30Grounding checks against retrieved text, sampled review, evaluationBlock when detected. An undetected error may surface later through sampled review, user feedback, audits or comparison with trustworthy ground truth; monitoring shows signals and changes but does not establish correctness
Unauthorized or unsafeGrants a credit above the limit, or reveals data from another accountDeterministic authorization in codeRefuse the action; log it
Out of scope or low confidenceA legal threat, or a question the policies do not coverClassifier, retrieval score, rulesRoute to a person
Unavailable or slowProvider outage, timeouts, rate limitingTimeouts, error rates, health checksDegraded mode

The "wrong but fluent" row is the hard one. Nothing about it looks broken, so the strongest defenses are not clever detectors but reducing what a wrong answer is allowed to cause.

Bound the damage before you inspect the output#

The most reliable layer is the one that does not depend on the model behaving. Treat model output as a proposal and let ordinary code decide what happens. OWASP's list of risks for LLM applications gives these concerns their own entries, including improper output handling (when a downstream component trusts model output without checking it) and excessive agency (when the model is given more capability than the task needs) (OWASP Top 10 for LLM Applications, 2025).

For the credit tool, that means:

  • The customer's identity comes from the authenticated session, never from text the model produced. The model cannot ask for another account's data because no code path accepts that parameter from it.
  • The credit amount is computed or capped by code from the order record. The model can propose "issue a credit for the missed box", and the service decides the amount, checks the order was actually missed, and confirms no credit has been issued already.
  • The tool is least-privilege: it can issue a credit up to a fixed limit and nothing else. Anything above the limit becomes a request that a person approves.
  • Actions are idempotent, so a retry after a timeout does not credit the customer twice.

Each of these keeps working when the model is confused, jailbroken or has just been updated, which is the property a model-based check cannot offer. When you design a failure-handling answer, put these constraints first and say that the checks that follow are for what the constraints cannot express.

Layered checks, and what each layer can see#

Checks are ordered by how much to trust them. Cheap deterministic checks go first and catch a lot. Statistical checks go after them and catch what rules cannot express.

Input side. Validate sizes and formats, authenticate the caller and rate-limit. Screen for known abuse patterns, with the expectation that screens miss things. Retrieved documents deserve special care, because a document can contain text that reads like an instruction, and the model may follow it. OWASP's entry on prompt injection says it is unclear whether fool-proof prevention exists, given how models work, and recommends reducing impact instead: constrain behavior, validate outputs, filter inputs and outputs, enforce least privilege and require human approval for high-risk actions (OWASP LLM01:2025). That is the design stance to copy. Keep retrieved content clearly separated from instructions, grant it no authority over tools, and rely on the constraints above for the cases that matter.

Output side, deterministic. Parse against a schema and reject anything that does not conform. Check that identifiers exist, that amounts are within range, that a cited policy section is among the passages retrieved, and that the reply contains no pattern it must not contain, such as an account number. These checks are exact, cheap and explainable, so apply them to every response.

Output side, statistical. For wrong-but-fluent output, a second component can judge whether each claim in the reply is supported by the retrieved policy text. This can be a classifier or another model call. It catches things rules cannot, and it has limits worth stating plainly:

  • It makes mistakes of both kinds. It lets some bad answers through and flags some good ones.
  • It can share blind spots with the model it checks, especially when it is the same model or the same family, so its errors are not independent of the first model's.
  • It reads text an attacker may control, so it can be targeted by the same injection that fooled the first model.
  • It adds latency and cost to every checked request.
Warning

A second model does not guarantee correctness, and a single guardrail does not make a system safe. Treat a model-based check as one more imperfect classifier. Measure its misses and its false alarms on labelled examples, the way the evaluation lesson describes, before counting on it.

The scenario's numbers show why this matters. Of 50,000 daily requests, 2 percent bad is 1,000 bad outputs and 49,000 good ones. Say the statistical check catches 90 percent of bad outputs and wrongly flags 3 percent of good ones:

caught            = 0.90 * 1,000   =   900
slip through      = 0.10 * 1,000   =   100 per day
good, but flagged = 0.03 * 49,000  = 1,470 per day
total flagged     = 900 + 1,470    = 2,370 per day

Two things follow. A 90 percent catch rate still lets 100 bad answers through every day, so the check reduces the problem and does not remove it. And most of what it flags is fine: 1,470 of the 2,370 flagged items (62 percent) are good answers. Stacking a second, independent check with the same 90 percent catch rate would cut the misses to about 10 a day, but "independent" is an assumption that is often false for model-based checks. If both fail on the same kind of input, the second adds little. The only way to know is to measure them together on labelled failures.

Designing the degraded mode before you need it#

When a check fires or the model is down, the system needs a defined behavior, and that behavior has to be built and exercised in advance. A common ladder, from least to most degraded:

  1. Retry once, within a deadline derived from the latency budget. A retry that happens after the user has already given up helps no one. Retries need a cap and some jitter, because a provider that is struggling can be pushed over by clients that all retry at once.
  2. Switch to an alternative. Another model or provider, or a conservative prompt. This keeps generating, but it may behave differently, so the alternative has to be evaluated before an outage and not discovered during it.
  3. Skip generation. Where the fallback contract for this request type defines it, show the retrieved policy passages with a short fixed message, or a templated answer for the most common questions. This is less fluent and writes no new text, but it is only as good as the passages, so the contract says what is allowed: passages filtered by the user's permissions, taken from approved current sources, and labelled as source text and not presented as an answer.
  4. Hand off to a person, when policy requires it, residual uncertainty remains, or the earlier steps failed or ran out of time. Queue the conversation for an agent and tell the customer what happens next.
  5. Fail honestly. A clear error is better than a confident wrong answer.

The ladder is conditional, not a sequence every request walks. A response that passes the checks is used, and the system stops there. Output from a retry or from the alternative is not trusted because it is a fallback: it goes through the same schema, rule and grounding checks as the first attempt, and nothing reaches the user or an action handler unchecked. The system moves to the next step only when the previous one is rejected, errors or cannot finish in the time left. A person is involved when policy requires it (an action above the limit, a legal topic), when uncertainty remains, or when nothing else in the ladder is available, and not as an automatic last stop for every failure.

Each recovery attempt has its own accept or reject branch, every attempt's output passes the checks again, and the system moves to the next fallback only when the previous one fails or runs out of time

Two design decisions are easy to skip. First, each check needs a stated failure mode: when the grounding checker itself is down, does the system fail closed (block the answer) or fail open (send it unchecked)? The answer depends on the consequence. Failing open for a harmless FAQ may be right, and failing open for a credit above a limit is not. Second, the fallbacks need regular testing, for example by injecting provider errors in a staging environment or a controlled production drill. An untested fallback is code that has never run, and it will fail at the moment it is needed.

Streaming, if the product uses it, interacts with all of this. Checking a complete answer before the user sees it conflicts with showing it as it generates. Choose per request type: stream and accept the risk for low-stakes answers, or buffer and check for the ones where a retracted statement would matter.

Putting a person in the loop on purpose#

"Add human review" is not a design until it says which cases, at what point and with what capacity. There are three distinct placements:

  • Before the action. A person approves a credit above the limit, or a reply to a legal complaint. This is the strongest control, and it adds delay for that request.
  • After the fact, on a sample. A team reviews a random slice of sent answers to estimate quality and to find failures nothing flagged. It does not stop an individual bad answer, but it produces the evidence that tells you whether the automated checks work.
  • On escalation. Low-confidence or out-of-scope requests go to a person who answers directly.

Review capacity is a budget, and the earlier arithmetic shows how quickly it runs out. Sending every flagged item to a person means 2,370 reviews a day. At an assumed 2 minutes each that is 4,740 minutes, about 79 hours. At an assumed 6 productive hours per reviewer that is about 13.2 reviewer-equivalents, so around 14 whole reviewers. Most of those reviews are false alarms. A better design reserves people for the cases where a wrong answer is costly and a person adds judgment: actions above the limit, legal and safety topics, and a thin sample. Lower-impact flags take an automated fallback such as step 3 of the ladder above. Whatever the choice, say what happens when the queue grows, because an unbounded review queue quietly turns into no review.

The reviewer is part of the system and can fail too. A literature review of human use of automation reports that people tend to over-trust automated aids and miss errors the aid does not flag, and that this tendency is not removed by training or instructions alone (Parasuraman and Manzey, 2010). That is a reason to design the review screen, not just to tell reviewers to be careful. A heuristic worth trying is to show the evidence (the retrieved policy text) next to the draft, to ask for the reviewer's judgment before revealing the model's suggestion for some cases, and to mix in known-bad drafts so the team can measure how often reviewers catch them. Verify that these help on your own reviewers before relying on them.

Changing the system safely: monitoring, canaries and rollback#

Many failures arrive through a change: a new prompt, a different model version, a new retrieval setting, a tightened threshold. Treat each as a release.

Make every response traceable. Log the prompt template version, the model identifier, the retrieval configuration, the check outcomes and whether a fallback ran. Without that, a bad answer in the logs cannot be tied to the change that caused it, and a rollback cannot be verified.

Watch rates, not only incidents. Useful signals include the share of outputs failing schema checks, the grounding-check flag rate, the fallback rate, the escalation rate, the human override rate in review, the refusal rate, latency, and cost. A change in one of these relative to its baseline is often the first sign of trouble, and the online signals from the evaluation lesson belong on the same dashboard.

Roll out in steps. Run the new version in shadow mode on live traffic, with its output compared but not shown. Then canary it: expose a small share of traffic and compare it to the control group. Google's SRE workbook describes canarying as a partial, time-limited deployment followed by an evaluation that decides whether to proceed, advises choosing a small set of metrics that represent real problems and are attributable to the change, and says a canary that evaluates as bad should be rolled back or escalated to a person (Google SRE Workbook, Canarying Releases). Agree on the rollback trigger before the rollout, not during it.

Return to the scenario. The check flag rate is about 4.7 percent (2,370 of 50,000). A new prompt version goes to 5 percent of traffic, about 2,500 requests a day. The sampling noise on a rate near 4.7 percent with 2,500 requests is about:

margin ~ 1.96 * sqrt(p * (1 - p) / n) = 1.96 * sqrt(0.047 * 0.953 / 2500) ~ 0.0083   (about 0.8 points)

That figure is the approximate sampling margin of one 2,500-request estimate under the stated assumptions, with the 4.7 percent baseline treated as known. It is not the full uncertainty of a comparison: when the baseline is itself a measured estimate, or the control group is concurrent traffic, the noise in both groups⁠ has to be considered when judging the difference, and the margin alone does not make the rollout decision. With that caveat, a canary showing 9 percent is a substantial increase over a 4.7 percent baseline, well beyond what this margin could explain, and it would justify invoking the rollback trigger agreed in advance. A canary showing 5.0 percent is not distinguishable from the baseline on one day's traffic, so a small regression would need more traffic or more days to confirm. Count what the canary can and cannot detect before treating a quiet canary as good news. This is an approximation that assumes independent requests, as in the evaluation lesson.

Make rollback a switch, not a deployment. Keep the previous prompt, model and configuration available as one versioned bundle, and flip traffic back with a flag. A few cautions apply:

  • Pin the model by an explicit version where the provider allows it, and remember that the old version is a temporary safety net. One provider's published policy (documentation checked in October 2026) promises at least 60 days' notice before retiring a publicly released model, and says requests to a retired model fail (Anthropic, model deprecations). A rollback target that no longer exists is not a rollback.
  • Keep a conversation on one version. Switching mid-conversation can change behavior in the middle of an exchange.
  • Some effects do not roll back. Credits already issued, cached responses and indexes written by the new version remain until cleaned up. Include version identifiers in cache keys and index names so that returning to the old version means returning to old entries, and keep a way to find and reverse actions taken during the bad window.

Proportion, and what to say in the interview#

None of this means building every layer for every product. Match the response to the consequence. An internal tool that drafts text for an employee to edit needs schema checks, logging and a way to turn it off. An assistant that moves money needs hard authorization limits, human approval for large actions, tested fallbacks and staged rollouts. In a design discussion, start from the worst plausible bad output and what it would cost, put deterministic constraints around anything with side effects, add statistical checks for what rules cannot express while saying what they miss, define the degraded behavior, and name the rollback trigger.

Tip

Sample wording for a candidate: "The model only proposes. Authorization and amounts are enforced in code, so a bad output can't grant more than the limit. I'd validate structure on every response and add a grounding check for policy answers, knowing it will miss some and flag some good answers, so I'd measure both. If anything fails there's a fixed fallback, ending with a handoff to a person, and I'd test that path regularly. Prompt and model changes go out behind a flag as a canary with a rollback threshold agreed in advance."

A system that contains failure still needs a way to say whether it is good at all. That comes from the evaluation plan, which is why the checks, the review samples and the canary metrics above should all feed the same set of labelled examples.