Design a live incident summary that an on-call engineer can trust
Design a system that condenses chat, alerts and deploy events from a live incident into a handoff summary, and decide how to tell whether the summary can be trusted.
- Difficulty
- Medium
- Level
- Intermediate
Category: ML system design
Part of GenAI System Design / Reasoning about a model request: cost, knowledge and quality
Topics: summarization · evaluation · incident-response · long-context · faithfulness
The challenge
You're designing a feature for the incident-response tooling at a mid-sized software company. When an incident is declared, a dedicated chat channel opens. Responders talk there, monitoring alerts and deploy notifications are posted into the same timeline, and the team keeps a ticket whose status and comments change as the incident goes on. Incidents can run for many hours, so responsibility passes from one on-call engineer to another, and extra responders join part-way through.
I'd like you to design a system that keeps a handoff summary for each open incident, so that someone taking over can understand where things stand without reading the whole channel. The summary should cover four things: the current status and impact, the suspected causes with the evidence for each, the actions taken so far, and the open questions. It should be generated with a language model.
Here is what I can tell you about the setting.
- Inputs. Four sources feed each incident: chat messages, monitoring alerts, deploy events and ticket updates. Each one is an event with a timestamp, a source and an author or system name. Including that information, an event averages about 40 tokens.
- Size. A typical incident has 200 to 500 events. A few incidents a month are much larger, up to about 5,000 events over as long as 12 hours. At most about 30 incidents are open at the same moment.
- Model. Security has approved one hosted language model for this data, and nothing else may receive incident chat. Its context window is 32,000 tokens.
- Sensitivity. Chat often contains customer identifiers, and responders occasionally paste credentials by mistake.
- Reading. People open the summary at shift change or when they join an incident. It should open within 10 seconds and reflect every event up to 2 minutes before it was opened.
- Consequences. An engineer who acts on a confident but wrong cause can lose time or take a harmful action, such as rolling back a healthy service. A summary that leaves out an action already taken can get that action repeated.
- Labels. Nobody has labelled summaries. The team has postmortem documents written after past incidents, once the cause was known, and some hand-written handoff notes. People wrote a note for roughly one handoff in five, and the quality varies.
- Cost. Model spend matters, so the design shouldn't call the model more often than the freshness requirement needs.
Acting on an incident automatically, finding root causes, writing postmortems and designing the interface are out of scope. The company, the tooling and every figure above are hypothetical givens for this exercise.
Describe the design, and tell me how you would decide that the summaries are trustworthy enough to launch. Take the pieces in whatever order you like.
Before you start
Prerequisites
Knowledge applied
- Chunking long inputs for modelsKnowledge
Interview
Step 1 of 7
Frame
Understand the situation before you solve it: clarify what is asked, find out what you need to know, state what you assume.
Before reading on, list the questions you would ask the interviewer and the assumptions you would state.
Show how the reference frames the problem
Scope clarification
What the reference treats as the core. A handoff summary that a responder can check, kept current while the incident runs. The core is the four parts the prompt names: status and impact, suspected causes with evidence, actions taken, open questions.
What it leaves out. Automatic remediation, root-cause finding, postmortems and the interface. The reference touches the interface only where its data contract matters, such as letting a reader see the evidence behind a statement.
Assumed for this reference. The givens in the prompt stay as given. The statements below are assumptions the reference adds, and each one is used where it matters.
| Assumption | Where it matters |
|---|---|
| Every event has a stable ID from its source. Alert events say whether the alert fired or resolved. Chat replies point at the message they answer. | Evidence links, alert state, references across long distances |
| Events can arrive seconds late and out of timestamp order. A chat message can be edited or deleted after it was summarised. | The update cursor, retractions |
| Anyone who can read the incident channel may read its summary, and nobody else. | Storage and access |
| The summary is advisory. Responders decide, and the system takes no action on an incident. | The cost of an error, safe failure |
| The approved model can be asked for output in a fixed schema, and the system rejects output that does not parse. | Verification |
Questions a candidate would raise, still open. The answers would change the design, so the reference notes how.
- Who may read the summary of a restricted incident? An audience other than the channel's own changes where summaries are stored and how they are filtered.
- Where do severity, incident roles and declared status live: in the ticket, or only in chat? If the ticket holds them as fields, status can be read from those fields and the model is needed only for narrative.
- How do responders record a suspected cause today? If they already tag one, the system can report it and need not infer it.
- Is late delivery of events bounded? If a source can deliver minutes late, the 2-minute freshness has to be stated per source.
- What do the source systems' deletion and retention rules require, and may the summariser keep any version of a deleted or edited message? The answer decides what a replay can reproduce, as the data and labels sections explain.
Step 2 of 7
Requirements
Say what the system must do and how well, and name the constraints that will shape the design.
Show reference for Requirements
Functional requirements
- Ingest events from the four sources into one ordered timeline per incident, keeping each event's source, author or system, time and stable ID.
- Maintain, for each open incident, a handoff summary with the four parts the prompt names.
- Attach evidence to every statement about a cause, an action or an impact: the events it rests on, reachable from the summary.
- Show when the summary was produced and which events it covers, and keep every event after that point reachable.
- Reflect edits and deletions. A statement that rested on a deleted or corrected message is re-examined, and so is any derived representation built from it.
- Rebuild a summary from the full timeline on demand, for example after an outage of the pipeline or a change of prompt version.
- Keep working at the sizes given: incidents too long to read in one call, and many incidents open at once.
Requirements 5 and 6 are easy to defer and hard to add later, because they decide how events and statements are stored.
Non-functional requirements
| Requirement | Source | What it drives |
|---|---|---|
| In normal operation the summary covers events up to 2 minutes old, and the view says so when it does not | Prompt | Refresh cadence, a stated covered cursor and processing lag, and access to the events the summary has not absorbed |
| The view opens within 10 seconds, whatever the processing lag | Prompt | The read path never waits for a model call |
| A confident wrong cause costs the most, and a missing or repeated action costs too | Prompt | Evidence on every statement, mechanical listing of structured changes and alert transitions, the error costs in the ML objective |
| Pasted credentials and customer identifiers should not travel further than needed | Prompt (sensitivity) | Redaction before storage with no unredacted copy kept, access limited to the channel's audience |
| Model calls follow from the freshness requirement and no more often | Prompt (cost) | A cadence cap, skipping updates when nothing is new |
| The view stays useful when the model or the pipeline is unavailable | Follows from the 10-second read | A view assembled from stored claims and the stored event timeline |
Freshness in normal operation, and when degraded. The ingest watermark is the highest ingest sequence stored. The covered cursor is the highest ingest sequence such that every event up to it has been processed with nothing left unresolved, as the inference section defines. Processing lag is the age of the oldest event after the covered cursor. The 2-minute target is for normal operation, and the operational measure is the lag per incident: the share of time it stays within 2 minutes, and its 95th percentile. A provider failure, a rate limit or a sustained backlog can push the lag past 2 minutes, and nothing in this design guarantees a hard bound under those conditions. What the design does guarantee is different. The view opens within 10 seconds from stored data whatever the lag. It states the covered cursor and its timestamp, the most recent ingested event and the lag. It marks the summary stale when the lag exceeds the target. And every event after the covered cursor stays reachable from the view.
The numbers that decide the design. Sizes follow from the givens, with 3,000 tokens reserved in each call for instructions and the answer. The 3,000 is an assumption.
typical incident 200 to 500 events x 40 tokens = 8,000 to 20,000 tokens (fits in 32,000 with room)
largest incident 5,000 events x 40 tokens = 200,000 tokens
reading it once 200,000 / (32,000 - 3,000) = 6.9, so 7 calls just to read the timeline once
refresh ceiling 30 incidents / 1 minute = 30 updates per minute if every incident refreshed at the cap
Most incidents fit in one call, and the largest do not fit in six. The largest are also the ones with several shift changes, so they are the ones where handoff matters most. Reading the whole timeline on every refresh would multiply the cost of those incidents, which is the argument for maintaining the summary incrementally. Anatomy of a model request covers the token arithmetic behind these figures.
Step 3 of 7
Design
Sketch the core: the entities, the interfaces and a high-level design that meets the requirements.
Show reference for Design
High-level design
Baseline. The simplest design: when someone opens the view, collect the incident's events, send them to the model with instructions and show the answer. For a typical incident that fits in one call, needs no stored state and has nothing to drift. Build it first, because it gives a quality baseline to measure the later design against.
It fails in three ways the givens predict. The largest incidents do not fit. The reader waits for a model call, and the prompt allows 10 seconds. And every open produces a fresh free-text answer, so nothing ties a sentence to the events behind it and nothing can be checked mechanically.
The design that follows from those failures. Two changes. The summary is maintained as events arrive and read from storage, so reading never waits on a model. And the model produces claims, each one a single statement with a type (status, hypothesis, action or open question), the events that support it and a state (active, superseded, refuted or settled). The summary is a view over the claims.
Each component answers a requirement. Normalisation gives stable IDs for evidence. Redaction answers the sensitivity given. The event store keeps every source event as its own record, ordered by ingest sequence, which is what makes evidence inspection, replay and rebuild possible. Alert floods are handled when the model input is built, by grouping repeats, and the store is not altered by it. The update worker keeps cost per refresh roughly flat as an incident grows. The checks stand between a model's proposal and anything a responder reads, in the way Containing failure treats model output: a proposal that code decides about. The mechanical sections list structured deploys, rollbacks, configuration changes and alert transitions straight from their source events, so the model cannot drop them. They do not cover an action that a person reports only in chat or in a ticket comment. That is a claim the model extracts from text, validated and evaluated like any other, and the design does not promise that every such action is found.
What the view shows. The claims grouped by type, each with its evidence a click away. A status line gives the covered cursor and its timestamp, the most recent ingested event and the processing lag, and marks the summary stale when the lag is over the target. Beneath that, the mechanical sections: every structured deploy, rollback and configuration change in the incident, and the alerts currently firing, each reachable to its source events. A firing and a later resolution are separate transitions, and the section shows the state the alert was left in. Then the timeline after the covered cursor, newest first, with a count and each event's processing status. Older pages of that timeline load on request, so nothing after the cursor is hidden. If any event is unresolved or failed, the status line says the summary is incomplete and names how many.
This is what the view offers in place of a freshness promise it cannot always keep: a summary that is complete through a stated cursor, kept apart from the events after it, all of which the reader can open.
Step 4 of 7
ML Reasoning
Frame the learning problem: objective, data and labels, features, model, and how it trains and serves.
Show reference for ML Reasoning
ML objective
What the model is asked to produce. Not a summary. It is asked to propose changes to a set of claims, each resting on named events: add a claim, update one with new evidence, or retire one that later events contradict or settle. This turns "write a good summary", which only a reader can judge, into smaller statements that can each be checked against the events they cite.
What a mistake costs. The prompt gives the consequences, and they rank the errors:
- An unsupported cause stated with confidence. An engineer may act on it. This is the costliest.
- A missing or wrong action. A rollback gets repeated, or two people make conflicting changes.
- A stale status. The picture of impact is out of date. It may be noticed sooner than the others, because status is easy to check against the channel.
- A missing minor detail. The reader has to search for it.
This ordering is a judgement, and the people who run incidents should set it. It shows why coverage and faithfulness pull against each other. The harder the model is pushed to include everything, the more it paraphrases and infers, and inference is where error 1 comes from. Two controls follow. Part of error 2 can be removed mechanically: structured deploys, rollbacks, configuration changes and alert transitions are listed from their source events by code and not left to the model's selection. Actions that people report only in chat or in a ticket comment have no structured record, so they stay claims that the model extracts, with evidence, and they are validated and measured, not assumed complete. And causes are reported with who suspected them and what supports them, not asserted by the system.
Safe failure. Every claim type has a way to say less. A hypothesis can be listed as "raised by a responder, no supporting events yet", an open question can say the point is unclear, and when nothing can be said the view shows the raw timeline after the covered cursor. Saying less is a good outcome, and the evaluation should not penalise it as an omission unless the information was available in the events.
What the objective leaves out. Fluency. A summary can read well and be wrong, so readability is not a launch criterion, and the metrics in the evaluation section do not measure it.
Data
The event stream is the data. There is no training set in the first version, because the model is prompted and not trained. What matters is how the events are prepared, since the model sees only what the preparation step produces.
- Normalise. One timestamp convention, stable event IDs (source plus the source's own ID plus a version), the author or system, and for chat the ID of the message it replies to. Each source event is stored as its own record, and an edit stores a new version of it. Keeping reply links matters: a reply that says "that deploy is the cause" is useless if the deploy is not in view.
- Group repeats for the model, never in the store. A single alert rule can fire hundreds of times in a storm. The canonical store keeps every event as its own record. When the model input is built, repeats of the same rule on the same resource in a window are grouped into one line with a count, the first and last time, and the IDs of every event in the group. The grouping is a derived view that can be rebuilt, and it saves tokens and keeps a storm from crowding out the three human messages that matter. A firing and a resolution are different transitions and are never grouped together: a resolution ends a group, so the line shows the state the alert was left in. An evidence link to a grouped line resolves to the individual events.
- Redact. Replace credential patterns and customer identifiers with typed placeholders before an event is stored. The store holds only the redacted record, nothing keeps an unredacted copy of event content, and grouped lines and model input are derived from the redacted records. Detection is imperfect, so treat redaction as one layer. The stored events and summaries carry the same access rules as the channel.
- Deletion, edits and retention. What follows is a proposed policy. It rests on the answer to the open question about the source systems' rules, which this Problem does not have, and it invents no legal or privacy requirement. A deletion in the source removes the stored content of that event in every version, any grouped line containing it is rebuilt without it, and cached model inputs, prompt logs, search indexes and copies in evaluation sets are purged on a bounded schedule. Claims that cited it are withdrawn from the view at once and then re-derived from their remaining evidence or, when none supports a claim, left withdrawn, and earlier text versions of those claims are removed, because a claim can repeat the deleted content. A claim's identity, version numbers and timestamps may remain without text, as the operation rules specify. What may remain is a tombstone with the event ID, source, ingest sequence, event time and deletion time, and no content, so that sequence numbers, the cursor and "an event was deleted here" stay coherent. A tombstone counts as resolved for the cursor. Whether even that may be kept follows the approved policy. An edit is handled the same way. The earlier version is kept only if the source's rules allow retaining prior content. Otherwise the edit replaces the content, a version marker remains, and the earlier text is not recoverable. Operation records, retry payloads and conflict candidates are purged as well, as the operation rules in the inference section specify. Deleted content is not kept indefinitely so that replay can use it.
- Order by ingest sequence. The summariser's cursor is the sequence number at which an event reached the store, not its timestamp. A cursor on timestamps silently skips an event that arrives late with an older time, and that event is then never summarised.
- Treat text as data. Chat and alert text can contain wording that reads like an instruction, because people paste logs and alerts quote user input. Events go into the prompt in a clearly delimited block and the model has no tools, so a hostile line can at most distort a proposal, and the checks and the evaluation are where that would show up. No wording of the prompt is a complete defence on its own. Grounding answers in your own data makes the same point about retrieved text.
Drift. New services, alert formats and team abbreviations appear over time. The preparation step and the prompt will meet inputs they were not tested on, which is a reason to keep evaluating after launch and not only before it.
Labels
What exists, and why it misleads. The givens offer two sources of reference text, and each carries a bias.
- Hand-written handoff notes exist for about one handoff in five. Someone chose to write them, probably when they had time or the incident was hard, and they record what that writer found worth recording. They show what a human includes, not what a summary must include, and they were written without any system to compare against.
- Postmortems were written after resolution, with the cause known. A live summary never has that. Scoring a summary against a postmortem rewards a system for knowing the answer in advance, and training on them would teach hindsight.
Neither is a reference summary, and treating either as one would measure the wrong thing.
What to build instead: labelled replays. This needs complete event histories, and the givens do not promise them. They offer postmortems and some handoff notes, not retained timelines. The plan therefore rests on an additional assumption, stated here and not given by the prompt: for some past incidents, the full event history was retained with enough fidelity to replay, meaning every event with its source ID and time, and the versions that existed at each cut. It stays an unverified assumption until the source systems' retention rules and the history actually available are confirmed. How many incidents meet it, and how far back they go, is a question to ask first.
Deletion limits what a replay can promise. A replay rebuilds the view from the events still retained, so it reflects deletions made since and does not reproduce the exact view as it stood before them. A cut is reconstructible only if every event up to it is still retained in the version that existed then. A cut that depended on an event since deleted, or on an edit whose earlier text is gone, is marked incomplete, is dropped from the scored set, and supports no claim of replay fidelity. It can still be kept as a test of how the design behaves with a gap. Constructed cases are synthetic and are not affected. If a retention policy allows prior versions to be kept, exact reproduction becomes possible for the content kept, and the plan can count those cuts as reconstructible.
An incident whose history is incomplete, for either reason, cannot be a replay example. A summary generated from part of the events would be judged against a story the reviewer cannot see, and a real omission could not be told from missing data. Use the complete subset. If it is small or skewed, build targeted cases for the failures that matter and are rare, such as a late alert, a deleted message, an alert that fired, resolved and fired again, a cause that was raised and then ruled out, and an action reported only in chat. Start retaining full event histories now, so replayable traces accumulate prospectively. Until they do, the launch evidence is what the subset and the constructed cases can show, and the plan should say that and not claim more.
For each chosen incident, pick cut points, for example each real handoff and a few moments shortly after key events and during quiet stretches. Generate the summary from the events up to the cut point only, through the production pipeline. Then have a responder who was not at that incident, or a reviewer who reads only events up to the cut, label two things:
- Claim support. For each claim, is it supported by the events it cites, partly supported, or unsupported? Also whether it is useful to a reader taking over.
- Coverage. Against a checklist derived from the events available at the cut: which actions, changes, status changes and unanswered questions should the summary have carried?
The postmortems still have a use. After the fact, they show what turned out to be true, which lets a team audit whether a hypothesis the summary listed was later refuted, and they help pick incidents for the replay set. They are not the target.
Cost and bias of this plan. Labelling takes reviewer time per cut point, so the set will be small, and the incidents available are the ones that happened and were retained, with their mix of services and severities. State that mix.
Model
The model is fixed by the givens. One hosted model is approved, so there is no model-selection decision. What remains is what to ask of it, in what format, and which steps should not use it.
Prompted, not trained. There are no labelled summaries to train on, the task shifts as services and tooling change, and a prompt change can be tested in a day. Training or tuning would be a later option, and the evidence for it would be a consistent failure that a fair prompt attempt did not remove.
What the prompt contains. In a fixed order, stable parts first: the instructions and the claim schema, the rules (every claim cites events, a cause is stated only as raised by a named responder, say "unclear" and do not guess), then the active claims, then the new events. The instructions and schema are identical on every call, so they are the part a provider's prefix caching can reuse where it is offered, and billing and expiry terms for that vary by provider, as Serving within a latency and cost budget explains.
What comes back. A list of operations in the schema: add a claim, update a claim, retire a claim, each with the IDs of the events it relies on. While a re-evaluation after a source deletion is outstanding, the response also carries a separate list of re-evaluation decisions, defined below. Use the lowest-randomness setting the provider offers and still expect run-to-run variation, which is why the evaluation repeats runs.
Where code beats the model. Anything that can be read from a structured event without judgement is read by code: the list of deploys, rollbacks and configuration changes, alert state transitions, the timeline after the covered cursor, redaction, grouping of repeated alerts and ID checks. That covers structured events only. A manual restart that a responder mentions in chat or a ticket comment has no structured record, so code cannot list it and the model must extract it from the text. The model is kept for what needs judgement, including that extraction: the current status in words, who suspects what, which open questions were answered, and which earlier claims later events settle.
An optional second check. A separate model call can judge whether a new or changed hypothesis is supported by the events it cites. It adds cost and latency only for the claims it checks, and it can catch unsupported statements that string checks miss. It also errs in both directions and can share blind spots with the model it checks, so measure it on labelled claims before relying on it, as Containing failure describes for any model-based check. It is one layer and not the guarantee.
Inference
The update loop. For each open incident, an update runs when there are new events and the cadence interval has passed since the last one. The 2-minute target comes from the prompt, so it is treated as an acceptance requirement for normal operation and the initial cadence is 1 minute. The lag is roughly the cadence plus the time an update takes plus any queueing, so the target holds only while updates finish well inside the remaining minute. The update duration is not given and has to be measured. If the measured lag runs over, the options are a shorter cadence at a higher cost, or an explicit change to the target. A shorter cadence does nothing for a provider outage or a sustained backlog, where the degraded behaviour below applies.
- Read events whose ingest sequence is above the incident's cursor, up to a cap of 100 events (about 4,000 tokens at most, fewer lines when repeated alerts are grouped).
- Build the prompt: instructions and schema, the active claims, the new events, with repeated alerts grouped and every event ID kept.
- Call the model and parse the operations.
- Verify each operation (next paragraph). When the update ends, each event in the batch has a processing status: absorbed, meaning no rejected operation cited it, or unresolved, meaning a rejected operation did cite it. If the whole response is unusable, or a rejected operation cites no valid event, every event in the batch is unresolved.
- In one transaction, apply the accepted operations, record each operation's status and each event's status and attempt count, and recompute the covered cursor. The identities and version checks specified below make a retry or a duplicate delivery safe.
- Record the prompt version, the model identifier and the range of events with every change.
Verification before anything is stored. The checks are deterministic. Operations must match the schema. Cited event IDs must exist in this incident's timeline, and the ID of a grouped line stands for the events in it. A claim that an action was taken must cite a deploy or ticket event, or a message by a human. Numbers, service names and identifiers in a claim's text must appear in a cited event. A retirement must name the event that settles the claim. These checks are exact and cheap, so they run on every operation. They show that a statement is anchored to real events. They cannot show that the events mean what the claim says, and that is what the labelled evaluation is for.
Partial rejection, retries and the cursor. This is a proposed design and not a description of an existing system, and its numbers are assumptions to tune. Two watermarks are kept apart. The ingest watermark moves whenever events arrive. The covered cursor is the largest prefix of events that are all absorbed, so it cannot pass an event whose processing is unresolved or failed, even when later events in the same batch were processed well. Claims can therefore reflect events beyond the cursor, and the view claims completeness only through the cursor.
- Retry. Unresolved events go to the front of the next input, together with their pending operations and the rejection reasons, ahead of newer events within the cap. Each retry adds one to the event's attempt count. A provider error or a timeout does not count, because it says nothing about the event. It only grows the lag.
- Terminal failure. After an assumed 3 attempts the event becomes failed and retries stop. It keeps its rejection reasons, appears in the timeline as not summarised, and holds the covered cursor where it is. Later events continue to be processed.
- Operator view. Per incident, the count and age of unresolved and failed events, and for each its status, attempts, rejection reasons, the operations proposed and any identity conflict with the operation it may duplicate. Any failed event on an open incident raises an alert. An operator or the incident owner can request recovery after a prompt or schema fix, as defined below, or mark it waived with a note that the view shows, and a waived event counts as resolved for the cursor. Nothing is dropped silently, and the raw events stay readable throughout, unless the source deletes them.
- What the cursor cannot see. Absorbed means that no operation citing the event was rejected. If the model proposes nothing for an action it should have recorded, no check notices. The evaluation finds those omissions.
Claim identity, operation identity and versions. This too is a proposed design, and the field names are illustrative. Four things are kept apart, because tying any two of them together loses or duplicates a claim.
- Logical claim identity (
logical_claim_id) says which claim is meant. The system assigns it, not the model, when anaddis first recorded. It never derives from wording or from evidence, and it survives rewording and updates. - Evidence (
evidence_ids) is provenance: the events a claim rests on. It is a list on the claim and on each operation. Many claims may cite the same event, and nothing requires evidence to be unique. - Claim version is a number the system increments each time an update or a retirement is applied or a withdrawal is recorded.
- Operation identity (
operation_id) says which add, update or retire this is. The system assigns it when it first records a proposal, as a pending operation, derived from the incident, the model call's own ID and the proposal's position in that response. A repeat delivery of the same response therefore yields the same IDs. A retried operation keeps itsoperation_id, its target and its expected version across attempts, and only its wording may change.
The model sees each active claim with its logical_claim_id and version. A proposal is an operation of one of three kinds. An add carries a response-local label, and the system maps each label to a new operation_id and a reserved logical_claim_id. An update or retire names the logical_claim_id it targets and the expected_claim_version. A retry of a rejected operation names the operation_id it replaces. Association therefore comes from durable pending state and an explicit reference the system checks, and never from comparing wording.
When a re-evaluation is outstanding, the model sees each one, identified by its invalidation_id and the withdrawn logical_claim_id, with the retained events. The response carries one re-evaluation decision for each, in a list separate from the operations. A decision names that invalidation_id and logical_claim_id and chooses exactly one of two values: replacement, naming the response-local label of an add in the same response, or no_replacement_warranted. It is not an operation and has no operation_id, kind or version, so there is no fourth operation kind. A replacement closes the obligation only after its add passes the checks and applies. A no_replacement_warranted is accepted only as a schema-valid decision that matches an outstanding invalidation and claim, and it then creates the content-free disposition record described under edits and deletions. An omitted decision, an empty operation list, a model failure, an invalid response, a rejected operation or a missing add leaves the obligation open. Decisions are keyed by invalidation_id and logical_claim_id, so delivering a response again records at most one disposition and, through the add's own operation_id, at most one replacement.
- Operation states. An operation is
pendingafter a check rejects it, and it is then retried. It ends in one of four terminal states:applied,failedafter the bounded attempts,supersededwhen its precondition went stale, orclosedby a reconciliation. A fifth state,identity_conflict, is a hold described below. A terminal state and its recorded outcome never change, even when a later source edit or deletion invalidates its effect, which is recorded separately as described under edits and deletions. An operation's target andexpected_claim_versionnever change under itsoperation_ideither. - Reworded retry. A retried
addreuses its reservedlogical_claim_id. When it applies, the claim exists once under that ID whatever the new wording says. - Duplicate delivery. Before anything else the system looks up the
operation_id. An operation that is applied, failed, superseded or closed is a no-op that returns its recorded outcome and repeats no effect. - Distinct claims with identical evidence. Two
addoperations get twooperation_idvalues and twological_claim_idvalues even when their claim type and cited events match. Neither is a retry of the other, and each keeps its own evidence links. - Updates and retirements. They apply only if the claim's version equals
expected_claim_version. A later legitimate update is a new operation with a newoperation_idagainst the current version, so it is not mistaken for an earlier one. Updates for one incident run one at a time, so a stale version arises from a retry, a duplicate delivery, an earlier operation in the same response or an operator edit. - Stale operations. If the target's version no longer equals
expected_claim_versionand theoperation_idis not applied, the operation becomessupersededwith the recorded outcome "stale at version N". The system records a successor with a newoperation_id, asupersedeslink to the old one, the same cited events and the claim's current version as its precondition, and presents it to the model with the claim's current state. The old operation is never revised, and delivering it again changes nothing and creates no second successor. If two operations in one response target the same claim at the same expected version, the first applies, and the second becomessupersededwith a successor against the new version, so both changes survive in order. An operation that targets a retired or withdrawn claim fails at once with the reason "target retired" or "target withdrawn", because no retry can succeed, and an operator can request recovery under the rule below, which never reactivates that claim. - Partial batch acceptance. Each operation has its own state. Applied operations are recorded in the same transaction and are never sent again. Pending operations and successors are presented again. Superseded, failed and closed operations are not, and only their successors are.
- Attempts and failure. Attempts are the per-event count already defined. Every retry and every stale successor adds one to the events its lineage cites, so a chain of supersessions ends at the same assumed limit, where the latest operation becomes
failed. A failed operation is terminal and visible, keeps its predecessors linked for audit, and holds the covered cursor like any failed event. - Recovery after terminal failure. A failed operation is never reopened. Its recorded outcome, rejection reasons, attempts and original identity stay as they were, and delivering it again is still a no-op. A recovery request, with the operator's name, time and reason, creates a recovery operation: new work with a new
operation_idand arecoverslink to the failed operation, so the lineage reads failed, then recovery. A recoveredaddreuses thelogical_claim_idthe failedaddreserved, since nothing was applied under it, and applies only if no claim holds that ID. A recovered update or retire keeps its kind and takes the target's current version as its precondition. The one exception is an update that failed with "target retired" or "target withdrawn". It reserved no ID, and such a claim's ID is never reused or reactivated, so its recovery is a newaddwith a freshlogical_claim_id, reserved when the recovery operation is created and kept for its retries. The recovery operation linksrecoversto the failed update andreplacesto that claim, which stays unchanged. A retire of a retired or withdrawn claim is already satisfied, so only a waiver applies. A retired target is no prompt or model fault, so the operator's confirmation that a replacement is wanted serves as the override and counts toward the cap, and it follows a view of active claims of the same type with overlapping evidence, one of which is flagged as a possible duplicate. A repeated request returns the open recovery operation and reserves no second ID. The model is shown the cited events as they are retained now, the target claim's current state and the earlier rejection reasons, and not the failed proposal's wording. The budget is bounded twice. A recovery request is accepted only if the prompt, schema or model version differs from the one in force at the failure, or the operator records an explicit override that counts like any other request. And a lineage accepts at most an assumed 2 recovery requests, each with a fresh allowance of the assumed 3 attempts. All epochs are kept for audit, so repeated requests cannot restore an unlimited budget. The lineage follows the failed operation's chain, so a fresh claim ID starts no new lineage, and the cap and the epoch allowance apply the same either way. When the cap is reached, only a waiver or a discard remains. - Cursor and event status. An event is resolved only when the latest operation of every lineage citing it is applied, or is closed or waived by an explicit decision, and the re-evaluation obligation of an invalidation is closed by a replacement that applies or a recorded
no_replacement_warranteddisposition. While a lineage has a pending operation, a successor not yet applied, a recovery operation in progress or an identity conflict, the event stays unresolved however many operations in the lineage have reached a terminal state. A recovery request moves a failed event back to unresolved, and the cursor stays held. A recovery operation that applies resolves the event, and the cursor can then advance over the contiguous resolved prefix. A recovery operation that fails makes the event failed again, with a new epoch recorded. A waiver counts as resolved for the cursor, is recorded and is shown. A superseded operation does not resolve its events by itself. Resolution is judged against the event's current content, so an edit returns the event to unresolved without touching any earlier operation, and the cursor, which is derived from event status, can fall below it until the re-evaluation applies. - Edits and deletions. Operation records are derived content, so the deletion policy applies to them as it does to claims. Content-bearing fields are the proposed wording, retry payloads, rejection reasons that quote event text and the content of an identity-conflict candidate. When a source event is deleted, or edited where the source's rules do not allow retaining the earlier text, those fields are purged from every operation that cites it, whatever its state: pending, conflict, failed, superseded or applied. What may remain is metadata: the
operation_id, kind, state, outcome code, links, timestamps, attempt counts, reconciliation decisions and the cited event IDs, in the same minimal form a tombstone may keep and subject to the approved policy. A pending operation, a conflict candidate or an unapplied successor that cited the deleted event is closed with the outcome "source deleted", and one that cited an edited event becomessupersededwith "source edited". Each is a first move into a terminal state. An operation already terminal keeps its state and outcome permanently, and only its content fields are purged. A separate invalidation record names the event, deleted or edited, the time, and the affected operations and claims, and holds only metadata the retention policy permits. A claim resting on the event is withdrawn at once, in the same transaction, while the original operation still says applied. The claim's text in every version derived from the event is purged. What may remain, subject to the approved policy, is metadata without content: thelogical_claim_id, claim type, version numbers, timestamps, the IDs and kinds of the operations that produced each version, and the cited event IDs. Withdrawal also records a version marker one above the last version, naming the invalidation record. A withdrawn claim is terminal like a retired one, so its ID is never active again. If retained evidence still supports work, a new operation with a newoperation_idis created from it, linked to the affected operation and the invalidation record, and re-evaluated. For a closed or supersededaddit reuses the reservedlogical_claim_id, since nothing was applied under it. For a withdrawn claim it is a newaddwith a freshlogical_claim_idand areplaceslink, shown only the retained events. Its precondition is that the claim is withdrawn at the marker version with no replacement yet, so a repeat creates no second claim. If no retained evidence remains, or a completed re-evaluation concludes that it supports no replacement, the claim stays withdrawn as its final state and the system issues no update, retire or replacement. A withdrawn claim accepts no operation, whether proposed by the model or generated by the system, and the marker version serves only as the replacement's precondition. Each invalidation record has aninvalidation_id. A validno_replacement_warranteddecision, as defined in the claim identity section, is recorded in that record as a disposition of the same name, with its time, the claim ID and the retained event IDs considered, and no content. It is not an operation against the claim and changes no operation's outcome. Anything else leaves the obligation open and the existing retry rules apply. The disposition counts neither as an attempt, a recovery request nor an operator waiver. Where the source allows retaining the earlier text, nothing is purged and re-evaluation is an ordinary update or retire against the current version. A failed lineage whose cited events are all deleted is marked closed in the invalidation record, its failed operation keeps its state, and recovery is not offered for it. Superseded predecessors citing the deleted event lose their content too, and a successor that cites only retained events keeps its own. Claim text derived from the event is removed, as the deletion policy says, and nothing in a replacement is built from it. Evaluation copies of operation records and retry payloads are purged the same way, and a cut whose operation history can no longer be reconstructed is marked incomplete. These closures and re-evaluations count neither as attempts nor as recovery requests, because the source changed and the model did not fail. The deleted event itself becomes a tombstone that counts as resolved, while every other event cited by a re-evaluated operation or a withdrawn claim stays unresolved until its replacement applies or the disposition is recorded. The disposition closes only this obligation, so a pending operation, a conflict or any other lineage involving the event still holds the cursor. - Identity conflict. It arises when a retry names an unknown or non-pending
operation_id, or when a newaddnames none while a pending or conflictingaddof the same claim type cites overlapping events, because only then is "a retry of the pending operation" a live explanation. The candidate becomesidentity_conflict. It keeps its cited events and its content, receives its own provisionaloperation_idand reservedlogical_claim_id, and is not applied. It is not an active claim and does not appear in the summary body. The operator view lists it beside the pending operation it may duplicate. Events cited by the candidate or by that pending operation stay unresolved, so the covered cursor cannot pass them. A conflict never times out into acceptance, and its age raises the same alert as a failed event. An operator or the incident owner reconciles it explicitly, and the decision is recorded with who and when. Retry of the pending operation: the candidate is closed with a link to it, and its wording is used for that operation's next attempt, which counts as one attempt. Distinct: the candidate becomes an ordinary pending operation under its own identity, passes the checks and, if applied, is a separate claim that keeps its own evidence although it shares events with another claim. Discard: the candidate is closed with a reason. A newaddthat resembles an already applied claim, with no pending predecessor, is a distinct claim by default. It is flagged as a possible duplicate for review without a hold, and the evaluation counts it as a duplicate if it is one.
Cost arithmetic, with assumed sizes. Take 1,500 tokens of instructions and about 3,000 tokens of active claims (roughly 40 claims of 75 tokens each, with retired claims kept in storage and out of the prompt). The largest incident averages 5,000 / 720 minutes, about 7 events a minute, so a 1-minute cycle brings about 7 events, or 280 tokens. One update reads about 1,500 + 3,000 + 280, close to 4,800 tokens, and at most about 8,500 when the cap of 100 events is hit.
incremental: 720 updates x 4,800 tokens = 3.5 million input tokens
regenerate: timeline grows from 0 to 200,000, mean 100,000
720 refreshes x 100,000 tokens = 72 million input tokens (about 20 times)
The regeneration figure leaves out the extra calls needed to combine windows, so it understates that design. Multiply by the model's current rates to price either one. Company-wide, 30 updates a minute, or 1,800 an hour, is a ceiling that holds only if every open incident has new events in every cycle. The cadence is the lever that sets the spend, and it is a product decision, because a longer interval is cheaper and leaves less margin against the 2-minute target.
Reading. The view assembles stored claims, the mechanical sections, the status line and the first page of the timeline after the covered cursor. It makes no model call, so opening it is a few reads from storage and does not depend on the provider's latency, or on how far behind the summary is.
A flood or a backlog. Grouping repeated alerts keeps most of a storm out of the model input. If more events than the cap remain, further updates run immediately, up to a per-incident limit per hour. Beyond the limit, updates return to the normal cadence, the lag grows, and the view marks the summary stale. The status line shows the growing lag and the count of unabsorbed events, with the newest as a preview. The whole backlog stays reachable. Older pages load from the event store, a page at a time, without waiting for any summarisation, so an unabsorbed action from early in the backlog is never hidden behind a count. The summary is complete through the cursor. The events after it are shown with their status, absorbed, unresolved, failed or not yet processed, so the view keeps what is covered apart from what is not. The backlog drains when the storm passes.
Rebuild. Replay the stored timeline through the same update loop, in ingest order, in batches. It is the same code as the live path, so a rebuilt summary is what the live one would have been from the events still retained, apart from model variation. After a deletion it reflects the deletion and does not reproduce the earlier view. It is also how a new prompt version is tried against a past incident.
Step 5 of 7
Evaluate & Scale
Show how you would know it works and how it grows: evaluation, integration and production scale.
Show reference for Evaluate & Scale
Evaluation
The plan comes before the build, in the order its decisions depend on each other, as Evaluating output you can't unit-test lays out for any generated output.
Offline: the replay set. Use the labelled replays from the labels section, drawn from the incidents with complete histories and from the constructed cases. As an illustration of the plan, and not a promise that the data exists: if 40 incidents have complete retained histories, 3 cut points each and about 12 claims per summary make 1,440 claims to label. Report by claim type and never as one blended score:
- Supported-claim rate. The share of claims supported by their cited events. Hypotheses are reported on their own and held to the strictest bar.
- Evidence validity. A claim can cite a real event that says nothing relevant, which passes every string check. Label whether the cited events actually support the claim.
- Coverage. Recall against a checklist derived from the events available at each cut. Structured deploys, rollbacks, configuration changes and alert transitions are listed from their source events, so a miss there is a defect to fix and not a rate to report. Actions reported in chat or ticket text are extracted by the model, so their recall is a measured rate. A missing or duplicated consequential action is an evaluation failure and is read case by case. That includes an action whose events were unresolved or failed at the cut, which counts as missing, and a retry that produced a duplicate claim. Status and open questions are judged by people.
- Operations. Processing lag per incident (the share of time within 2 minutes, and the 95th percentile), backlog size, rejected-operation rate, retry rate, unresolved and failed event counts, duplicate claims, failed updates, tokens per incident hour.
Set the launch bars before running the replay, with the people who run incidents, from the error costs in the ML objective.
Five constructed cases for identity, retries and deletion. They are test designs for the replay harness, described by their expected behaviour. This Problem reports no result for them and does not claim they were run.
Case 1, a reworded retry. One chat event E1 says that deploy 4812 was rolled back. The first response proposes one action claim citing E1 as operation O1, with reserved claim C1, and it fails a check, so O1 is pending. The retry names O1 and words the claim differently, and passes. In a second run, O1 is applied and then delivered a second time. It passes if:
- exactly one claim exists, its
logical_claim_idis C1 in both runs, and its wording may differ from the first proposal; - O1 is applied once, the claim's version and history show no second entry, and no side effect is repeated;
- the second delivery leaves the claim state unchanged;
- E1 is not absorbed and the covered cursor does not pass it while O1 is pending, and the cursor passes it once O1 is applied and every earlier event is resolved;
- the scoring counts the rollback as covered once, neither missing nor duplicated.
Case 2, two claims with the same evidence. One chat event E2 says that the API was restarted and the cache was disabled. The response proposes two action claims, both citing E2 with the same claim type, as O2 and O3. O2 passes and O3 fails a check and is retried with new wording. It passes if:
- two claims exist, C2 and C3, with distinct
logical_claim_idvalues, each listing E2 in its evidence; - while O3 is pending, C2 stays applied, visible and unchanged, and E2 is unresolved, so the cursor holds below it;
- after the retry there are still exactly two claims, C2 is unchanged in wording, version and evidence, and C3 exists once;
- replaying O2, or delivering O3 again after it applied, changes nothing;
- the checklist holds two action items for E2, each matched to exactly one claim by its content, and neither is counted missing or duplicated because the event IDs overlap. A harness that matches on event ID alone fails this case.
Case 3, a lookalike that names no predecessor. Continue Case 2 at the point where O3 is pending. A later response proposes a cache-disable action claim citing E2 and names no operation, as O4. It passes if:
- O4 is recorded as
identity_conflictwith its own provisional operation ID, a reserved claim ID and evidence E2, and it is not applied; - no active claim exists for O4 and the summary body is unchanged, C2 is unaffected, and the operator view lists O4 beside O3;
- E2 stays unresolved and the covered cursor stays below it for as long as the conflict remains, however many cycles pass;
- reconciling O4 as a retry of O3 closes O4 with a link to O3, and after O3 applies exactly one cache claim exists;
- in a second run, reconciling O4 as distinct leaves C2 and C3 unchanged and adds exactly one more claim that keeps its own evidence link to E2. The ground truth is that E2 holds one cache disable, so the scoring counts that extra claim as a duplicated action. The mechanism has preserved the identity it was told to preserve, and the duplicate count records that the operator's reconciliation was wrong, which keeps identity preservation apart from semantic correctness;
- neither run counts a missing action. The retry-of run counts no duplicate, and the distinct run counts exactly one.
Case 4, stale operations. Claim C5 is at version 1. One response proposes two updates, O5 and O6, both with expected version 1. It passes if:
- O5 applies and C5 moves to version 2, and O6 becomes
supersededwith the recorded outcome "stale at version 2"; - a successor O7 exists with a new operation ID, a link to O6, the same cited events and expected version 2, and the old O6 is not revised;
- delivering O6 again changes nothing and creates no second successor, and delivering O5 again changes nothing;
- the events cited by O6 stay unresolved and hold the cursor until O7 applies, and then C5 is at version 3 with O5 and O7 each recorded once and in that order;
- if O7 and each later successor also fail or go stale until the attempt limit, the last one is
failedand visible, the lineage links are intact and the cursor stays held.
Case 5, a deleted source under an applied claim. Claim C6 is at version 2. O8 added it citing E3 and E4, and O9 updated it citing E3 only. Both are applied. The source then deletes E3. It passes if:
- an invalidation record names E3, the deletion, O8, O9 and C6, and O8 and O9 stay
appliedwith their kind, outcome, IDs and cited event IDs, while their wording is purged; - C6 is withdrawn from the view at once, the text of versions 1 and 2 is gone from storage, caches, logs and evaluation copies, and what remains is the ID, type, version numbers, timestamps and a content-free version 3 marker naming the record;
- a model-proposed update or retire aimed at C6 at any version fails with "target withdrawn", and nothing reactivates C6;
- E3 is a tombstone that counts as resolved, and E4 is unresolved again, so the cursor does not pass E4;
- a replacement
add, O10, has a new operation ID, a fresh claim ID C7 and areplaceslink to C6, and sees only E4. Once it applies, E4 is resolved and the cursor advances, and delivering O10 again creates no second claim; - in another run, E4 remains and a completed re-evaluation explicitly concludes that it supports no replacement. C6 stays withdrawn, O8 and O9 stay
appliedwith their wording purged, and no update, retire or replacement is issued. Ano_replacement_warranteddisposition resolves the invalidation obligation for E4 without counting as an attempt, and the cursor advances past E4 only if no other unresolved lineage holds it. An empty, unavailable, invalid or rejected response records no disposition; - in a second run where E4 is deleted too, C6 stays withdrawn as its final state, no retire is issued, no replacement exists and no event holds the cursor.
How certain the numbers are. If 1,440 claims were independent, a 95 percent supported rate would carry a margin of about 1.96 x sqrt(0.95 x 0.05 / 1,440), roughly 1.1 points. They are not independent. Claims in one summary share an incident, and summaries cut from one incident share events, so the effective sample is closer to the 40 incidents than to the 1,440 claims, by an amount that depends on how much quality varies between incidents. Compute intervals over incidents, compare two versions on the same cut points with the paired analysis in Comparing two system versions on an evaluation set, and for rare costly failures such as an unsupported cause, report the count and read every case.
Model judges. People are the reference. A model judge can score claim support at scale once it has been calibrated against human labels from this data. A judge that sees only the cited events can confirm support but cannot see the event that contradicts a claim, so keep a human sample that reads the whole timeline. Repeat runs of each cut point for the comparisons that matter, and average within a cut point before comparing.
Online. Run in shadow first: summaries generated and stored, and not shown, so they can be compared with what responders did. Then pilot with a few teams where a responder can flag a claim as wrong or missing. Flags are sparse and skew toward strong reactions, so add sampled review of live summaries, and after resolution compare the summary's history with the postmortem to find hypotheses that turned out wrong. Watch the flag rate, the rejected-operation rate, the share of cause claims resting on a single event, lag and cost.
Before launch, besides the bars. The view works with the model switched off, the kill switch works, and a user outside the channel cannot read the summary.
Production and scaling
Long incidents. The claim store grows with the incident, and the prompt must not. Superseded and refuted claims leave the prompt, and a settled phase collapses to one line while its detail stays in storage.
Drift in the state: the consolidation audit. Each update sees the active claims and a small batch, never the whole timeline, so an early mistake can persist and a claim that should have been retired can linger. Every few hours, and when a handoff is requested on a long incident, process the stored timeline in independent windows, extract claims from each, and compare them with the stored ones. Disagreements are logged and sampled for review, and can mark a claim as disputed in the view. Chunking long inputs for models covers choosing windows: here, windows of 400 events (about 16,000 tokens), well below the context limit, overlapping by 60 events so that a reply up to 60 events after the message it points to stays visible in one window. Windows start every 340 events, so 5,000 events take 15 windows, about 240,000 input tokens for one audit. Audits at hours 3, 6, 9 and 12 of the largest incident take 4, 8, 11 and 15 windows, about 0.6 million tokens in all, roughly a sixth of the 3.5 million for the updates.
When parts fail. If the model is unavailable or rate limited, updates fail, the previous claims stay, and the lag grows. The view marks the summary stale and shows the unabsorbed timeline, as in the backlog paragraph above, and it does not claim that the 2-minute target is met. Retries use a cap and jitter. If the whole pipeline is down, responders still have the channel itself, because the summary is an aid and incident response must not depend on it.
A feedback loop. The summary changes what responders write. If someone pastes a listed hypothesis into chat, the restatement looks like new support the next time the model reads the channel. Count as support for a cause only alerts, deploys and human observations that do not quote the summary, record the source types behind each cause claim, and show a cause that rests on a single event or a single source type as such.
Releasing changes. Every change to the prompt, the schema, the model version or the redaction rules is a release. Replay it on the labelled set, run it in shadow, then expose it to a few incidents, with the rollback trigger agreed beforehand and the previous bundle one switch away, as Containing failure describes. Rolling back also means rebuilding the summaries the bad version touched, by replay. A kill switch hides the claims and leaves the mechanical sections and the channel.
Step 6 of 7
Depth & Trade-offs
Go deep where it matters and defend your choices: what each buys, what it costs, when you would choose differently.
Compare with the topics the reference goes deep on
Regenerate from the whole timeline, or maintain claims as events arrive?
Show reference: Regenerate from the whole timeline, or maintain claims as events arrive?
Choice. Maintain a store of claims and update it from new events. Keep regeneration from the whole timeline as the baseline, the audit and the rebuild path.
What it serves. The freshness requirement and the cost ceiling. An update reads the active claims and the new events, so it stays near 5,000 tokens however long the incident runs, while regeneration grows with the timeline and cannot be done in one call for the largest incidents. It also gives each statement a lifetime, a history and a reason for retirement, which regenerated text does not.
What it costs. State that can drift. Each update sees little of the timeline, so an early error can persist and a wrong retirement can hide a claim that mattered. The price is the audit pass, the retirement rules, a schema that has to change carefully, and two code paths that must agree.
When another choice is better. If the evaluation shows that regeneration meets the bars on incidents that fit in one call, regenerating on the cadence for those incidents removes drift where it is least needed, at up to about 20,000 tokens a refresh against 5,000. The comparison is measured, since quality at 20,000 tokens of input may be below quality at 8,000. For a feature with few incidents and slow event rates, regenerating when someone opens the view may be cheaper than updating on a schedule, at the price of the model wait that the 10-second requirement rules out unless the previous summary is shown first.
Should the system propose causes, or only report the ones responders raise?
Show reference: Should the system propose causes, or only report the ones responders raise?
Choice. The model reports causes that responders have raised, attributed to them and with their supporting events. Separately, code lists correlations: deploys, rollbacks and configuration changes shortly before the first alert, labelled as timing and not as cause. The model does not propose a cause that nobody has stated.
What it serves. The costliest error in the ML objective. An invented cause cannot appear, and a reader can see who holds each belief and why.
What it costs. Help where help is most wanted. If no responder has spotted the cause, the summary does not find it, and a model that notices a pattern across alerts and deploys might have. The reference does not promise root-cause finding, which the prompt puts out of scope.
When another choice is better. If replay shows that model-proposed hypotheses are supported at a rate the people who run incidents accept, including on incidents where the cause was not obvious, show them in a separate list labelled as model suggestions, and measure whether responders act on them differently. The measured precision on hypotheses is the condition, not the average over all claims. If the ticket already has a structured cause field, report that field and skip both.
Free-text summary, or structured claims with evidence?
Show reference: Free-text summary, or structured claims with evidence?
Choice. Structured claims, each with a type, evidence IDs and a state, rendered by a template. The model's free text is limited to the short statement inside each claim.
What it serves. Checkability. A claim can be verified against its cited events, retired with a reason, compared between versions and shown with its evidence. Reading is decoupled from generation, so the view loads from storage.
What it costs. Fluency, and fit. A chain of events that explains something can be split across claims or flattened into one that loses the chain, and a rigid schema can push the model to force-fit. The schema, the renderer and the migrations when the schema changes are real work.
When another choice is better. For a first prototype on typical incidents, a free-text answer plus a list of the cited events is enough, and it is the baseline the structured design should be measured against. It is also the right shape if the evaluation shows readers value connected narrative over checkable statements. Even then, ask for citations, because an uncited summary cannot be checked at all.
Step 7 of 7
Follow-ups
Expect to be pushed on your decisions, your blind spots and the depth of your understanding.
Show the follow-up questions
What changes if the longest incident were ten times longer?
Show reference: What changes if the longest incident were ten times longer?
What it probes. Which costs grow with incident length and which do not, and whether the candidate sees that the context limit applies to the prompt and not to the incident.
A strong answer covers. The cost of an update stays flat, because it reads active claims plus new events, while the claim store grows, so retirement, merging and collapsing settled phases carry more weight. The audit and any full rebuild grow linearly with events, so they become rarer or more targeted. If active claims cannot be kept bounded, the state needs a hierarchy: a short current picture plus per-phase records fetched on demand. A reader joining a very long incident may want the current situation more than the history, which changes what the summary leads with. Defensible alternatives include time-bucketed summaries and a separate summary of the last few hours; the answer should say what evidence would choose between them.
Responders now trust the summary and stop reading the channel. What do you change?
Show reference: Responders now trust the summary and stop reading the channel. What do you change?
What it probes. Whether the candidate treats the system as part of a loop with its users and not as a component that is simply right or wrong.
A strong answer covers. Measuring reliance: how often readers open the evidence, and how often a flagged claim was one nobody checked. Keeping evidence one step away and showing uncertainty in the claim itself, where a general disclaimer would be ignored. Showing the covered cursor, the lag and the unsummarised timeline so readers can see what the summary has not seen. Checking that the summary's own text is not counted as support, as in the feedback loop above. Sampling reviews that seed known-bad claims to measure how often they are noticed. Weak answers stop at telling people to verify.
The provider announces a new version of the approved model. How do you decide whether to move to it?
Show reference: The provider announces a new version of the approved model. How do you decide whether to move to it?
What it probes. Release discipline for a change the team did not author, and whether the candidate has the means to compare two versions.
A strong answer covers. Replaying the labelled set through both versions on the same cut points, with repeated runs, and comparing them per incident with a paired analysis. Reading the breakdown by claim type and the rare costly failures one by one, since an average can hide a regression in hypotheses. Re-checking any model judge against human labels if the judge changed. Running in shadow and then on a few incidents, with the version recorded on every change and a rollback path. Allowing for the old version being available only for a limited time, which bounds how long a comparison against it stays possible.
Review
After the interview: study one strong approach, then review your reasoning.
Reference Solution
One strong approach, consolidated. Other designs can be just as strong; the self-review judges reasoning, not this architecture.
Show the reference solution
Reference design
One design that satisfies the requirements, built in the order a candidate could present it.
- Baseline. Regenerate the summary from the full timeline for incidents that fit in one call, with claims and evidence IDs in the output. Measure it on the replay set first.
- Prepare events. Normalise and redact, and store each source event as its own record with a stable ID, ordered by ingest sequence. Group repeated alerts only when building model input, keeping every event ID.
- Mechanical sections. Structured deploys, rollbacks, configuration changes and alert transitions are listed from their source events by code. Actions reported in chat or tickets are evidence-linked claims.
- Claim store. Maintained by an update worker that asks the approved model for operations on the active claims, with a cadence cap, and verified by deterministic checks before storage. Each event carries a processing status, claims have system-assigned identities, operations have stable IDs and checked versions, retries are bounded and the covered cursor passes only resolved events. The output contract is the baseline's, and only the input changes.
- Causes. Reported as raised by named responders with supporting events. Code lists correlations as timing.
- View. Reads storage only. It shows the covered cursor, the processing lag and a stale marker, and gives access to the whole unabsorbed timeline.
- Audit and rebuild. Periodic window-based audit against the stored claims, and rebuild by replay.
- Evidence of trust. Labelled replays of incidents with complete retained histories, plus constructed cases, cut at real handoff times, labelled by people who were not there, claim by claim, by type, with intervals over incidents. Then shadow, a pilot and a kill switch.
Sample wording for a candidate: "I'd make the summary a set of claims, each tied to the events that support it, so a responder can check any statement. The model proposes changes to the claims and code verifies them. Deploys and alert changes come straight from the events, and actions people report in chat are claims I'd measure, because missing one is expensive. I wouldn't let the model name a cause nobody has raised. To decide whether it's ready, I'd replay past incidents cut at real handoff times, label claim by claim with people who weren't there, and read the unsupported causes one at a time instead of trusting the average."
Self-review
Compare your reasoning with what interviewers look for, at the bar you are preparing for.
Open self-review
Expectations by seniority
This Problem targets mid-level and senior candidates. The expectations below are for what a strong answer shows at each level, and a candidate who takes a different defensible route can still meet them.
Mid-level. Clarifies what the summary is for and what a wrong statement costs. Sees from the numbers that the largest incidents do not fit in one call and proposes a way to read them in pieces. Ties each statement to its source events. Names faithfulness and omission as the things to measure, and proposes checking against real incidents. Handles the sensitive data and the model outage at a basic level.
Senior. Does the above with the arithmetic, and separates the baseline from the extension. Makes the main decisions explicit and defends them: maintained state against regeneration, structured claims against free text, whether the model proposes causes. Designs the evaluation so that it avoids hindsight, reports by claim type, and reflects that claims within an incident are not independent. Names the failure modes of the design itself: state drift, late and edited events, a feedback loop with responders. Plans a staged release with a kill switch.
Staff and principal. This Problem is not written as a separate bar for them. An answer from them is expected to include everything above and would tend to go further on the parts that cross team boundaries: who owns the evaluation set and the incident-owner sign-off on error costs, how access rules and retention are enforced across the incident tools, and how the feature is measured for harm to response practice and not only for accuracy.
Common mistakes
Show common mistakes
- Summarising summaries. Re-summarising the previous text each cycle drops qualifiers, and the source is no longer in view to recover them. Keep claims with evidence, and keep a way to go back to the events.
- Scoring against postmortems. A postmortem knows the cause. A live summary does not, so it rewards hindsight. Replay at cut points and label against the events before the cut.
- One blended quality score. An average hides a rare unsupported cause among many correct claims. Report by claim type and read the costly failures.
- Treating claims as independent samples. They share incidents. An interval over claim counts is narrower than the evidence supports.
- Letting the model assert causes. The system then owns an error the prompt calls the costliest. Attribute causes, and label correlations as timing.
- A cursor on timestamps. Late events fall behind it and are never summarised. Use the ingest sequence.
- Ignoring edits and deletions. A deleted message that held a credential, or a retracted statement, stays in the summary. Propagate changes to the claims that cited them, and do not promise that a replay reproduces a view built on content since deleted.
- Keying a claim's identity on its evidence or its wording. Evidence keys merge two actions that cite one message, and wording keys duplicate a claim on every retry. Assign identity to the claim, and make operations carry their own stable ID.
- Advancing the cursor past a rejected operation. The summary then looks complete through an event that was never absorbed, and nothing retries it. Move the cursor only over events that are resolved.
- Sizing from the average incident. Most incidents fit in one call, and the one that does not is where the handoff matters most. Size from the tail.
- Trusting redaction as complete. Pattern detection misses things. Combine it with access rules and retention, and test what a missed credential would reach.
- Putting the model on the read path. The view then waits for a call that can be slow or fail. Read from storage, state the covered cursor and show the events since.
- Deduplicating in the store. Replacing repeats with one aggregate destroys the events that evidence, replay and edits depend on, and can blur a firing and a resolution. Group when building model input and keep the individual events.
- Showing only a count of the backlog. A count and a recent preview hide an earlier action that was never absorbed. Keep the whole unabsorbed timeline reachable.
Related Lessons
Plan for wrong, unsafe and unavailable output: layered checks, graceful degradation, where a person belongs in the loop, and how to roll back a model or prompt change.