Module 2 of 2 · Lesson 1 of 3
Lesson · Module 2 · GenAI System Design
Serving a model within a latency and cost budget
Compare the main levers: batching, caching, streaming and routing between models. Learn which lever suits which workload and what each one gives up.
Builds on
Every serving optimization is a trade: it buys speed or savings in one place and pays for them somewhere else. Batching trades waiting time for utilization. Caching trades freshness and correctness risk for skipped work. Streaming trades simplicity for a faster first impression. Routing trades operational complexity for a lower average price. The skill this lesson builds is choosing among them for a stated workload, and saying out loud what the choice costs, which is also what a senior interviewer is listening for.
It builds directly on the estimates from the first lesson. A request's latency is roughly queueing plus prefill plus output tokens times the time per output token, and its cost is roughly input tokens times the input rate plus output tokens times the output rate. Each lever below acts on a different term of those two formulas, so the first step is always to find out which term is actually large.
| Lever | Acts on | Helps most when | What it gives up |
|---|---|---|---|
| Batching | Utilization, queueing | Hardware sits partly idle, or nobody is waiting on the result | Waiting time for individual requests; memory headroom |
| Caching | Skipped prefill, or a skipped call | The same prefix or the same question recurs | Freshness, and the risk of serving a stale or wrong answer |
| Streaming | Perceived latency | A person reads a long answer as it appears | Ability to check the whole answer before showing it; more connection handling |
| Routing | Cost per request, sometimes latency | Requests differ a lot in difficulty | Extra components, a second set of quality measurements, a failure mode for each route |
Start from the workload, not the technique#
The same service can face opposite constraints. A chat assistant has a person waiting, wants the first words quickly, and sees traffic that rises and falls over the day. An overnight job that tags two million product reviews has no one waiting, wants the lowest cost per item, and can take hours. A technique that is obviously right for one is often wrong for the other, so a good answer begins with questions that separate them:
- Who or what is waiting, and does the latency target apply to the first token or to the complete answer?
- How much of each request is identical across requests (instructions, policy text, tool definitions) and how much is new?
- How long are the answers? Output length usually sets total latency, as the first lesson showed.
- How much do requests differ in difficulty? Is a short factual lookup arriving next to a multi-step question?
- What happens if an answer is stale, or slightly worse than the best model would give?
- Is this a hosted model API or hardware you operate? Some levers belong to the provider, and others are yours.
A scenario with every number assumed#
The numbers in this lesson are invented to keep the arithmetic visible. None is a measurement of a real model, provider or price. Costs are written in abstract units so that no price is implied.
An online retailer runs a returns assistant. A customer types a question, the assistant answers using the returns policy and the customer's order details. Assume:
- Each request carries a 1,500-token shared prefix (instructions, the policy text and tool definitions) followed by 200 tokens that are specific to the request (the question and the order facts). Input is 1,700 tokens.
- The answer averages 150 output tokens.
- An input token costs 1 unit and an output token costs 5 units. Providers do price the two separately, and the ratio varies, so look up the real one when it matters.
- Latency uses the first lesson's model with assumed figures: 50 ms of queueing, 400 ms of prefill for the 1,700 input tokens, and 30 ms per output token.
- The product target is first visible text within 1 second and the complete answer within 6 seconds.
cost = 1,700 * 1 + 150 * 5 = 2,450 units per request
latency = 50 + 400 + 150 * 30 = 4,950 ms (time to first token about 450 ms)
Both targets are met on average, but those are mean values, and the targets are written for the slow end. Queueing grows when traffic bursts, so the real question is how the system behaves at its busiest minute. The rest of the lesson uses this scenario to see what each lever does to those two lines.
Batching: using the hardware, and the cost in waiting#
Decoding one request at a time tends to leave an accelerator underused. In the usual regime the limit is how fast model weights and cached attention state move through memory, and a decode step reads the same weights whether it advances one sequence or many. Advancing many sequences per step therefore raises throughput per unit of hardware. That is a general explanation, and the size of the gain depends on the model, the hardware and the serving software. The vLLM paper states the consequence directly: serving at high throughput needs many requests batched together, and wasted cache memory limits how large the batch can get (Kwon and others, 2023).
How requests are grouped matters as much as whether they are. In request-level batching, a batch forms and runs until every request in it finishes. The Orca paper describes the weakness: requests that finish early cannot be returned until the batch ends, and newly arrived requests wait for the entire current batch (Yu and others, 2022). Its response, scheduling at the granularity of a single model iteration, is the idea behind what is now usually called continuous batching: finished requests leave and waiting requests join between steps. A small illustration with invented lengths shows why this matters. A batch of four requests produces 50, 60, 80 and 400 tokens at 30 ms per token:
batch runs for 400 steps * 30 ms = 12 s
the request that needed 50 tokens could have finished in 1.5 s
useful token slots = 50 + 60 + 80 + 400 = 590 of 4 * 400 = 1,600 (about 37%)
With iteration-level scheduling, the short requests free their slots early and new ones take them, so the same hardware does more useful work and short requests return sooner.
What batching still costs is waiting and memory. If a server holds requests back to fill a batch, that hold is added directly to queueing time, so the maximum wait must come out of the latency budget and not out of the throughput target. And batch size is capped by memory. Every in-flight request holds cached attention state that grows with its context, so a workload with long contexts supports fewer concurrent requests, and a larger batch on paper may not fit.
Who controls this depends on how the model is served:
- Hosted API. The provider batches across its customers. Your levers are concurrency limits, request shaping and, for work that is not urgent, asynchronous batch interfaces. Two providers' documentation, checked in October 2026, describes one. OpenAI's Batch API lists a 50 percent discount against its synchronous APIs and a 24-hour completion window. That window is a deadline (batches that miss it expire), not a typical completion time; the page says batches often finish sooner and gives no typical duration. Anthropic's Message Batches API likewise processes requests asynchronously, and its pricing page lists a 50 percent discount on input and output tokens. Discounts, windows and supported models change, so read the current terms before relying on them.
- Self-hosted. Batching policy is yours: use a serving engine with iteration-level scheduling, cap the queue wait using the latency budget, and size the batch against accelerator memory at your real context lengths.
For the retailer's chat traffic, the right batching answer is mostly "let the serving layer do it, and protect the tail": a queue with a hard deadline, and a rule for what happens to requests that would exceed it. For the overnight review-tagging job it is the opposite. No one waits, so use an asynchronous batch interface or a large offline batch and accept the delay for the saving.
Caching: three different things under one name#
"Add a cache" hides three designs with very different risk. The right question for each is what could make a stored result wrong.
Prefix caching: reusing work, not replaying an answer#
Providers and serving engines offer prefix caching. Both providers whose documentation was checked for this lesson (October 2026) describe the same conditions: the prefix must match exactly, the cache lives for a limited time (minutes by default for some, longer options for others), very short prompts are not cached, and cached input tokens are billed at a fraction of the normal rate. Prefix caching reuses the processed state of a matching prompt prefix, which reduces input-processing work and cost on cache hits. It does not make the model deterministic or guarantee identical responses across requests: the model still runs and generates a fresh answer each time. It does not remove the processing of new input either, since everything after the matching prefix, such as the customer's question, is processed in full. And it does not guarantee a hit: a request benefits only if its prefix matches and the entry is still live. The benefit is reused computation, not a promise that every generated answer will be the same. Because it stores no earlier answer and replays none, and because the match is exact, it carries much less correctness risk than the two designs below. Whether a given request is faster or cheaper depends on whether it hits and on the provider's serving and billing behavior.
Billing terms differ by provider, model and cache duration, so the example below needs a stated basis. Anthropic's pricing page lists, per model, cache writes at 1.25 times the base input-token rate for the 5-minute duration and 2 times for the 1-hour duration, and cache reads at 0.1 times the base rate for most listed models, with lower multipliers for some newer ones. OpenAI's prompt caching guide documents a 1.25 times write rate and a 0.1 times read rate for some newer models, no extra write charge for earlier ones, and retention options that vary by model. Do not carry one provider's or one model's multiplier over to another. The scenario in this lesson is hypothetical and uses one such configuration: reads at 0.1 times and writes at 1.25 times the base input-token rate.
Because the match is exact and ordered, prompt layout becomes a performance decision:
- Put stable content first (instructions, policy, tool definitions) and per-request content last.
- Keep volatile values out of the prefix. A timestamp or request ID near the top defeats the cache for everything after it. Changing a tool definition or the system prompt invalidates what follows.
- Expect misses on quiet traffic, because an entry that expires between requests is never reused.
- Keep conversation history append-only where you can. History is billed again on every turn, so truncating or summarising old turns has a place, but rewriting earlier turns on every request changes the prefix and forfeits the saving. Compact the history in occasional, deliberate steps instead of continuously.
Apply it to the scenario. Assume a cache read is billed at 0.1 times the base input-token rate, a cache write (the miss that creates the entry) is billed at 1.25 times the base input-token rate, and 80 percent of requests hit. Cache writes are billed at 1.25 times the base input-token rate: that is the total rate for those tokens, not an additional 1.25 times charged on top of the base rate:
hit : 1,500 * 0.1 = 150 units
miss : 1,500 * 1.25 = 1,875 units (the whole prefix is billed at the write rate)
prefix cost = 0.8 * 150 + 0.2 * 1,875 = 120 + 375 = 495
cost = 495 + 200 + 150 * 5 = 1,445 units (41% below 2,450)
Under these assumed rates a miss costs more than sending the prefix uncached (1,875 against 1,500), so caching lowers the prefix cost only when the hit rate is high enough: the break-even is about 22 percent (375 / 1,725). At a low hit rate, prefix caching can cost more, not less.
The effect on latency is smaller than the effect on cost. If prefill time were proportional to the uncached tokens, an optimistic simplification, prefill would fall from 400 ms to about 47 ms, first-token time would drop to roughly 100 ms, and the total would fall only from 4,950 ms to about 4,600 ms, because 4,500 ms of it is decoding. After this change, output tokens are 750 of the 1,445 cost units, so the next saving has to come from output, not from input.
Response caching: skipping the call#
An application can store a finished answer and return it for a matching request without calling the model at all. This saves everything, which is exactly why it is risky. A stored answer is correct only while everything that produced it is unchanged, so the cache key must include every input that could change the answer: the normalized question, the model and prompt version, the sampling settings, the version of the documents it was grounded in, and the scope of the user (tenant, role, permissions). A key missing the last item is how one customer's answer reaches another.
Safe candidates are questions with the same answer for everyone and a slow rate of change, such as a general return window. Unsafe candidates include anything personalised, permission-scoped or time-sensitive (an order's status), and anything where users expect variation. Staleness needs an explicit policy: a time to live, a version in the key so a prompt or policy change makes old entries unreachable, and an event that purges entries when a source document changes. A single popular entry that expires can also trigger many identical concurrent calls, so coalescing duplicate misses is worth designing in.
Semantic caching: skipping the call for similar questions#
A semantic cache matches on embedding similarity instead of equality, so "can I return a worn jacket" can hit an entry stored for "are used clothes returnable". It raises the hit rate, and it adds a failure that exact matching cannot have: two questions that read as near neighbours but have different answers, such as a 30-day window and a 60-day window, can match. The wrong answer is returned fluently and nothing flags it. Treat it as a heuristic with a measured error rate: set the similarity threshold using labelled pairs that include deliberate near misses, restrict it to low-stakes question families, and remember that the embedding lookup is paid by every request, hit or miss. It pays off only when the saving on hits, times the hit rate, exceeds the lookup cost on all requests, plus whatever the occasional wrong answer costs.
Streaming: faster to start, not less work#
Streaming sends tokens to the client as they are generated. For a person reading, this changes what they experience from "wait 5 seconds, then see everything" to "see the first words after about half a second." The total generation time, the number of output tokens and the cost do not change. In the scenario, streaming moves the user's wait from 4,950 ms to about 450 ms for the first text while the last token still arrives at about 4,950 ms.
That makes it a perception tool, and it has uses and limits:
- It helps when a person reads a long answer. It does nothing for a program that needs the complete result before it can continue, or for a short output such as a label.
- The system cannot validate the whole answer before the user sees the first part. A check that needs the complete output either blocks the stream or runs on chunks, and a problem found late means retracting text the user has already read. The next lesson on containing failure comes back to this.
- Structured output such as JSON is not usable until it is complete, so streaming gains little there.
- Errors can arrive mid-stream. Decide what the client shows for a partial answer, and check how your provider bills a request that is cancelled partway.
- Proxies and gateways that buffer responses silently remove the benefit, so verify first-token time at the client, not just at the server.
Routing: not every request needs the strongest model#
Request difficulty varies. A simple lookup does not need the model that handles a multi-step policy exception. Routing sends easy requests to a cheaper, faster model and hard ones to a stronger one. There are two shapes, and they fail differently:
- Router first. A classifier or rule set predicts difficulty or topic before any model call and picks a model. It adds a small step to every request, and its mistakes are silent: a hard request sent to the small model gets a worse answer with no signal.
- Cascade. Try the cheap model, judge the answer with a confidence or verification step, and escalate if it fails. The FrugalGPT paper names this "LLM cascade" alongside prompt adaptation and LLM approximation, and reports that on the benchmarks it used a cascade could match the best single model's quality at a much lower cost (Chen, Zaharia and Zou, 2023). Those are results on those benchmarks and prices, and they are no forecast for your workload. In a cascade, the escalated requests pay for both attempts and wait for both.
Cascade economics are easy to estimate. Suppose the small model's request costs one fifth of the large model's (about 289 units against 1,445) and 70 percent of answers pass the acceptance check. This hypothetical calculation leaves out the cost of the acceptance check itself and treats it as negligible. If the check is a model call or a verifier, add its cost to every request:
expected cost = small + (1 - accepted) * large = 289 + 0.3 * 1,445 = 722.5 (50% of 1,445)
The arithmetic hides the quality cost. Accepted answers are only as good as the acceptance check, and a check that passes weak answers sends them to users. Escalated requests take longer than the baseline, which hurts exactly the hard cases. The operational cost is real too: two prompts to tune, quality to measure separately for each route (a single blended score hides a bad route), a fallback for when either model is unavailable, separate rate limits, and a split prefix cache because each model keeps its own. Routing also drifts: when traffic changes, the router's thresholds need re-checking against fresh labelled examples.
Because of that overhead, routing is the lever to reach for last. It tends to earn its place when the volume is high, the quality gap between models on your easy requests is demonstrably small, and the saving is large enough to cover the engineering and the evaluation work.
Choosing, and when not to#
Diagnose the term that dominates before reaching for a technique:
| What the measurements show | Try first | Why |
|---|---|---|
| A long shared prefix and slow first token or high input cost | Prefix caching and prompt layout | Large input saving on cache hits, with no stored answer to go stale |
| Answers are long and total time is the problem | Shorter answers, streaming for perception, then a faster model | Decode dominates, and only output length or per-token speed changes it |
| Self-hosted accelerators are underused at the busy hour | Continuous batching, tuned queue limits | Utilization is the bottleneck |
| Nobody is waiting on the result | An asynchronous batch interface | Trade time you do not need for a lower price |
| Many identical questions with context-free answers | Exact response caching with a versioned key | Skips the call entirely |
| Output cost dominates and difficulty varies widely | Routing, after the cheaper levers | Largest remaining saving, largest added complexity |
Three honest cases favour doing less. At low volume, the saving from a lever may not cover the engineering and the quality monitoring it brings. When the latency target is already met with room to spare at the busiest hour, optimizing further adds risk for no benefit. And when answers are consequential, such as legal or financial, a stale or misrouted answer can cost more than the compute saved, so response and semantic caches and aggressive routing stay out of those paths.
Whatever you pick, measure each lever on its own, because the combined effect hides which one is working: cache hit rate and the quality of cache hits, queue wait at the 95th and 99th percentiles, time to first token at the client, the share of requests escalated, and quality broken down by route. Also decide ahead of time what the system does when load exceeds capacity, whether that means shedding requests, shortening answers or sending overflow to a smaller model. Making that choice before an incident is cheaper than making it during one.
Quoting a speed-up or a saving without its assumptions is the commonest way to lose credibility on this topic. Every figure in this lesson depends on prefix length, answer length, hit rate or acceptance rate. State those inputs, and say how you would measure them on real traffic.
Sample wording for a candidate: "I'd first split time and cost into queueing, prefill and decode, then look at which dominates. Here the prefix is long and shared, so I'd lay the prompt out for prefix caching and watch the hit rate. I'd stream to protect time to first token, but I wouldn't claim it reduces total time. I'd only add model routing once cost per request is mostly output tokens and I have labelled examples to measure quality by route."
Caches, routers and fast paths all make the system quicker or cheaper and none of them makes the model right. The next lesson turns to the other half of running it in production: what the system does when the output is wrong, unsafe, malformed or unavailable.
Related Knowledge
Practice
Problem
Design a pipeline that extracts structured attributes from millions of product listings with a language modelPlan a recurring batch job that turns free-text product listings into structured attributes under a fixed model budget.
HardBuilds on this lesson
Related Lessons
A decision lesson for senior and staff-level discussions: which kind of change should carry a quality improvement, what evidence the next step needs, and how to defend the choice under questioning.