Skip to content

Lesson · Module 1 · GenAI System Design

Anatomy of a model request: tokens, context and latency

Follow one request through tokenization, the context window, prefill and decode to see which steps set cost and latency. Opens with who this Track is for, what it assumes and what it covers.

Level: FoundationalDifficulty: Easy4 chapters1 diagram

Retrieval, evaluation, serving and failure handling are all ways of managing what happens inside a single model request, so a design discussion about any of them rests on being able to trace that request and estimate what it costs in time and money. This lesson does that, following one request from text to tokens to generated output, and the rest of the Track builds on the estimates it produces.

Who the Track is for, and what it assumes#

The Track is written for engineers preparing for ML and GenAI system design interviews. The same material serves three kinds of reader, who should enter at different points:

  • Building the fundamentals. You're comfortable with APIs, queues, caches and databases, and you know what training and inference mean, but you haven't designed a system around a language model. Read Module 1 in order.
  • Interview-ready, needing structure. You've called a model API or built a prototype and want a repeatable way to reason through a design round. Start with Module 2 and use Module 1 as reference when a term is unfamiliar.
  • Senior and above. You want decision rules, failure modes and the trade-offs a senior interviewer pushes on. The decision-oriented parts of Module 2 and the design problems are aimed at you.

Assumed background. Three things, none of them deep:

  • Backend fundamentals at the level of a typical system design interview: HTTP APIs, caching, queues, and relational or key-value storage.
  • The difference between training a model and calling a trained one. You don't need to know how training works.
  • Comfort with back-of-envelope estimation: multiplying request rates, sizes and times, and saying your assumptions aloud.

Having called a hosted model API once is helpful but not required. No machine learning research background is assumed.

Scope. Text-in, text-out systems built around a language model, together with the retrieval, evaluation, serving and failure design that surrounds the model. The treatment is at design level and holds whether the model is reached through a hosted API or run on infrastructure you operate.

Out of scope. Pretraining or training a foundation model, model architecture research, image, audio and video generation, vendor tutorials, prompt-writing catalogues, tool-by-tool platform comparisons and implementation code. Provider limits and prices change often, so the lessons teach how to reason with them and point out when to look up the current figure instead of quoting one.

Outcome. By the end of the Track you should be able to take an open-ended GenAI design prompt and:

  • clarify scope and success criteria before choosing components;
  • estimate where a request's cost and latency come from;
  • decide whether and how to ground a model in data;
  • define how output quality will be judged before building;
  • choose serving levers and say what each one gives up;
  • plan how the system behaves when output is wrong, unsafe or unavailable;
  • defend a choice at your own seniority.

The outcome is reasoning skill. It isn't a prediction about any interview result, and nothing here presents one design as the single correct answer.

How the Track is arranged. Module 1 builds the mental model for one request in three steps: this lesson on mechanics, then grounding answers in your own data, then evaluating output you can't unit-test. It ends with a design problem that needs all three. Module 2 moves from one request to a running system: serving within a latency and cost budget, containing failure, and choosing and defending among prompt changes, retrieval and fine-tuning.

Following one request through a model#

The running example in this lesson and the next two is an internal help assistant. Employees ask questions about company policies and tools, and the system answers with the help of a language model. Every number below is an assumption chosen to keep the arithmetic readable. None is a measurement of a real model or provider.

Text becomes tokens#

A model doesn't read characters or words. A tokenizer splits the input into subword pieces drawn from a fixed vocabulary and maps each piece to an integer ID, and the model works on those IDs. Common words are often a single token, while rare words, identifiers and unusual strings split into several.

Three consequences follow, and each affects design:

  • Token counts define context limits and token-based billing, not word or character counts. Latency is measured in time, and it is influenced by input and output length together with serving conditions such as queueing and hardware.
  • Different models use different tokenizers, so the same text can have different token counts on different models. A count is only meaningful for the tokenizer of the model you'll deploy.
  • Text far from what the tokenizer was built for, such as source code, tables, long identifiers or languages that were less represented in its training data, often uses more tokens per character than plain English prose.

For a first estimate, a common planning assumption is that English prose comes to somewhere around three quarters of a word per token. Treat that as a heuristic to say aloud, not a property to rely on. The real ratio depends on the tokenizer and the text, and a design that works only at one ratio is fragile.

Warning

Don't size a budget in words or characters and convert at the end. Count tokens with the tokenizer of the model that will serve the traffic, using samples of your real inputs, and leave headroom for the difference.

The context window holds everything#

The context window is the maximum number of tokens a single request can hold. It has to fit all of the following at once: the instructions, any conversation history, any retrieved documents, the user's question, and the answer being generated. Many providers also set a separate cap on the length of the output.

A model API call is typically stateless. The model doesn't remember the previous call, and a chat feels continuous only because the application sends the relevant history again with each request. That makes cost grow faster than conversation length. Assume the help assistant has an 800-token instruction block, user messages of about 100 tokens and replies of about 300 tokens:

input tokens at turn k = 800 + (k - 1) * (100 + 300) + 100

turn 1:   900
turn 2: 1,300
turn 3: 1,700
turn 5: 2,500
turn 10: 4,500

input tokens summed over 10 turns = 27,000
distinct text actually written     =  4,800   (800 + 10 * 100 + 10 * 300)

Over ten turns the system processes more than five times as much input as there is new text, because the early turns are paid for again and again. Truncating or summarising old turns and reusing work already done for a repeated prefix are the usual responses, and both appear in Module 2. The point for now is that a multi-turn feature has a per-conversation cost curve, not a flat per-request cost.

Prefill, then decode#

Generation runs in two phases, and they behave differently.

During prefill, the model processes every input token. All of those tokens are known up front, so the work can be spread across the hardware in parallel. Prefill ends with the model's probability distribution over the next token. The first output token is selected from that distribution at the boundary between the phases, and every later token comes from decoding.

During decode, the model generates autoregressively: each step takes the token selected in the previous step, computes a new distribution and selects the next token. Because each step depends on the one before it, the steps for a single request can't run side by side. Servers keep the intermediate attention state of tokens already processed, usually called the key-value cache⁠, so that a decode step doesn't redo the work for the whole prompt. That state sits in accelerator memory, grows with context length and is held for every in-flight request. Long contexts and high concurrency therefore compete for the same memory. A hosted API hides this from you, and a self-hosted deployment makes it your capacity-planning problem.

One request passes through tokenization, a parallel prefill phase and a sequential decode loop

Why does long input get expensive? The original Transformer paper compares layer types, and for a standard full self-attention layer it gives a cost on the order of n squared times d operations, where n is the sequence length and d is the representation dimension (Vaswani and others, 2017, Table 1). That result describes the attention component only. A model also spends compute in other layers, such as the feed-forward layers, which the same paper applies to each position separately, so their work grows roughly in proportion to the number of tokens. Total prefill compute and latency therefore depend on the model's architecture, the attention variant, the implementation and the hardware. It helps to separate what an attention optimization changes. A variant in which each token attends to only a subset of positions can lower the asymptotic compute, but it changes what the model computes. An exact-attention implementation that reorganises the same computation to use less memory or move less data improves memory use, memory traffic, constants or wall-clock speed, and it still performs the quadratic number of attention score computations. Whether either shows up in end-to-end prefill latency depends on the architecture, the implementation, the hardware and the workload, so doubling the input does not have a universal effect on latency.

What carries over to a design discussion is narrower. With full self-attention, the attention share of the work grows fastest as inputs lengthen, and the cached state described above grows with context length. The practical move is to measure prefill time at several input lengths on the model you'll use, and to avoid quoting a quadratic latency curve you haven't measured. Input length is a design variable, not a free parameter.

Where the time goes#

A usable first-order model of one request's latency has three terms:

latency ~ queueing + prefill(input tokens) + output tokens * average time per output token

The last term is a first-order approximation that applies an average cost to every output token. Strictly, the first output token is selected at the end of prefill, so time to first token is queueing plus prefill, and a request with N output tokens runs N - 1 decode steps. For answers of tens or hundreds of tokens that distinction is small enough to ignore in a first estimate, and the figures below use the simpler form.

Using invented figures for the 2,500-token turn above, a 250-token answer, 50 ms of queueing, 400 ms of prefill and an average of 30 ms per output token:

50 + 400 + 250 * 30 = 7,950 ms         (time to first token about 450 ms)

halve the input  (prefill 200 ms):   50 + 200 + 7,500 = 7,750 ms   about 2.5% faster
halve the output (125 tokens):       50 + 400 + 3,750 = 4,200 ms   about 47% faster

For a chat-style answer, decode dominates, so shortening the answer helps far more than trimming the prompt. Streaming leaves the total unchanged but lets the user start reading after roughly the first-token time, which changes how slow the system feels. The balance flips for a task such as classification, where a 6,000-token document goes in and a few tokens come out. With 50 ms of queueing, 1,000 ms of prefill and 5 output tokens at an average of 30 ms, the request takes 1,200 ms, and nearly all of it is prefill. Which phase dominates depends on the workload, so it has to be asked about before it can be optimized.

Where the cost goes#

Hosted providers commonly price input and output tokens separately, often at different rates, so cost per request is roughly input tokens times the input rate plus output tokens times the output rate. Look up the current rates when it matters, and use symbolic rates in an interview. Some models also generate intermediate reasoning tokens that are billed and counted like output even if the user never sees them, so check how a particular model reports them. For a self-hosted model the unit changes to accelerator time, and cost per request depends on how well the hardware is kept busy.

Limits and failure modes#

  • Over-budget requests. If instructions, history, retrieved text and the reserved output together exceed the window, the call fails or something gets cut, depending on the API and on application code. A naive "drop the oldest messages" rule can remove the instructions along with them. Decide explicitly what is protected and what is trimmed first.
  • Truncated output. Generation can stop because it hit the output cap, not because the answer was finished. Providers typically report why generation ended, and the application should check it, because a cut-off structured response can look valid at a glance.
  • Fitting is not using. A fact that fits in the window may still be used poorly. Across the tasks and models the authors tested in 2023, answer accuracy was generally highest when the relevant information was at the beginning or end of the input and dropped when it sat in the middle, including for models built for long contexts (Liu and others, 2023). Newer models may behave differently, so read this as a reason to test with your own inputs and not as a law.
  • Variable output. At each step the model computes a probability distribution over the next token, and a separate decoding strategy picks one. Some strategies sample from the distribution, and a setting such as temperature reshapes how much randomness that sampling allows. Others select deterministically, for example always taking the most probable token, and temperature does not apply to them. When a deployment samples, the same input can produce different answers, which matters for how output is judged in the evaluation lesson.
  • Estimates inherit their assumptions. The arithmetic above is only as good as the token counts, the answer lengths and the per-token timings behind it. Measure them on real traffic before committing to a capacity plan.

Using this in a design discussion#

When a prompt involves a model call, a short estimate usually exposes the real constraint:

input tokens   = instructions + history + retrieved context + question
output tokens  = typical answer length (plus hidden reasoning tokens, if the model produces them)
latency        ~ queueing + prefill(input) + output tokens * average time per output token   (time)
cost           ~ input tokens * input rate + output tokens * output rate                     (currency per request)
daily cost     = requests per day * cost per request                                         (currency per day)
request memory = peak concurrent requests * memory per request                               (bytes)

The last line estimates only the request-related memory contribution, mainly the cached attention state that grows with each in-flight request. It is not total model-serving memory, which also includes items such as the model weights and runtime overhead.

The questions worth putting to the interviewer follow from it: how large the input typically is and where it comes from, how long answers are, whether the latency target applies to the first token or the complete answer, whether conversations are multi-turn, and what peak concurrency looks like.

Tip

Sample wording for a candidate: "Before choosing a model, I'd pin down input size, answer length and whether the latency target is time to first token or time to full answer. For long answers, decode time dominates, so I'd look at output length and streaming first. For long inputs with short outputs, prefill dominates and the context I include matters most."

The most common place a design runs into this arithmetic is when the system adds its own knowledge to the request. Grounding answers in your own data takes that up next: retrieved text competes for the same token budget described here.

Full outline of GenAI System Design