Design a pipeline that extracts structured attributes from millions of product listings with a language model
Plan a recurring batch job that turns free-text product listings into structured attributes under a fixed model budget.
- Difficulty
- Hard
- Level
- Advanced
Category: System design
Part of GenAI System Design / Running it in production: budget, failure and trade-offs
Topics: batch-processing · data-pipeline · extraction · cost-control
The challenge
You're joining the catalogue team of an online marketplace. Sellers create listings with a title, a free-text description and a few optional fields. Search wants to offer filters such as screen size, material or connector type, and those filters need structured attributes that most listings don't have. The catalogue team has decided to extract them from the listing text with a language model.
I'd like you to design the job that does this: a recurring pipeline that keeps structured attributes current for every active listing, within a fixed model budget.
Here is what I can tell you.
- Catalogue. About 10 million active listings across 60 categories. A listing's title, description and seller-filled fields together average about 400 tokens. About 1 percent are longer than 4,000 tokens, mostly because sellers paste specification sheets, and the longest are about 20,000.
- Starting point. No attributes have been extracted yet.
- Schemas. For each category, the catalogue team keeps a schema of 8 to 25 attributes. Each attribute has a name, a definition and a type: one value from an allowed list, a number with a unit, yes or no, or short text. The team changes schemas a few times a month, usually by adding an attribute or an allowed value, and occasionally by changing a definition.
- Seller fields. Sellers can fill some attributes themselves. Many leave them blank, and some fill them in wrongly.
- Change rate. On a typical day about 250,000 listings are created or edited. On the busiest days of the year, around seasonal sales, it reaches about 1 million.
- Freshness. On a typical day, a new or edited listing should have its attributes within 12 hours.
- Model. One hosted model is approved. Its context window is 16,000 tokens, input and output together, and it can be asked for JSON that matches a schema. The model account is shared with other teams, and this job may use at most 600,000 input tokens a minute.
- Budget. The job has a monthly allowance of 6 billion input tokens and 1.5 billion output tokens. Going over it is not acceptable, and unused allowance does not carry over. Every token counts the same: the account has no discounted batch option and no discount for repeated prompt text.
- Failures. The provider has outages of up to a few hours, a few times a month. Outside outages, about 1 request in 100 times out or returns output that does not parse.
- Consumers. Search and the catalogue team read the attributes from a table that this job writes. When a seller disputes an attribute, support needs to see what produced it.
- Review. Nobody has labelled any attributes. The catalogue team can review about 400 listings a week.
Image analysis, search ranking, seller-facing screens and training or fine-tuning a model are out of scope. The marketplace and every figure above are hypothetical givens for this exercise.
Walk me through the design, and tell me how you would decide that an attribute is good enough to offer as a search filter.
Before you start
Prerequisites
Knowledge applied
- Chunking long inputs for modelsKnowledge
Some material for this Problem is included with full access.
Interview
Step 1 of 6
Frame
Understand the situation before you solve it: clarify what is asked, find out what you need to know, state what you assume.
Before reading on, list the questions you would ask the interviewer and the assumptions you would state.
Show how the reference frames the problem
Scope clarification
What the reference treats as the core. A table holding, for every active listing, a value for each attribute in its category's schema or an explicit "not stated", where every value is supported by the listing's own text, kept current as listings change and paid for within the allowance. Three questions shape most of the decisions: how much model work each listing needs, what a wrong value costs once search uses it, and what the job does when the model is unavailable or the work exceeds what the limits allow.
What it leaves out. The out-of-scope items in the prompt. Images matter most among them: an attribute such as colour is often clearer in a photo than in the text, and the reference accepts lower coverage for those attributes rather than designing image analysis. How schemas are decided is also outside the design. The job treats a category's schema as an input with a version.
Assumed for this reference. The givens in the prompt stay as given. The statements below are assumptions the reference adds, each used where it matters.
| Assumption | Where it matters |
|---|---|
| Each listing has a stable ID, and the marketplace emits an event when a listing is created, edited or removed, carrying the listing's category. | Picking up changes |
| Many edits change only price, stock or photos, which this job does not read. | Skipping edits that change nothing the model sees |
| A category's schema carries a version number, and every change to it creates a new version. | Recording what produced a value |
| The marketplace gives each listing a version number that increases with every edit and is carried on its events. | Keeping an older answer from replacing a newer one |
| The instructions and one category's schema take about 1,500 tokens in a request. | Cost per listing, the choice of work unit |
| The model's answer for one listing averages about 150 tokens when it returns only the attributes it found. | Output cost |
| A request can set a maximum output length, and the provider stops the answer there. | Bounding what one request can spend |
| Search reads the attribute table when it refreshes a listing, so a value this job writes reaches shoppers. | The cost of a wrong value |
| The listing text is the standard of truth. Whether the physical product matches its description is the seller's responsibility and outside this job. | Labelling and evaluation |
The two token figures are estimates, not measurements. The baseline's first job is to measure them, and every figure derived from them is recomputed when it does.
The last assumption in the table is worth saying aloud in the interview. Without it, nobody can label a value as right or wrong, because the reviewer cannot see the product.
Step 2 of 6
Requirements
Say what the system must do and how well, and name the constraints that will shape the design.
Show reference for Requirements
Functional requirements
- For every active listing, store a value or "not stated" for each attribute in its category's schema, extracted from the title, description and seller-filled fields.
- Run a first pass over the whole catalogue, which starts with no attributes.
- Pick up created and edited listings, and skip edits that change none of the text the job reads. Mark the attributes of a removed listing as inactive.
- Store only values that fit the schema: an allowed value, a number in the attribute's unit, yes or no, or short text within a length limit.
- Record with each value what produced it, so that support can retrieve the exact input and the model's answer behind a disputed value, including after the listing has changed or been removed, for as long as the retention policy keeps that evidence.
- Extract again where needed when a schema, the prompt or the model changes.
- Stay within the monthly allowance and the rate share, whatever happens to the volume of changes.
- Keep going through provider failures. No listing is lost, and no request is retried without a limit.
- Never let an answer replace values that came from newer listing text, or from an extraction configuration (schema, prompt and model) that has since replaced the one the answer was produced under.
Requirement 5 is the one a first design most often leaves out. Without it, support cannot explain a disputed value, and nobody can later tell which stored values a particular prompt or model produced.
Non-functional requirements
| Requirement | Source | What it drives |
|---|---|---|
| A new or edited listing has attributes within 12 hours on a typical day | Prompt | The order in which work is sent, how much of the rate share it uses |
| Spending stays within 6 billion input and 1.5 billion output tokens a month, with no overrun | Prompt | Token counting, the work unit, skipping unchanged text |
| At most 600,000 input tokens a minute | Prompt | How fast large amounts of work can go |
| A wrong value usually costs more than a missing one | Follows from search using the table (assumption above) | An explicit "not stated", checks before storing, a quality bar per attribute |
| Every value can be explained | Prompt (support) | Retained inputs and evidence per value, a retention policy |
| Failures lose no work | Prompt | A durable queue, bounded retries |
| Answers that arrive out of order never overwrite newer values | Follows from retries and from listings edited while a request is out | A version check on every write |
| Quality is measured with about 400 reviewed listings a week | Prompt | How the review sample is spent |
Why a wrong value usually costs more than a missing one. A shopper who filters on a 15-inch screen does not see a laptop whose screen size is missing. That listing loses some filtered traffic but can still be found by a text search. A laptop stored with the wrong size shows up under a filter that does not match it, and is missing from the filter that does. So when the text does not state an attribute, "not stated" is the right answer, and the model has to be allowed to give it. An attribute where a gap is costlier than an error would change this, and none is given here.
Freshness, stated so it can be measured. For each change, the clock starts at the listing event and stops when values for that listing version are committed, or when the change filter finds that nothing the model reads has changed. The prompt's commitment covers typical days: every new or edited listing within 12 hours. It is reported daily as the share of changes finished within 12 hours, the number still waiting past 12 hours and the age of the oldest. A change still waiting at 12 hours counts as a breach for as long as it waits, and it is never dropped from the measure. The prompt makes no commitment for the busiest days.
The numbers that decide the first design. The simplest shape sends one listing per request. With the assumed 1,500 tokens of instructions and schema:
one listing per request 400 + 1,500 = 1,900 input tokens
typical month 250,000 a day x 30 days = 7.5 million listings
input, one per request 7.5 million x 1,900 = 14.25 billion, over twice the 6 billion allowance
output (150 is an estimate) 7.5 million x 150 = 1.125 billion of 1.5 billion
typical day at the rate share 250,000 x 1,900 = 475 million / 600,000 a minute = about 13 hours of sending
Two things follow. Most of the input in this shape is instructions and schema repeated for every listing, so the input allowance is lost to repetition and not to listing text. And output does not depend on how listings are grouped: a typical month already uses three quarters of the output allowance whatever the request shape, so output is the limit to watch once the instructions are shared.
Step 3 of 6
Design
Sketch the core: the entities, the interfaces and a high-level design that meets the requirements.
Show reference for Design
High-level design
Baseline. Send each listing, with its category's schema and instructions, to the model, ask for the attributes as JSON, and write what comes back. It is worth building for a few thousand listings, because it produces a quality reference that later changes are measured against. At full scale it fails on the numbers: over 14 billion input tokens in a typical month against an allowance of 6, and about 13 hours of sending on a typical day against a 12-hour target. It also has two faults that the numbers do not show. It writes whatever the model returns, including values the listing never states. And it sends a listing again every time the seller changes its price.
The design that follows. Four changes: send only listings whose text changed, put several listings from one category in each request, check every value before it is stored, and record what produced each value.
Change filter. For each listing event, compute a fingerprint of exactly what the model would read: title, description, seller-filled fields and category. If it matches the fingerprint of the listing's committed values, or of a request already out for it, and those values or that request belong to the extraction configuration now in force, nothing is sent. Price, stock and photo edits stop here. A change to the schema, the prompt or the model leaves the text untouched, so it can never be detected by this filter, and it creates extraction work of its own. A removed listing has its attributes marked inactive without a model call.
Work queue. Durable, with one entry per listing. A listing edited three times while it waits is extracted once, from its latest text.
Request builder. Several listings from one category share a request, so the instructions and the schema are sent once for all of them. Each listing gets a short key in the request, and its text is marked off as data to extract from, not as instructions. Requests are packed by token count, not by listing count: the instructions, the schema and answer format, every listing's text and framing, and the room reserved for the answer must fit the 16,000-token window together, and a request is closed before the next listing would push it over. How many listings a request should hold, and what to do with a listing too long to share a request or to fit in one at all, are decisions of their own. Chunking long inputs for models covers the second.
Extraction request. The instructions say to extract only what the text states, to answer "not stated" otherwise, to choose list values only from the allowed list and to give numbers with the unit written in the text. The answer is JSON keyed by the listing keys. The request sets a maximum output length, so a malfunctioning answer cannot run on.
Checks. The model proposes values, and code decides what is stored. This is the layered checking described in Containing failure: checks, fallbacks and human review, applied to a batch job.
- The answer parses, and every listing key in the request appears exactly once, with no other keys.
- Every attribute belongs to the category's schema, every list value is in the allowed list, every number converts to the attribute's unit, and a number outside a plausible range for the attribute, which the catalogue team sets, is rejected.
- For list values and numbers, the value, a known synonym or the number itself appears in that listing's own text. This catches a value the model invented and a value it attached to the wrong listing in the same request.
- Short text values have a length limit, and their words must appear in the listing's text.
A value that fails is stored as "not stated" with a reason code, and the listing's other values are kept. The failure is counted, because a rising rate of one reason is often the first sign of a problem with the prompt or the schema.
Normalisation. Spellings and synonyms, such as brand names, map to canonical values from a table the catalogue team owns. A value with no mapping is stored as text and listed for the team.
Attribute table. One row per listing and attribute: the value or "not stated", the reason when there is one, and the extraction record that produced it. A listing's values from one extraction are written together, so a reader never sees half of one extraction mixed with half of another.
Writing results safely. Writing a listing's values together is not enough on its own. Requests overlap: a listing edited while its earlier request is still out, or one whose timed-out request was retried, can have two answers in flight, and the answer for the older text can arrive last. A write that is merely all-at-once would then replace the newer values with older ones, completely and consistently. Two things identify an answer, and the write checks both. The content it was built from: the listing ID, version and text fingerprint. The configuration it was produced under: the category's schema version, the prompt version and the model version. The text alone is not enough, because a schema or prompt change can send the same text again, and two answers for identical text then differ only in configuration. So each request records both, and the write is conditional. In one transaction the job reads the listing's state row, which holds the latest version seen in events and its fingerprint, and the content and configuration of the committed values, together with the configuration now in force for the category. It commits only if the latest fingerprint still equals the request's, the request's configuration is still the one in force, and the committed values do not already come from the same text and configuration. An edit after the request that changed only the price keeps the fingerprint, so the answer still applies and is committed. Otherwise nothing is written, and the outcome is recorded as one of:
- Stale content. The text changed after the request was built.
- Stale configuration. The configuration in force is no longer the one the request was built under.
- Duplicate. The committed values already come from this text and this configuration, for example after a retry whose first attempt succeeded after all. Nothing changes.
- Removed. The listing was removed while the request was out. Nothing is written, and its attributes stay inactive.
After either kind of stale outcome, the job checks that work for the latest text under the configuration in force is committed, in flight or queued, and queues it if it is none of these, so the newest work is never lost. Retries follow the same rule, because each attempt carries the content and configuration of the request it repeats. How configurations are ordered, and what a configuration change extracts again, belong to the open decision on changes below.
Evidence for a disputed value. A fingerprint shows that the text changed, not what it said, and the marketplace's retention of old listing text is not given. So the job keeps its own record of every extraction it commits: a snapshot of the exact listing text it sent, stored once per fingerprint; the listing ID and version; the category and schema version; the prompt and model versions; the request ID and attempt number, with the IDs and versions of the other listings in the same request; the model's answer for this listing as returned; the normalisation applied to each value; each value's check result and reason; and the write outcome. A support or catalogue reviewer looks a value up by listing and attribute and sees the record that produced it, even after the listing has changed or been removed. Full raw answers of requests that failed or were parked are kept apart, for a short debugging window.
How long to keep these records is a policy decision for the marketplace, and its actual dispute and data-retention policy is not given. The recommendation: keep the record behind every value currently served and the one before it; keep superseded records for a fixed window that covers the dispute period, for example 180 days; hold any record under an open dispute until the dispute closes; and run the same window from the day a listing is removed. A legal deletion request overrides all of these. Listing text can contain personal details that sellers typed, so access is limited to support and catalogue roles, one listing at a time, and every access is logged. Storage does not drive the window: at an assumed 2 kilobytes per listing version, 7.5 million changes a month add about 15 gigabytes a month before compression.
Failures. A request that times out or returns output that does not parse is retried a bounded number of times with backoff. A listing that still fails is parked with its error, visible to the catalogue team, and keeps whatever values it had before.
Token meter. Every request's input and output token counts are recorded against the month.
What the baseline does not settle. Four decisions are open at this point, and a candidate should name them rather than let the diagram imply an answer:
- How many listings go in one request, and how long listings are handled.
- In what order the queue releases work when more is waiting than the limits allow, including the first pass over 10 million listings.
- How the job makes sure it stays within the allowance, as opposed to recording what it has spent.
- What is extracted again when a schema, the prompt or the model changes, and which values readers see while that work is in progress.
Step 4 of 6
Evaluate & Scale
Show how you would know it works and how it grows: evaluation, integration and production scale.
PremiumSome material in this stage is included with full access.
Hints nudge your thinking without giving the answer. Use as few as you need.
Show a hint, hint 1 for Evaluate & Scale
Hint 1 of 1
Two limits in the prompt decide how fast this job can go: its share of the rate limit and the monthly allowance. Work out which of the two runs out first on a typical day, on the busiest day and for one pass over every listing. Then decide what the job should do with work that is ready but cannot be sent yet, and what readers of the attribute table see while it waits.
No more hints for Evaluate & Scale. Open the reference when you're ready.
Show reference for Evaluate & Scale
Evaluation
What is measured, and against what. The standard is what the listing text states. For each attribute, three numbers:
- Precision of stated values. Of the values stored, the share the text supports.
- Unsupported values. Values stored for listings whose text does not state the attribute at all. These are the invented values, the error that hurts filters most, and they are counted separately even though precision already includes them.
- Coverage. Of the listings whose text states the attribute, the share that received the right value. A wrong "not stated" lowers coverage.
Report them per attribute and per category. A catalogue-wide average is dominated by easy attributes such as brand and hides the attribute that is wrong half the time. Evaluating output you cannot unit-test covers why the average misleads and how to build the set.
Where the labels come from. No labels exist. Seller-filled fields look like free labels, but they are filled only by sellers who chose to, some are wrong, and they are part of what the model reads, so agreement with them is partly circular. They are useful for finding listings to review, where the model and the seller disagree, and not for scoring. The labels come from the catalogue team: a reviewer reads the listing text, marks each stored value as supported, wrong or not stated in the text, and adds any value the text states that was missed. A written guide with examples per attribute type keeps reviewers consistent on the hard cases, such as ranges ("fits 13 to 15 inch laptops") and units the seller got wrong. Two reviewers label an overlapping subset. An attribute on which reviewers often disagree needs a clearer definition before any extraction can be judged against it.
Spending 400 listings a week. Spread across 60 categories it is about 7 listings per category a week, which decides nothing. So the review time goes where decisions are waiting: the categories whose filters search wants first, with listings sampled within each so that long listings and listings with and without seller fields are all represented. Long listings carry more text and more room for error, so they get their own share of the sample.
When an attribute is good enough to be a filter. Search and the catalogue team agree a bar per attribute, typically a ceiling on unsupported values and a floor on coverage, because a filter that covers few listings hides most of them. The measurement has to be able to show the bar is met. With 200 labelled listings for one attribute and 196 correct, the observed precision is 98 percent, and a 95 percent Wilson interval runs from about 95.0 to 99.2 percent. That sample cannot show precision above 99 percent, so a bar of 99 percent needs more listings or a different decision. When results are pooled across attributes, count listings, not values, because one listing's values come from the same text and often fail together. Comparing two system versions on an evaluation set covers the interval and the sample sizes.
Before launch. Run the one-listing baseline and the grouped design on the same labelled listings and compare them attribute by attribute, as a paired comparison. Grouping listings is chosen for cost, and this is how the team finds out whether it costs quality.
After launch. Each week a random sample of the values written that week, rotating through the launched categories and labelled with the same guide. If an attribute's measured rate falls below its bar, search can switch that one filter off while the cause is found.
Correctness scenarios. Several properties are about the system rather than about extraction quality. Each can be constructed with synthetic listings, recorded or stubbed model answers and a simulated provider, and none of them uses the 400 weekly reviews. They are proposed scenarios a design should pass, not tests that have been run.
1. The allowance is nearly spent. Set a month's recorded output spending close to the output allowance, and queue a request whose possible output would cross it. Expected: that request is not sent; at no point does spending, counting requests in flight, attempts with unknown outcome and retries, exceed either allowance; the queued work stays queued and the listings keep their previous values; the backlog and the freshness risk are visible as counts, the oldest waiting change and the changes past 12 hours; and when allowance becomes available again, at the start of the next month or after an approved increase, the waiting work is sent in the job's priority order.
2. Failures cost more than planned. Make the simulated provider time out on several times the given 1 request in 100, and report no usage for those attempts until the next day. Expected: every attempt, retries included, is sent only if the allowance can absorb the most it could cost; an attempt whose outcome is unknown stays counted at that most until the provider's usage is known, and is not counted twice afterwards; neither allowance is crossed; typical-day changes keep their 12-hour target for as long as the remaining allowance can cover them, and lower-priority work waits first; waiting work stays visible and nothing is dropped; and the alerts, the month-end figures and the next month's catch-up all use the reconciled balances, not the plan.
3. The busiest days fall early, then late. Run a month with four of the busiest days at its start, and again with them at its end. Expected in both: the typical days keep their 12-hour target whenever the measured usage and the forecast allow it; work from the busiest days, which the prompt makes no commitment for, waits first; anything waiting is reported with its age and is carried into the next month without being dropped; and if the measured figures show that even the typical days cannot fit, the report says so as a breach instead of claiming every listing met the target.
4. Answers for different text complete out of order.
- Send a request for version A of a listing.
- Edit the listing's text to make version B, and send a request for B.
- Let B's answer arrive first, and let it commit.
- Let A's answer arrive last.
Expected: A's write is refused and recorded as stale content against version A; the stored values, and the version they record, are still B's; and because B is the latest version and is committed, no further request is queued. If a version C had arrived before A's answer and was not yet committed, in flight or queued, a request for C is queued. Two variants have the same shape: a duplicate answer for B changes nothing and is recorded as a duplicate, and a listing removed while A is out gets no values and stays inactive.
5. A newer configuration's answer arrives first.
- Send a request for version A of a listing under prompt version A.
- Put prompt version B in force for the listing's category, without editing the listing, so that the same text is queued again under B.
- Let B's answer arrive first, and let it commit.
- Let A's answer arrive last.
Expected: A's write is refused and recorded as stale configuration; the stored values are B's and record prompt version B; and B stays the configuration in force.
6. An older configuration's answer arrives first. Repeat the setup, but let A's answer arrive after B is in force and before B's answer. Expected: A's write is refused as stale configuration, so A's values are never served; until B commits, readers keep seeing the complete values the listing had before; and when B's answer arrives it commits, with nothing for the latest text lost.
7. A schema changes while a request is out. Send a request for a listing under schema version A, then put schema version B in force for its category without editing the listing. Expected: the change creates extraction work for the listing even though its text fingerprint is unchanged, so no text-only check suppresses it; A's answer, whenever it arrives, cannot replace a committed B result; and the listing ends with a complete extraction under B, covering the whole schema or only the affected attributes as the job's rule for that kind of schema change says.
8. A dispute after the listing changed. A value committed from version 3 of a listing is disputed a week after the seller edited it to version 5, or removed it. Expected: an authorised support reviewer looking up that listing and attribute retrieves the record behind the disputed value: the snapshot of version 3's text as sent, the schema, prompt and model versions, the request and attempt IDs and the other listings in that request, the model's answer for the listing, the normalisation and each value's check result. The reviewer can also see the current value and the version it comes from, and the access is logged. A dispute raised after the retention window gets an explicit answer that the evidence has expired, never another version's record.
9. Cost by request shape. Run a sample whose listings take every request shape the design uses: ordinary listings grouped by category, listings too long to share a request, and listings too long to fit in one, which go in several pieces. Include timed-out requests, answers that fail to parse and requests split into smaller ones, retries, and one listing that never succeeds and ends parked. Expected:
- every attempt's input and output, or the most it could have cost while the provider has not reported its usage, is recorded and attributed to the listings it was serving, including attempts that completed none of them;
- each listing counts once in the workload, however many attempts or pieces it took, and the parked listing still counts, with everything it spent;
- the recorded totals reconcile with the provider's report without counting anything twice;
- the per-shape costs and the forecast are recomputed from the measured figures, separately for input and output;
- the budget held back for the typical-day commitment and the budget kept for surprises stay separately visible, and neither is spent by work it is not meant for;
- a request is not sent if reserving its counted input or its maximum output would take either allowance past its cap, or would spend budget held back for work with a higher claim;
- and when the measured costs for long listings come out above the plan, the forecast shows the effect on typical-day freshness, on the work that has to wait and on the next month's catch-up.
10. A rollback to an earlier configuration.
- Extract a listing's current text under configuration A, and keep the complete, checked result.
- Put configuration B in force, and let the listing be extracted under B.
- Roll back by putting configuration A in force again.
Expected when the earlier result can be reused (same text, exactly configuration A, every required attribute present and checked, evidence still kept and no rule changed since): the listing serves A's values again with no model call and no tokens spent; the records still show where those values were produced, under A, with A's versions, answer and check results unchanged, and separately that the rollback brought them back. Expected when it cannot (no kept result, a changed rule, missing attributes or expired evidence): nothing claims reuse; the listing is queued for a fresh extraction under the configuration now in force, counted in the budget and the freshness report; and the fresh answer is served only if the text and configuration are still current when it arrives. If the listing's text changed after A's extraction, A's old result is never served for the new text.
Step 5 of 6
Depth & Trade-offs
Go deep where it matters and defend your choices: what each buys, what it costs, when you would choose differently.
PremiumSome material in this stage is included with full access.
Compare with the topics the reference goes deep on
One listing per request, or several listings from the same category in each request?
Show reference: One listing per request, or several listings from the same category in each request?
Choice. Several listings from one category per request, packed by token count, with the group size chosen by measuring cost and quality together rather than by the largest group that fits.
What it serves. The allowance and the freshness target. With n listings sharing 1,500 tokens of instructions and schema, a listing costs about 400 + 1,500 / n input tokens: 1,900 alone, 1,150 in a pair, 775 in a group of four. Each doubling halves the shared part, so most of the saving comes from the first few listings in a group. A group of four already brings a typical month from about 14.25 billion input tokens to about 5.8 billion, just inside the 6 billion allowance with almost nothing left over, which says how far grouping has to go. Grouping by category is what makes this work, because one schema then serves every listing in the request.
What it costs. Listings in a request share a fate. One answer that does not parse fails every listing in the request, and the model can attach a value to the wrong listing. One seller's text now sits next to other sellers' listings, so text written to steer the model can reach values it should not touch. The explicit listing keys and the check that each value appears in its own listing's text contain most of this, and they are extra code that one listing per request would not need. Quality may also fall as the group grows, because the model has more text to keep apart, and how much it falls is measured, not assumed.
When another choice is better. If the paired comparison on labelled listings shows larger groups losing quality in some categories, use smaller groups there and pay the extra input. If the allowance were several times larger, or the change rate much lower, one listing per request is simpler to build and to debug. If the provider discounted repeated prompt text, which this account does not, most of the argument for grouping would disappear and only the rate share would still favour it. For an attribute whose definition needs long examples, asking about that one attribute across many listings can be cheaper than repeating the examples in every request.
Step 6 of 6
Follow-ups
Expect to be pushed on your decisions, your blind spots and the depth of your understanding.
Show the follow-up questions
What might the interviewer ask once the design is on the table?
Show reference: What might the interviewer ask once the design is on the table?
Sellers learn that the attributes drive search filters, and some write descriptions to game them. What changes?
What it probes. Whether the candidate treats listing text as untrusted input to a model, and sees that a request with several sellers' listings shares risk between them.
A strong answer covers. Two different behaviours. A seller who claims attributes the product lacks, such as listing every screen size, has written that in the text, and the extraction reports it faithfully. That is a listing-quality matter for marketplace policy, though the job can flag a listing that states several values for an attribute that should have one. A seller who writes instructions aimed at the model is a different problem. Marking listing text as data, constraining the answer to the schema and checking every value against its own listing's text together limit what such text can change, mostly to that seller's own listing. Because grouped requests put sellers side by side, the check matters more here than with one listing per request. A weak answer adds a second model as a filter and stops there. Containing failure: checks, fallbacks and human review covers why a second model that reads the same text can be steered by it too, while a deterministic check cannot.
The marketplace opens to listings in a second language. What changes?
What it probes. Whether quality is measured per segment, and whether the candidate notices where language enters the design.
A strong answer covers. The schema's allowed values stay canonical, and the model maps text in the new language to them. The presence check needs synonyms in the new language, or it rejects correct values and coverage drops without any visible error. Labelling needs reviewers who read the language, and results are reported per language. Token counts per listing can differ between languages, so the cost per listing is measured again before anyone relies on the old figures. Anatomy of a model request covers why the same text can cost different numbers of tokens.
Search wants attributes within 15 minutes of a listing being created. What changes?
What it probes. Whether the candidate can price freshness in a design built on grouping.
A strong answer covers. Waiting to fill a request from one category can take a long time in a small category, so the request builder needs a time limit after which it sends a partial group. A smaller group costs more input per listing, 400 + 1,500 / n, so a group of three costs 900 tokens a listing, and the candidate should price the fast path against the allowance before promising it. The fast path can be limited to new listings, since edited listings keep their previous values. A strong answer also asks whether search can index that quickly at all, and whether a seller-filled value could serve as a provisional one until extraction finishes.
Review
After the interview: study one strong approach, then review your reasoning.
Self-review
Compare your reasoning with what interviewers look for, at the bar you are preparing for.
Open self-review
Expectations by seniority
This Problem targets senior, staff and principal candidates. It is not written as a bar for mid-level candidates, and a mid-level candidate who meets the senior expectations on it is doing very well. A candidate who takes a different defensible route can meet any of these expectations.
Senior. Sizes the work before designing it, for input and output separately, and shows that one listing per request does not fit. Chooses a work unit and defends it on cost and on what it risks. Lets the model answer "not stated" and checks every value before it is stored. Stores what produced each value in a form support can actually inspect, and keeps a late answer from overwriting a newer one. Plans evaluation per attribute against the listing text within the review capacity, with an honest view of what a few hundred labels can show. Names the open decisions about pacing, staying within the allowance and handling changes, and gives a workable answer to each.
Staff. Does the above, and works out for each scale which limit decides the design, then arranges the work around it. Plans how schema, prompt and model changes reach the table and what readers see while they do. Treats the allowance as a limit the system enforces, with behaviour agreed in advance for when it runs short, and keeps the work that carries a commitment from being crowded out by the work that does not. Keeps the content an answer was built from apart from the configuration it was built under. Agrees the quality bar for each filter with search rather than choosing it alone.
Principal. Goes further on questions that cross teams. Who owns the schemas and their versions, and who signs off each filter's bar. How the allowance is set and revised against the value the filters bring. Whether some attributes should come from sellers through structured forms instead, which removes part of the problem rather than solving it, and whether extraction should become a shared service for other uses of the catalogue.
Common mistakes
Show common mistakes
- Calling the model once per attribute. The instructions and the listing text are sent again for each attribute, the most expensive shape there is. Ask for all of a listing's attributes in one answer.
- Sizing input and forgetting output. Grouping listings shares the instructions, but each listing's answer still costs the same, and in a typical month the answers alone take three quarters of the output allowance.
- Making the model fill every attribute. A required field invites a guess, and a guessed value puts a listing under the wrong filter. Offer "not stated" and count it as a correct answer when the text is silent.
- Storing whatever the model returns. Check types, allowed values, units, ranges and presence in the listing's own text first.
- Matching a grouped answer to listings by position. If the model skips one listing, every later value moves to the wrong one. Use explicit keys and check that each appears exactly once.
- Scoring against seller-filled fields. They are optional, sometimes wrong, and part of the model's input, so agreement with them is partly circular. Use disagreement to choose listings for review.
- Extracting again on every edit. Price and stock edits do not change what the model reads. Fingerprint the text the job actually sends.
- One quality number for the catalogue. Easy attributes dominate it. Report per attribute and category, and decide filters one attribute at a time.
- Counting values instead of listings in an interval. One listing's values come from the same text and fail together, so a count of values overstates the evidence.
- Storing a value without its origin. Support then has no answer for a disputed value. A fingerprint alone is not enough either, because it cannot show what the old text said.
- Trusting an all-at-once write to keep order. It stops half-written listings, not old answers. Without a version check, a late answer for older text overwrites a newer one.
- Deduplicating on the text alone. A schema or prompt change leaves the text as it was, so a text-only check skips the work the change requires, and a late answer from the replaced configuration can win.
- Packing requests by listing count. Several individually short listings can still overflow the window together. Count every token in the request, including the room for the answer.
- Leaving long listings for later. A specification sheet of 20,000 tokens does not fit in a 16,000-token window. Decide how it is split and how conflicting pieces are merged before it arrives.
Related Lessons
Build an evaluation plan for open-ended output: what to measure offline, how to use human and model judges without fooling yourself, and what to watch after launch.