One Prompt, Thirty-Two Calls, or Seven

Thirty-two requirements, each one a yes or a no, each one judged against its own handful of retrieved passages. That is the shape of a compliance check, a rubric grader, a security questionnaire, and more than one “does this document satisfy the policy” feature I have worked on. The model is competent at judging one requirement. The engineering question is how many calls you make, and there are three obvious answers: one, thirty-two, or about seven.

The strongest argument against everything below is the single big prompt, and it deserves to be stated at full strength. One call is one round trip, one response to validate, one retry policy, one trace to read when something goes wrong. A cached shared preamble makes the repeated context cheap. The model sees the whole file at once, so it can notice that the evidence settling requirement 19 also settles requirement 4, which thirty-two isolated calls can never do. Modern context windows hold a 32 item checklist and its evidence without strain. If your evaluator is cheap per token and your list is short, the mega prompt is the right answer, and the simulation below agrees with you in one corner of its parameter space. I will show you that corner rather than hide it.

Everywhere else, the interesting cost of the big prompt is not tokens. It is that one failure takes the whole run with it.

The long prompt costs something a token counter cannot show

Liu and colleagues measured what happens when the information a model needs sits in the middle of a long input. In their multi-document question answering task, GPT-3.5-Turbo with a 16K window scored 73.4 percent when the answer document sat at index 0 of thirty, and 50.5 percent when it sat at index 9. With twenty documents the same model went from 75.7 percent to 54.1 percent across the same two positions. The comparison that should worry you is the baseline: handed only the document containing the answer, that model scored 88.6 percent, and handed no documents at all it scored 56.0 percent. Buried in the middle of thirty documents, a model holding the answer in its context did worse than the same model working from memory alone.

Levy, Jacoby and Goldberg pushed on the other variable. Holding the reasoning sample fixed and padding the input to different lengths, they report degradation “at much shorter input lengths than their technical maximum”. A window large enough to hold the input says nothing about whether the model reads all of it.

A 32 requirement mega prompt is a document with a long middle. The requirements in that middle are the ones that will be answered carelessly, and they are also the ones you are least likely to notice, because a plausible verdict and a careful verdict look identical in the response.

The fan-out arithmetic has nothing to do with language models

Now take the opposite design. Thirty-two calls, one per requirement, perfectly isolated, each with a short focused prompt. This is the design that looks safest and it imports a problem from distributed systems that predates all of this.

Dean and Barroso put it in one sentence in 2013. Consider a service where a server typically responds in 10ms but has a 99th percentile latency of one second: “If a user request is handled on just one such server, one user request in 100 will be slow (one second).” Then the consequence: “If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second.”

That number is pure arithmetic, and my reproduction script recomputes it as 63.40 percent from one minus 0.99 to the hundredth power. The same arithmetic for the shapes in question:

Fanning out does not create slowness. It harvests it. A one percent tail you never see on a single call becomes a better than one in four event once a run depends on thirty-two of them.

Three shapes, simulated against a fake evaluator

I wrote a small seeded simulator rather than argue from intuition. It models a call as a service time of a fixed 2.0 seconds plus 0.6 seconds for every requirement it carries, multiplied by lognormal noise with sigma 0.35, and it fails with probability 0.02. Those are declared assumptions, not measurements of anything. The pool is four workers, because the real constraint is usually a provider rate limit rather than your hardware.

The part worth reading is the scheduler, because it is where “run them concurrently” stops being free:

def wall_clock(durations: list[float], workers: int) -> float:
    """Makespan of `durations` on `workers` servers, submitted in the given order."""
    if workers >= len(durations):
        return max(durations)
    free_at = [0.0] * workers
    for d in durations:
        i = min(range(workers), key=lambda k: free_at[k])
        free_at[i] += d
    return max(free_at)

With more workers than calls, wall clock is the slowest call. With fewer, it is a makespan, which is much closer to a sum divided by the pool size. The Python documentation is blunt about the bound: a ThreadPoolExecutor “uses a pool of at most max_workers threads to execute calls asynchronously”. Thirty-two calls through four workers is not parallelism, it is a queue with four tills.

Twenty thousand seeded runs of each shape:

Figure 1. Under this cost model the grouped shape sits off the line joining the other two: fastest of the three, with a blast radius between them. The whiskers carry the second asymmetry, since a single call concentrates the whole run’s latency variance in one draw.

Two things in that table surprised me the first time I ran it. The per-requirement shape comes last on the stopwatch, because a bounded pool turns its fan-out back into a sum. And the single call has the worst tail by a wide margin, 37.43 seconds at p95 against a mean of 22.44, because all of the run’s latency variance is concentrated in one draw instead of being averaged across seven.

Expected loss is identical, and that is the point

Here is the result that changed how I think about this. The mean share of the checklist left unverified per run is 2.0 percent for all three shapes. Not approximately. Exactly, by linearity of expectation: every requirement sits in exactly one call, every call fails with the same probability, so the expected fraction lost is the per-call failure rate regardless of how you group them.

Batching does not change how much work you lose on average. It changes the distribution of that loss, and the distribution is the whole engineering decision.

The single call loses everything 2 percent of the time. The per-requirement shape loses something in 47.6 percent of runs and that something is one requirement out of thirty-two. The grouped shape loses a cluster in 13.5 percent of runs. Same expected loss, three completely different operational stories. A run that returns thirty-one verdicts and one honest “not verified” is a run a person finishes in a minute. A run that returns nothing is a refund and a support ticket.

Choose the failure you can operate, then let that choose the granularity. Performance comes second, which is the opposite of how this decision usually gets made.

The default failure mode of a naive fan-out deserves a look too. Python’s asyncio.gather documents that without return_exceptions, “the first raised exception is immediately propagated to the task that awaits on gather(). Other awaitables in the aws sequence won’t be cancelled and will continue to run.” So the obvious implementation turns one failed call into one exception at the top and a set of orphaned calls still burning tokens underneath. Isolation is something you write, not something concurrency gives you.

A grouped call will answer five of your six requirements

This is the failure that only exists because you batched, and it is quiet.

Give a model six requirements in one call and ask for six verdicts, and sometimes you get five. Not five plus an error. Five well-formed, confident verdicts in a perfectly valid response. Schema enforcement does not catch it: a contract saying the response carries a list of results is satisfied by a list of five results. OpenAI’s own documentation is careful about the boundary, promising a feature “that ensures the model will always generate responses that adhere to your supplied JSON Schema” while also stating that “in some cases, the model might not generate a valid response that matches the provided JSON schema”, with refusals and truncation as named causes. Even in the happy path, the guarantee is structural. Nothing in a JSON schema knows which six requirement identifiers you asked about.

So the grouped call needs a coverage check that the other two shapes do not. Send the identifiers, require them back, diff the returned set against the requested set, and treat the difference as a first-class outcome. The rule I hold to is that a missing requirement becomes an explicit “not verified” and never anything else. Not a pass, because nothing was verified. Not a failure, because the subject was never judged and inventing a non-conformity is worse than admitting a gap. And “not verified” then comes out of the denominator of whatever score you report, otherwise your engineering defect is being counted as the customer’s problem.

At an assumed 5 percent per-requirement omission rate, a grouped run over 32 requirements leaves 1.6 of them unanswered on average. I invented that rate and the number moves with it. What does not move is the asymmetry: a call carrying one requirement cannot omit quietly, because its omission is an empty response, and an empty response is loud. Coverage checking is part of the price of grouping, and if you are not going to pay it, do not group.

Call count is arithmetic, latency is not

Thirty-two divided by seven is 4.57. That is division. It is a true, checkable statement about the number of requests a run issues, and it is the kind of claim I will defend in a review because anyone can count the calls.

The sentence that wants to be written next is “4.5 times faster”. It is a different claim entirely and I have no right to it unless I have timed both shapes against the same evaluator on the same evidence. Call count is a property of your code. Latency is a property of a system that includes a provider’s queue, your rate limit, the token count of each prompt, and the time of day. One is arithmetic, the other is a measurement, and converting the first into the second is the most common way an honest engineering decision turns into a dishonest number.

My own simulator makes the point against me. Sweep one assumption, the marginal seconds each extra requirement adds to its call, and the ranking changes:

If a requirement adds nothing to the time of the call carrying it, the mega prompt wins the stopwatch by a factor of two and its only problem is blast radius. That single unmeasured parameter decides which shape is fastest. It does not decide which shape is safest, and that is why I would still group.

The grouped shape is what runs behind auditmalin.com. The number I state about it is the call count, because the call count is the number I can count.

What this does not show

The latency model is invented. A fixed overhead plus a linear per-requirement term times lognormal noise is a plausible shape, not a measurement of any provider, and the sensitivity table above exists precisely because the conclusion depends on it. Substitute your own numbers before quoting mine.

Failures here are independent. Real ones are not. A rate limit, a provider incident or a malformed shared preamble hits every call in the run at once, and correlated failure erases most of the isolation that grouping buys. The blast radius column describes a single independent failure, which is the easy case.

Grouping assumes the clusters are real. Seven groups of unrelated requirements is just seven smaller mega prompts with the same dilution problem and none of the shared-context benefit. The grouping has to follow something in the domain, so that the requirements in a call genuinely share evidence.

Token cost is not modelled at all. Seven calls repeat the shared instructions seven times, and evidence retrieved for one requirement may be resent inside two different groups. Depending on your cache behaviour, the grouped shape can cost more tokens than the mega prompt while issuing fewer calls. The call count went down; the bill may not have.

Retries are not modelled either. A per-call retry changes the effective failure rate and therefore every number in the loss column.

Choosing the granularity

1. Write down the failure you can operate. If losing the whole run is unacceptable, the single prompt is already excluded, whatever it costs.
2. Group by something real in the domain, so the requirements sharing a call share evidence.
3. Measure the pool you actually get. Concurrency you are not allowed to use is a queue, and a queue turns fan-out back into a sum.
4. Add a coverage check to every grouped call, and make a missing requirement an explicit unverified outcome that leaves the score denominator.
5. Isolate failures explicitly, because the default behaviour of a fan-out helper is usually to surface one exception and leave the rest running.
6. Report the call count as arithmetic, and report latency only if you timed it.

The question I have not answered is whether the grouping should be static at all. Clusters chosen by a human follow the structure of the checklist, not the structure of the evidence, and the requirements that actually share retrieved passages may cut across them. A grouping derived from retrieval overlap would batch the calls that genuinely benefit from seeing each other’s evidence and split the ones that do not. It would also make the blast radius data-dependent and different on every run, which is exactly the property that makes an incident hard to reason about. I do not know yet which of those two effects is larger.

Primary sources

• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, “Lost in the Middle: How Language Models Use Long Contexts”, with the full text and appendix tables carrying the per-position accuracies quoted above.
• Jeffrey Dean and Luiz Andre Barroso, “The Tail at Scale”, Communications of the ACM 56(2), February 2013.
• Mosh Levy, Alon Jacoby and Yoav Goldberg, “Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models”.
• Python documentation, concurrent.futures and Coroutines and Tasks, for the bounded pool and for the default exception behaviour of gather.
• OpenAI platform documentation, Structured Outputs, for what schema enforcement does and does not promise.