The Constant Nobody Could Re-Measure

Most systems that call a stochastic service carry a number that converts an input into an expected cost: characters to seconds, tokens to latency, pages to euros. Mine was seconds per letter of narration, used to predict how long a script would take to read before any audio existed. It had been fitted once, by hand, on six samples from a single voice. The comment beside it said to re-measure it when the voice or the model changed. Nobody ever did, over years, and the reason was not laziness. There was no data set to re-measure it against.

The strongest argument against everything that follows is that this is fine. A constant is not automatically a bad model. Replacing one with a regression converts a line of arithmetic into an artifact that has to be stored, versioned, validated and explained when it drifts. If the constant is close enough for the decision it feeds, the honest move is to leave it alone and write a better comment.

So the first move is not to build a model. It is to refit the constant properly and find out whether the model class was ever the problem. On the synthetic corpus below, the constant fitted on six samples of one voice gives 16.82 percent mean absolute percentage error, and the same one parameter model refitted on all 120 sentences gives 9.52 percent. Of the 7.93 points separating the inherited number from the best pooled model in this article, refitting alone closes 7.30. Almost all of it was sample size.

Which is why the regression is not the interesting part of this story. The interesting parts are where the calibration data came from, and the floor underneath every number in it.

Nobody re-measured it because measurement had no input

A number gets re-measured when re-measuring is cheap and somebody owns the cost of it being wrong. Neither held. Re-measuring meant synthesising a corpus of speech, timing it, and building a fitting harness, all before knowing whether the answer would move, and the gate that consumed the number was loose enough that being wrong never surfaced as a bug. It surfaced as occasional videos that ran long, which somebody fixed by editing the script.

That is the usual shape of a stale constant. It is not defended, it is unreachable. The question that unlocked it was not how to measure speech rate. It was which existing artifact already contains the measurement.

The calibration data was already being bought and thrown away

Every successful run of that pipeline paid a forced aligner for per word timings, so captions could be drawn on the right frames, and discarded them once the captions were burned in. Separately, a tracing layer kept for debugging captured whole HTTP response bodies, so old runs on disk still held the aligner output verbatim. That is a calibration set: sentence text on one side, measured duration on the other, hundreds of rows, already paid for, needing no network call and no new synthesis.

The pattern matters more than the instance. A system that calls a stochastic component and then measures what came back is already generating labelled data for a model of that component. Response headers carrying token counts, a queue’s recorded job durations, an OCR engine’s per page confidence: all logged, used once, dropped. Look at what your logs already hold before building anything to collect more.

The smallest version that shows the mechanism

The private corpus is not publishable and its numbers are not checkable by you, so everything numeric in this article comes from a synthetic generator I can hand you. It invents six voices across two categories, each voice reading at its own rate in letters per second, with a fixed pause charged per comma and a longer one per colon, and a lognormal jitter of 3.5 percent standard deviation applied independently to each render. Every sentence is rendered three times.

true_mean = (
    letters / rate
    + COMMA_PAUSE_S * commas
    + COLON_PAUSE_S * colons
    + spec["lead_in_s"]
)
renders = [true_mean * math.exp(j) for j in jitters]

The model under test is ordinary least squares on three features and an intercept: letters, commas, and colons plus semicolons. Letters means non whitespace characters: a space costs no time to say, and the punctuation that does cost time is priced in its own column rather than smuggled into the character count. The pause is what is being measured, not the mark.

The whole reproduction is one standard library Python file, seeded at 20260919, with no network access, and it prints every number quoted here. On the 120 sentence corpus, scoring each model against one render per sentence:

Against a properly refitted constant, the three feature model buys 0.63 points. On its own that is not worth an artifact, and anyone stopping here should keep the constant. What pays is fitting the same three features per voice, which lands at 3.39 percent on held out sentences, and which only becomes possible once a calibration set exists at all.

Two other rows are worth more than the headline. Adding word count on top of letters moves MAPE from 8.89 to 8.90 percent, which is nothing: the two columns are nearly collinear and the second buys no information. And the words only row at 13.68 percent measures the price of the wrong unit under a generator that is letter driven by construction. It is not evidence that letters beat words in real speech. I believe they do, from the private fit, but that belief is not something this script can hand you.

Both metrics are there because neither is sufficient alone. MAE in seconds is what a scheduler reads, and it is scale dependent, so it cannot compare series in different units. MAPE is comparable across sentence lengths but becomes arbitrarily large near zero targets, so it is only meaningful alongside the population it was computed over.

The floor, and why a number below it is a warning

Three renders of the same sentence do not have the same duration. That is not a defect, it is what a generative component is, and it means the corpus already contains a direct measurement of the component’s own variance.

Across 360 pairs of renders of identical text, the median absolute gap is 178.1 milliseconds and the mean relative gap is 3.82 percent. It is tempting to call 3.82 percent the floor. It is not. The gap between two noisy renders has the variance of both, so for independent errors its mean absolute value is larger than the mean absolute deviation of one render from its own expectation by a factor of the square root of two. The floor a text side predictor actually faces is 3.82 divided by 1.414, which is 2.7 percent. Reporting the raw pair gap as the floor overstates it by 41 percent and quietly grants the model headroom it does not have.

That correction has a practical edge. Fitting the same three features per voice gives 2.47 percent in sample, which is below the floor. A model cannot predict a quantity more precisely than the quantity varies, so this is not skill. Fitting p parameters on n points shrinks in sample residuals by the square root of one minus p over n, here the square root of one minus four over twenty, which is 0.894. A model sitting exactly at the floor should therefore report about 2.7 times 0.894, or 2.42 percent, in sample. The observed 2.47 percent is that number. Scored on a held out half of each voice instead, the same per voice fit gives 3.39 percent, comfortably above the floor and the only one of the two figures that is a claim.

Figure 1. The pooled three feature fit at 8.89 percent still carries about six points of addressable error, because pooling cannot represent a per voice reading rate. The in sample per voice bar crosses into the floor band, which is in sample shrinkage and not a model that beat the generator’s own variance.

So the module states the floor instead of quietly claiming past it. A gate that rejects a prediction disagreeing with reality by more than two percent is rejecting noise in a fixed share of cases, forever, and no feature work will fix it.

The hold one out check that killed the better looking model

Reading rate belongs to the voice, but the metadata you actually have tends to record a category. The tempting move is to fit one model per category and use the category row for a voice you have never rendered with. It looks more specific and more principled. The check that decides it is leave one group out: hold a voice out entirely, fit on what remains, predict it from a model that has never seen it. Grouped samples break the independence assumption ordinary cross validation rests on, so the fold boundary has to be the group boundary.

Two candidates, six folds. The category row is fitted only on the remaining voices of the held out voice’s own category. The pooled row is fitted on every remaining voice.

Pooling wins on the mean and on four of six voices. The specific looking model loses because in this corpus the category label carries no information about reading rate, so restricting the fit to one category buys nothing and costs two thirds of the training rows.

That result on its own would be suspicious, since I wrote the generator. So the script runs the identical procedure a second time on a corpus where the two categories are deliberately separated by 7.0 letters per second, with every other draw held identical. There the category row wins: 13.52 percent against 22.75 percent pooled, and pooling wins only two of six. The hold one out check picks the right answer both times, which is the reason to trust it on a corpus you did not design.

Look also at the spread. The pooled column runs from 3.91 percent on voice_c to 17.67 percent on voice_d, so the mean of 10.78 describes no voice in the table. A calibration that reports only its aggregate error says almost nothing about the voice you are about to use.

The zero column that makes the fit singular

One voice in the corpus never uses a colon. Fit that voice on its own and the colon column of the design matrix is all zeros, so the normal equations are singular and the solve has nothing to say about a coefficient no row constrains. A least squares routine is defined for under determined systems and will return something: the library documentation describes rank determination as treating singular values below a cut off as zero, which is a reasonable default and also a silent one.

Silent is the problem. The fit returns coefficients, the error looks fine on that voice because it has no colons to get wrong, and the artifact ships with a number nobody can explain. The fix is cheap: drop columns that are identically zero before the solve, record which were dropped, and refuse a result whose implied reading rate falls outside a plausible range. A fit on thin data can be arithmetically valid and physically absurd, and a bound on the coefficient catches that far more cheaply than an inspection of the residuals. The reproduction does both and prints one rank event, naming the voice and the column.

Where the number is finally used

The prediction is consumed per script, not per sentence, and per sentence errors are close to independent, so they cancel: 2.47 percent per sentence becomes 0.47 percent over twelve scripts of ten sentences. The gate asking whether a sixty second video will land at sixty seconds reads the aggregate, and it is far more accurate than the per sentence figure suggests.

That cuts both ways. Quoting the aggregate as the model’s accuracy overstates what it can do for a single sentence, which is what a caption timing or a scene boundary depends on. Report both, and say which one each consumer reads.

This is the design running behind the script length gate in pipelinevideo.com. In that non synthetic version, which you cannot rerun and which I am not presenting as checkable, the shape was the same: a replacement in the region of ten percent per sentence against a constant in the mid teens, a measured repeat variability of a few percent written into the module as a stated floor, and a hold one out that rejected the per category fit in favour of pooling.

What this does not show

The corpus is synthetic and I wrote its data generating process, so this demonstrates mechanisms rather than measuring speech. Real duration is not exactly linear in letters, real pause lengths depend on context, and real render variance is probably not a clean lognormal.

The square root of two correction assumes two renders of a sentence are independent draws with equal variance. A warm cache, a shared seed or per request adaptive behaviour breaks that, and the estimated floor moves.

The hold one out comparison covers two candidates on six groups, and the hold one out spread from 2.97 to 19.56 percent says plainly that a mean over six folds is a soft number. A third candidate, partial pooling with a per voice offset shrunk toward the global fit, would very likely beat both. This article does not test it.

R squared appears in the table because it is conventional, not because it decides anything. A high value does not establish that a model fits, and the residual structure is where the answer lives.

The checklist

1. Refit the constant on everything you have before proposing a model. If it was simply fitted on six points, fix that first.
2. Search your logs and traces for labels before building anything to collect them.
3. Measure the noise floor from repeated runs of identical input, and divide the mean pair gap by the square root of two before calling it a floor.
4. Compare any in sample error against the floor times the square root of one minus p over n. An in sample number below the floor is shrinkage, not skill.
5. Validate a per category model by holding the group out, never rows, and run the procedure once on data where you know the grouping matters, to check the instrument.
6. Drop identically zero columns explicitly, log the rank event, and bound the fitted coefficient by a physically plausible range.
7. State the floor in the module and refuse to claim accuracy below it.

What I have not resolved is when to refit. The calibration is cheap because the data arrives for free, but a coefficient that moves when a vendor silently changes a model version is a drift problem, and I do not yet have a trigger that distinguishes a real shift from a run of unusual scripts. Right now the answer is a human noticing, which is the same failure mode as the comment that asked somebody to re-measure a constant and was read by nobody for years.

Sources

• NumPy, numpy.linalg.lstsq, for the solve on under, well and over determined systems, and rank by a singular value cut off.
• Hyndman and Athanasopoulos, Forecasting: Principles and Practice, section 5.8, for scale dependent errors and where percentage errors break down.
• scikit-learn, Cross validation: evaluating estimator performance, for grouped samples and what a held out group tests.
• scikit-learn, mean_absolute_percentage_error, for a relative error near zero targets.
• NIST/SEMATECH, measurement process characterization, for repeat measurements of one item differing and precision as a standard deviation.
• NIST/SEMATECH, how can I tell if a model fits my data, for residual analysis and the limits of R squared.