A Detector With Perfect Precision and Zero Recall

A document authenticity check I had built cleared every file in a small set of genuine documents. Zero false positives, on two runs two days apart. Then I pointed it at a public labelled benchmark where most of the documents carried a forgery, and it cleared those too. Recall 0.0. Not one true positive in the whole set.

The lesson people draw from that story is “ask for recall”. It is the lesson I trust least, because recall is the cheapest number in the confusion matrix to buy. Drop the threshold far enough and recall goes to 1.0 on any labelled set you like, while the review queue fills with noise until the team stops opening it. Anyone who has staffed a fraud queue knows the detector that gets switched off is not the one that misses things quietly, it is the one that cries wolf on a Tuesday afternoon. Optimizing for recall is how you get there.

So the counterargument is real, and it is about where to sit on a curve. My failure was not a point on a curve. There was no curve. No threshold anywhere in that detector’s range would have produced a single true positive on those inputs, because the detector was not measuring the quantity the labels described. Precision and recall only trade against each other once there is signal to trade.

The smallest version of the failure

I cannot share the private run, so here is a synthetic one that has the same shape and that you can execute. A page is a flat array of 2000 samples, standing in for pixels. A genuine page is noise around zero. A tampered page has a contiguous patch repainted: a small block shifted by a constant. Across the 178 tampered pages in the run, the repainted patch averages 1.85 percent of the page and never exceeds 2.90 percent.

Three cohorts:

• G, 18 genuine documents and no positives at all. This is the in-house smoke set, the thing you run before a demo.
• P, 200 photographed pages, 178 of them repainted. Photographing a page destroys the container: there is no revision history, no producer string, no trailer to count. The pixels survive, the file structure does not.
• S, 200 born-digital files, 178 of which went through a second save path. The container changed. The pixels are byte identical to the originals.

Three detectors:

• never_fires returns “genuine” for everything. It is the null baseline, and it exists to be compared against.
• structure_level reads the container: revision count, producer consistency, trailer count.
• signal_level reads the samples: the largest standardized sliding window mean deviation on the page, against a scale estimated from the page’s own first differences.

The signal detector’s threshold is calibrated on 500 held-out genuine pages to a 5 percent false positive budget, which puts it at z greater than 4.007. It is calibrated once, on genuine data only, and then never touched.

The scoring is the boring part, which is exactly why it is worth writing out:

def rates(tp, fp, fn, tn):
    return {
        "precision":   None if tp + fp == 0 else tp / (tp + fp),
        "recall":      None if tp + fn == 0 else tp / (tp + fn),
        "specificity": None if tn + fp == 0 else tn / (tn + fp),
        "accuracy":    (tp + tn) / (tp + fp + fn + tn),
    }

None rather than a number, in all three places where the denominator can vanish. Hold on to that; it is where the title comes from.

The frozen run

Seed 20260919, Python 3.12.3, standard library only, no network. Two consecutive runs produce identical stdout.

Row five is the private incident, rebuilt from scratch. The structure detector fired zero times on 200 documents, 178 of which were tampered.

The arithmetic that makes zero recall look like a pass

Take the structure detector on cohort P and write the three rates out longhand.

specificity = TN / (TN + FP) = 22 / (22 + 0) = 1.000
recall      = TP / (TP + FN) =  0 / (0 + 178) = 0.000
precision   = TP / (TP + FP) =  0 / (0 +   0) = 0 / 0

Specificity is perfect, and it is perfect for a reason that has nothing to do with skill: a detector that never fires cannot produce a false positive, so FP is structurally zero and specificity is structurally 1.000. The same is true of its accuracy on cohort G, where the absence of positives means accuracy 1.000 is arithmetically forced.

Precision is the interesting one. It is 0 divided by 0, and the honest return value is “undefined”. Almost nothing returns that. The scikit-learn reference is explicit about what it does instead: “When true positive + false positive == 0, precision returns 0 and raises UndefinedMetricWarning. This behavior can be modified with zero_division.” The parameter accepts 0.0, 1.0 or NaN. So the same dead detector, on the same run, reports precision 0.000 or precision 1.000 depending on one keyword argument nobody reviews. A dashboard that was built with zero_division=1 to stop a warning from cluttering the logs will show you a detector with perfect precision and perfect specificity that has never detected anything.

The model evaluation guide says the same thing about F1 a paragraph later: “Note that this formula is still undefined when there are no true positives, false positives, or false negatives. By default, F-1 for a set of exclusively true negatives is calculated as 0, however this behavior can be changed using the zero_division parameter.” A set of exclusively true negatives is what a genuine-only smoke test is.

What the genuine set could not see

Figure 1. The 18-document genuine set assigns an identical perfect score to a working detector and to one that returns “genuine” unconditionally. The labelled set separates them on the first number it reports.

The genuine cohort is not a sloppy test. It is a test with zero discriminating power against this failure, by construction, because it contains no positives and every metric computable on it is a function of FP and TN alone. You can run it on more documents, you can run it twice two days apart, and it will keep agreeing with itself. Two runs of a measurement that cannot fail are not corroboration.

Saito and Rehmsmeier put the general version of this well in their 2015 comparison of precision-recall and ROC plots: the visual interpretability of ROC plots on imbalanced data “can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity”. Specificity is the metric people read as “it does not make mistakes”. It is the metric a dead detector maximizes.

The detector was not broken, it was answering a different question

The second half of the run is the part I did not expect to find so clean. Cohort S holds files whose containers were rewritten and whose pixels were left alone. Against that label, the structure detector scores recall 1.000, all 178 of 178, with precision 1.000. The signal detector scores recall 0.034, six of 178.

Figure 2. Each detector scores near zero recall on the other detector’s question, and each is excellent at its own. What failed was the pairing of a detector with a label, not the detector.

Neither row is a bad detector. They are competent instruments for two different questions:

• Did this file take an unusual save path?
• Were these pixels repainted?

The second is what a fraud label means. The first is what my pipeline was measuring, and it measured it well. Container forensics is a serious technique with a long literature, and the people who built it said clearly what it does not cover. Kee, Johnson and Farid extracted a 576-value camera signature from JPEG headers, showed it “is highly distinct across 1.3 million images spanning 773 different cameras and cellphones”, and then wrote in their discussion: “This analysis does not differentiate between benign and nefarious modifications.” They named the exact input that defeats it too, in the same section: the technique “is also vulnerable to a standard re-broadcast attack in which a digital image is manipulated, printed, and re-photographed.”

A photographed page of a tampered document is a re-broadcast attack that nobody carried out on purpose. It is also how a great deal of real document evidence arrives, because someone photographs a page with a phone. The benchmark was not being adversarial toward my detector. It was being ordinary, and my detector had no input left to operate on.

That is a level-of-analysis error, and it is cheaper to catch than it sounds. The check is to write down, in one sentence each, the question your detector answers and the question your labels ask, and then read them next to each other. Mine were two different sentences and I had never written either one down.

What this run does not show

The reproduction is synthetic, and its limits are load-bearing.

Its pages are one-dimensional Gaussian noise, not documents. The tampering is a constant offset on a contiguous block, which a sliding window statistic is well suited to find; real repainting has to survive JPEG re-encoding, rescaling, lighting and print-then-photograph pipelines, and the 0.775 recall here should be read as “a detector at the right level of analysis gets non-zero recall”, not as a performance estimate for anything. Document image forgery localization is an open research problem: the 2025 ADCD-Net paper states that natural-image forgery detectors “struggle with document images, as the tampered regions can be seamlessly blended into the uniform document background (BG) and structured text.”

The cohort sizes are small on purpose, to match the shape of the incident, which means every rate here carries wide uncertainty. Eighteen genuine documents in cohort G, and 22 in each labelled cohort, cannot pin down a false positive rate to any useful precision. The structure detector’s 1.000 and the null detector’s 0.000 are exact in the sense that they are structural, not sampled: they are what the arithmetic produces, not what a sample happened to give.

And the underlying real run stays private. I am not quoting its numbers here, because a reader cannot check them. What transfers is the arithmetic, and the arithmetic does not care whose pipeline it was.

The five minute version

1. Before anything else, ask for the confusion matrix, not a headline metric. TP, FP, FN and TN in four cells, with the denominators next to them. A headline metric has a zero-division policy hiding inside it, and a 2024 review of over 1.5 million papers found that even the widely repeated preference for one aggregate metric over another is “often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments”.
2. Check whether TP plus FP is zero. If it is, precision is undefined and any number your tooling printed for it is a configuration default.
3. Ask for recall on a labelled set that contains positives, before asking about precision. Not because recall is the goal, but because a detector with zero recall has no precision-recall trade to discuss.
4. Include the null detector in the comparison table, permanently. It costs one row and it turns “our specificity is 1.000” into a claim with a control.
5. Write down the question the detector answers and the question the labels ask, in one sentence each. Read them next to each other.
6. Never report a rate computed on a cohort with no positives as evidence of detection ability. A genuine-only run measures the false positive side and nothing else, and it is free.

Point three is the one I would ask a vendor. Not “what is your accuracy”, which is 0.110 for a dead detector on cohort P and 1.000 for the same detector on cohort G. Recall, on a labelled set, with the number of positives stated.

What is still open

The gap that produced this story is not closed. Pixel-level tamper localization needs a pixel-level model, and that is a different build, not a better parser: it has its own labelled data problem, its own robustness problem under compression and re-photography, and its own false positive budget to negotiate with a review team. The gap is still open in my own work, fidelityai.dev included, and saying so costs less than shipping a number whose only content is that the switch is off.

The part I am least sure about is the fix for the underlying habit. Writing the two sentences down catches the mismatch once you suspect it. I do not have a cheap way to make the mismatch visible to someone who does not suspect it yet, other than the null detector row, which at least makes a dead detector look exactly as good as doing nothing.

Primary sources

• scikit-learn, precision_score API reference and Metrics and scoring, for the definitions and for the documented zero-division behaviour.
• E. Kee, M. K. Johnson and H. Farid, “Digital Image Authentication from JPEG Headers”, IEEE Transactions on Information Forensics and Security 6(3), 2011, for the camera signature and for its authors’ own statement of what it does not cover.
• T. Saito and M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”, PLOS ONE, 2015, on the misreading of specificity.
• C. Qu et al., “Towards Robust Tampered Text Detection in Document Image: New Dataset and New Solution”, CVPR 2023, and the DocTamper repository, for the 170,000-image labelled benchmark of tampered text in document images.
• M. B. A. McDermott et al., “A Closer Look at AUROC and AUPRC under Class Imbalance”, 2024, for how far an aggregate-metric preference can travel without a citation behind it.
• K. Wong et al., “ADCD-Net: Robust Document Image Forgery Localization via Adaptive DCT Feature and Hierarchical Content Disentanglement”, 2025, for the current state of document image forgery localization.