Insurance AI Needs Evidence Graphs, Not Just Fraud Scores

A fully synthetic claim, SYN-CLM-0042, arrives in a human queue labelled INVESTIGATE. The label sounds actionable. It is also almost empty. What exact evidence path produced this route?

An adjuster should be able to answer that question without replaying a model’s private reasoning or trusting a polished summary. Which bytes arrived? Which facts came from which coordinates? Which sources shared an origin? Which rule fired first? Which fields were never measured? If those answers are missing, the recommendation is doing too much work.

Status first: this is a fully synthetic training fixture, not a customer record. Product traces and deployment observations are outside the exercise. The claim stays uncertain; the graph makes its lineage reconstructable, reviewable, and hard to erase.

The recommendation is not the record

Suppose a claim system returns a fraud score of 0.78. The number invites a decision before it invites a question. A reviewer can raise the threshold, lower it, or add another model, yet still have no durable account of what the score depended on. A later appeal then becomes an exercise in reconstructing a vanished run from logs, prose, and memory.

I would rather begin with the record. At minimum, it should identify the exact input artifacts, their hashes and timestamps, extracted facts with source coordinates, retrieval queries and ranked candidates, applicable policy citations, contradictions, shared dependencies, software versions, the ordered rule that determined the machine state, and every human action that followed. The recommendation occupies one field among all of them.

Even a conservative recommendation needs this structure. INVESTIGATE could mean a suspected loss, a missing timestamp, contradictory locations, or nothing more than an overloaded fallback label. Those four meanings demand different work. Reason codes and evidence links tell the reviewer which question remains open.

There is also a temporal reason to preserve structure. A document may be corrected, a policy excerpt superseded, or a reviewer may resolve a contradiction. Overwriting the old summary destroys the basis of the earlier route. An append-only lineage lets a second reviewer reconstruct what the machine knew at the time and compare it with what was added later.

Here is the test used throughout the article: can another person reproduce the path from frozen inputs to a bounded machine state, challenge any edge, and record a different human outcome while preserving the original uncertainty?

Five artifacts, seven facts

The fixture contains five exact UTF-8 artifacts, each with a terminal newline. The deliberately boring identifiers act as stable provenance handles; they encode no semantic conclusion.

A hash detects a byte change. Content certification is a separate task: a claimant can repeat an error in three files and produce three perfectly valid hashes. In this fixture, A1, A2, and A4 belong to the same claimant_intake_package, so their independence is not established. A4’s agreement with A1 remains inside that shared package.

A3 exposes a different weakness: its source image timestamp is unavailable. The note reports an observed area while the timestamp field remains empty; promoting it to timestamped image evidence would fabricate provenance. A5 is a wholly synthetic policy excerpt and not a real policy or legal rule. Its 30-day window exists only so the fixture has an inspectable rule input.

The extraction layer produces seven facts. Coordinates remain attached; the derived gap points back to both date facts.

These rows preserve the different source roles. F4 is supported by intake JSON and a driver statement from the same package. F6 estimates the same rear-left area. F5 comes from an inspection note and points to the front-right fender. The output is an explicit conflict between claimed and observed locations, alongside an independence warning; a majority vote for “rear left” would erase both.

Here is the first of two illustrative standard-library examples; they are fixture checks, not production code. It freezes A4’s exact bytes, including its final newline.

from hashlib import sha256

artifact = b"item,area,amount_eur\npaint_and_labor,rear_left_bumper,850.00\n"

assert len(artifact) == 61
assert sha256(artifact).hexdigest() == "a096b9d95e47b90184680a870f280073e95a5d53ecfe051885a37a8018d4b54d"

Run it after changing a comma or removing the newline and the assertion fails. That failure is modest but valuable: a reviewer knows whether two conclusions used the same artifact bytes before debating the conclusions themselves.

A graph is stricter than a transcript

A transcript tells a story in sequence. An evidence graph states which claim each artifact can support, how a fact was derived, and where two facts disagree. In this fixture the typed route is compact:

• A1 → F1 - extracts.
• A1 → F2 - extracts.
• A1 → F4 - extracts.
• F1 → F3 - derives.
• F2 → F3 - derives.
• A2 → F4 - supports.
• A3 → F5 - observes.
• A4 → F6 - estimates.
• A5 → F7 - governs.
• F4 ↔ F5 - contradicts; bidirectional conflict CG1, unresolved.
• F6 → F4 - consistent_with; independence unverified.

The edge types prevent a common category error. supports carries a claimant statement, observes carries the inspection note, and consistent_with preserves unverified independence. governs applies inside the declared synthetic policy scope. Two derives edges keep F3 attached to both dates instead of dressing the calculation up as a third source artifact.

The graph is also strict about absence. The missing source-image timestamp remains unavailable; filling it from the inspection date would invent provenance. That empty field affects the ordered decision rules and becomes a reason code for human review.

Figure 1. In this fully synthetic fixture, hashes and coordinates make the route reconstructable: F4 and F5 remain contradictory, so the machine abstains and INVESTIGATE routes the record to human review.

Similarity is a lead, not proof

Retrieval is useful when a claim has more material than a person can search manually. A generic retrieval pattern can send both sparse lexical matches and dense semantic candidates into fusion, then rerank the combined list. Lexical search can preserve an exact policy phrase or part number. Dense search can surface a semantically similar description that shares few words. Fusion gives the reranker a broader candidate set.

Each item in that set is a lead. Proof requires a matching event, applicable product, date, and jurisdiction. A reranker score orders candidates for inspection; evidence sufficiency is decided later.

I would therefore retain the query, sparse and dense candidate identifiers, original ranks, fusion method, reranker version, final ranks, and the exact spans later cited. The review layer can then ask separate questions: was the candidate relevant, did it establish the proposition, did another source contradict it, and was the source applicable?

This explanation follows Qdrant’s public hybrid-search material as a generic retrieval pattern. It is a useful starting point for thinking through how retrieval, evidence bundles, and human review can work together.

Where FidelityAI fits

FidelityAI grew out of the November 2025 Pioneers AI Hackathon, where our team explored evidence-led insurance workflows. It is a useful lens for the practical question behind this article: how should a system carry its reasons forward when a claim is incomplete or contradictory?

The team presented and explored a five-phase claim-review concept. The immutable static page shows five workflow phases, ten roles, ingestion and retrieval stages, a voice interview, adversarial analysis, a final-judge stage, human-readable Markdown artifacts, and a named technology stack.

The current site describes multimodal evidence, five workflow stages, per-stage artifacts, adjuster-facing recommendations, audit exports, and tenant-aware controls. The details belong to the product story; this article stays with the portable design question of evidence lineage, contradictions, and decision records.

The observable link to this article is the static deck’s human-readable Markdown-artifact claim. The reference design tightens that displayed idea: every artifact gets a hash, every fact gets a coordinate, each dependency is named, contradictions remain open, and the final machine behavior is allowed to abstain.

Three agents can share one failure

Specialized roles can be useful because they ask different questions. A coverage_hypothesis role may look for missing material, an inconsistency_hypothesis role may challenge incompatible facts, and a review_synthesizer may organize the result for a person. The danger begins when role names are mistaken for independent evidence.

In this example, all three share extraction_snapshot, context_bundle, base_model, and prompt_family. That makes agreement correlated: the same extraction error, omitted document, model blind spot, or prompt assumption could propagate through every role.

A useful record keeps the shared-dependency set beside role outputs. A reviewer should see whether two claims came from separate observations or from two prompts over the same context. Counting agents without exposing dependence turns presentation diversity into false statistical confidence. No judge role can vote correlated hypotheses into ground truth.

Audio needs the same discipline. In this design, audio supplies transcript and content evidence only. Acoustic deception, emotion, hesitation, and identity inference stay outside the evidence model. A statement can be cited by timestamp or transcript span; vocal delivery contributes no hidden fraud feature.

Figure 2. Three review roles share extraction, context, base-model, and prompt-family dependencies. Agreement needs to be interpreted in that context; there is no vote or risk score, and audio is limited to transcript and content evidence.

The useful output of these roles is a set of reviewable hypotheses: a missing timestamp, a candidate contradiction, or a citation that deserves inspection. Evidence strength still comes from the underlying artifact and its provenance. Agent agreement may help prioritize work while leaving source strength unchanged.

Abstention is a product behavior

Abstention is sometimes treated as an evaluation trick: move the threshold until the system answers fewer cases. In a product, it needs deterministic precedence and a useful exit. The fixture applies these rules in this exact order:

1. OUT_OF_SCOPE - no applicable policy.
2. CONTRADICTORY - unresolved material conflict.
3. INSUFFICIENT - missing required provenance.
4. SUPPORTED - required evidence and no unresolved conflict.

Order matters. An unresolved material contradiction takes precedence over the presence of another required field. A complete-looking bundle with conflicting locations stays contradictory. Policy applicability is checked before either evidence-quality branch.

The notice assessment is deliberately simple: 4 <= 30 supports timely notice only inside the wholly synthetic policy fixture. It says nothing about an actual contract. The location assessment is different: F4 conflicts with F5 in CG1, and that conflict is unresolved. The machine disposition is ABSTAIN, with reason codes UNRESOLVED_LOCATION_CONTRADICTION and MISSING_SOURCE_IMAGE_TIMESTAMP.

INVESTIGATE means workflow routing only; it is not a fraud judgment, approval, denial, confidence score, or adverse action. It moves the evidence bundle to a person with two concrete questions: which damage location should govern, and can the original image provenance be recovered?

The second fixture-only code block calculates the date gap, preserves the rule order, and shows the contradiction taking precedence over missing provenance.

from datetime import date

loss_date = date.fromisoformat("2026-02-14")
reported_date = date.fromisoformat("2026-02-18")
notice_gap_days = (reported_date - loss_date).days

RULE_ORDER = ("OUT_OF_SCOPE", "CONTRADICTORY", "INSUFFICIENT", "SUPPORTED")

def classify(*, applicable: bool, unresolved_conflict: bool, missing_provenance: bool) -> str:
    if not applicable:
        return RULE_ORDER[0]
    if unresolved_conflict:
        return RULE_ORDER[1]
    if missing_provenance:
        return RULE_ORDER[2]
    return RULE_ORDER[3]

state = classify(applicable=True, unresolved_conflict=True, missing_provenance=True)
machine = "ABSTAIN" if state == "CONTRADICTORY" else state
queue = "INVESTIGATE" if machine == "ABSTAIN" else None

assert notice_gap_days == 4
assert (state, machine, queue) == ("CONTRADICTORY", "ABSTAIN", "INVESTIGATE")

The example is small enough to audit. A production rule engine would add policy versioning, authorization, and more states while preserving the visible precedence and the distinction between route and verdict.

The human decision needs its own lineage

Human review needs the same lineage as the machine path. A resolution requires a reviewer pseudonym, reason, timestamp, and cited evidence. If the reviewer accepts F5, rejects it, or obtains a timestamped source image, that action gets a new record instead of silently editing the machine output.

The human decision record keeps the reviewer pseudonym, reason, timestamp, cited evidence, resolution, override, and appeal together. That makes a later decision easy to trace without turning the workflow into a black box.

An empty field is a prompt to gather more context, not a score. The same distinction matters for confidence, overrides, and appeal history: a useful record explains what is known and what needs attention next.

An override should point backward and forward: backward to the machine state, reason codes, and cited artifacts; forward to the new human state and its justification. The appeal record should do the same. That structure lets a later reviewer separate “the machine abstained because locations conflicted” from “a person resolved the conflict after obtaining a new image.”

For SYN-CLM-0042, the assigned reviewer needs A1 and A3, the F4/F5 conflict, both reason codes, and the coordinate links behind them. Wider dossier access adds no evidence for resolving CG1; the queue record should stay tied to this concrete review task.

A test that could prove the design wrong

Strong evaluation starts with a controlled set of claims, clear denominators, and an agreement on what every score is meant to capture. The scorecard below keeps those questions visible from the start.

Start with a blinded set of frozen claims containing complete evidence, missing provenance, seeded contradictions, irrelevant retrieval candidates, and adversarially similar cases. Retain each original rating plus a separate adjudication record, then evaluate in five stages.

Stage 1 - Reconstructability. Ask a second reviewer to recover every cited span and ordered state from the frozen artifacts. Report citation precision and unsupported-claim rate with their denominators.

Stage 2 - Decision behavior. After defining the target and denominator, report the calibration curve, expected calibration error, and Brier score. Declare the confusion threshold and publish the true positive, false positive, true negative, and false negative cells.

Stage 3 - Cost and subgroup analysis. Attach false-positive cost and false-negative cost to that threshold. For every fairness result, state the subgroup identity, denominator, and fairness metric.

Stage 4 - Robustness. Run separate slices for missing evidence, contradictory evidence, and adversarial evidence. A merged robustness average would hide which evidence failure breaks the route.

Stage 5 - Operations and review. Report latency p50 and latency p95, reviewer agreement, override rate, and the appeal result when appeals exist. Each value keeps its denominator and status.

The design would be falsified by several outcomes: citations that cannot be reconstructed from frozen artifacts, contradictions that vanish during synthesis, unsupported cases that bypass abstention, or reviewers who cannot determine why a case reached their queue. It could also fail operationally if the evidence record takes longer to inspect than the underlying claim or if reason codes do not help reviewers choose a next action.

Figure 3. An evaluation scorecard should connect reliability, cost, fairness, citation quality, robustness, and review. Any future number needs the relevant denominator beside it.

An implementation order

The controls are easier to build when provenance comes before orchestration. I would use this sequence:

1. Freeze each input’s exact bytes, media type, hash, acquisition time, and dependency group.
2. Extract typed facts with source coordinates and preserve absent provenance as absent.
3. Record derivations, support links, contradictions, and independence assumptions as typed edges.
4. Retrieve candidates with logged queries, fusion inputs, ranks, and reranker versions.
5. Apply the ordered evidence rules before any recommendation or route is exposed.
6. Store model, prompt, tool, calibration, and shared-dependency versions beside every run.
7. Make abstention actionable with reason codes and a routing-only human queue.
8. Record reviewer identity, reason, timestamp, cited evidence, override, and appeal lineage.
9. Evaluate citations, costs, calibration, fairness, robustness, latency, review, and appeals with declared denominators.

This order produces useful checkpoints before the full system exists. Exact bytes and coordinates already improve incident review. Typed contradictions already stop a prose summary from smoothing over incompatible facts. A deterministic abstention route already gives a human a more precise task. Multi-role orchestration should come after these controls because it multiplies whatever provenance quality is present upstream, good or bad.

The operational check uses the fixture itself. Give a second reviewer SYN-CLM-0042 without the original chat transcript and ask for the exact inputs, fact edges, conflict, rule, and human task. Any missing link identifies a concrete defect in the evidence graph.

Primary sources

• The public pitch-deck commit records five phases, ten roles, human-readable Markdown artifacts, and a named technology list.
• The separate team presentation repository captures the hackathon’s collaborative context.
• The current public site describes its present-day product positioning.
• EIOPA’s supervisory discussion provides European context on assisted AI under human oversight, explainability, hallucination, data protection, and vendor risk.
• The NAIC model bulletin announcement describes regulator expectations around governance, accuracy, fairness, documentation, and legal compliance. The NAIC model bulletin is not a universal law, and adoption varies by jurisdiction.
• NIST AI RMF 1.0 is a voluntary general framework for risk management, testing, documentation, and transparency rather than insurance regulation.
• Qdrant’s hybrid-search and reranking tutorial explains sparse and dense retrieval, fusion, and reranking. It supports the generic retrieval explanation; specific-product architecture remains outside its scope.

Together, the EIOPA, NAIC, and NIST documents supply governance context; product validation remains outside their scope.

For SYN-CLM-0042, the source ledger ends at a concrete reconstruction: A1 → F4 carries the claimed rear-left area, A3 → F5 carries the observed front-right area, and CG1 preserves their conflict. The ordered rules therefore yield ABSTAIN. A later reviewer can resolve that route with cited evidence or leave the contradiction open; either outcome stays attached to the exact path that produced it.