The Photograph

Two photos come back for the same search: one an exact tag match scoring well past 19, the other a semantic-only match scoring under 2. Shown side by side with no context, both just look like "a result." Nothing in a bare ranked list tells a visitor that one of them is a near-certain match and the other a loose, recall-driven guess.

The Problem

This system had two separate places where "the system says X" needed to be checked against "is X actually true," and they were not the same problem. On the search side: finalScore is unbounded and additive, mixing a 0–19+ lexical scale with a 0–2 capped semantic one — showing that raw number to a user would imply a precision the system doesn't have. On the assistant side: Claude can say almost anything in free-text prose, including a plausible-sounding relationship between two photos that nothing in the evidence pack actually supports.

The Investigation

For search, the question was how to communicate relative confidence without pretending it's an absolute one. matchPercentage normalizes every result against the top result in that specific query's result set — max(1, round(finalScore / topFinalScore × 100)) — not against any fixed or theoretical maximum. I tried framing it as a probability at first and rejected the idea outright: finalScore's scale varies by query shape (pure-lexical queries produce very different ranges than hybrid ones), so any fixed-scale mapping would produce numbers that looked precise but meant different things from query to query. The type system now enforces the ordering — rankPhotos() returns a type that has no matchPercentage field at all, so reading it before computeMatchPercentages() has run is a compile error, not a runtime bug I'd have to catch by testing.

For the assistant, the investigation ran deeper, because the failure mode is adversarial, not just cosmetic. The Citation Gate (validateCitations()) was built first and does exactly one thing: every citedPhotoIds value in Claude's response must exist in the evidence pack it was given, or the entire response is discarded and replaced with a fixed rejection message. That closed one attack — Claude referencing a photo it was never shown. It did not close a subtler one, found during a dedicated red-team pass (RT-008, 2026-07-02): Claude citing only real, in-pack photos, but narrating a relationship between them that the deterministic Evidence Relationship graph never actually computed — "these were taken with the same camera," when no camera relationship was ever derived. The Citation Gate checks which photos are discussed. It has no opinion on what is claimed about them.

The Decision

For match percentage, the decision was to keep the number honest by scoping its claim tightly: it answers "how good is this relative to the best match in this set," and nothing more. Floored at 1% for any returned result — never 0%, which would read as a bug rather than a low-confidence match — and never exposed as a raw score anywhere in the UI.

For the assistant, closing the narrower gap the Citation Gate left open needed a second, independent structural check, not a bigger prompt instruction. Claude's output schema grew a claims: AssistantClaim[] field alongside answer/citedPhotoIds, and validateClaims() runs as pure graph comparison — no language parsing, no regex, no second Claude call — checking every claim's photoIds against the same Evidence Relationship graph the pack was built from, bidirectionally. Any invalid claim discards the whole response, the same hard-fail pattern as the Citation Gate. A third, smaller fix followed once this shipped: Claude sometimes narrated the same true relationship twice and emitted a duplicate claim for each mention — a deduplication artifact, not a grounding failure — so normalizeClaims() collapses exact duplicates before validation runs, without ever touching what validateClaims() actually checks.

What I did not close, and say so explicitly rather than papering over it: a relationship asserted only in free-text prose, with no matching structured claim, still isn't caught by anything deterministic. That gap is tracked as its own residual risk (RR-011) rather than claimed as solved — closing it fully would need language parsing, which this phase deliberately excludes.

The Architecture

Search: rankPhotos() → RankedPhotoResult (no matchPercentage field — type-enforced) ↓ computeMatchPercentages() topFinalScore = max across result set ↓ matchPercentage = max(1, round(finalScore / topFinalScore × 100)) Assistant: structuredGenerate() → { answer, citedPhotoIds, claims } ↓ normalizeClaims() dedup exact-duplicate claims ↓ validateClaims() every claim checked against the Evidence Relationship graph, bidirectionally — any mismatch discards the whole response ↓ validateCitations() every citedPhotoIds value must exist in the evidence pack, or discard wholesale ↓ redactLeakedOutput()

Both checks fail the same way on purpose: wholesale rejection, never a silent partial correction. A response that's half-trustworthy is treated as not trustworthy.

Lessons Learned

I expected match percentage to be the harder design problem, since it's user-facing and I was worried about misleading someone. It wasn't — once I refused to let it be read as a probability, the rest followed mechanically. The Citation Gate/Claim Validator work was harder precisely because red-teaming kept finding a narrower version of the same problem after each fix, and I had to get comfortable documenting a residual gap as open rather than quietly calling the feature done.

The deeper lesson: "the system cited the right sources" and "the system said something true" are not the same guarantee, and I'd conflated them until the red-team pass forced them apart. A citation gate alone would have let me believe the assistant was fully grounded when it wasn't.

Looking Forward

Individual claims can now be checked one at a time against a graph of relationships. But that graph itself — which photos relate to which, and in what order — is a structure I hadn't built yet when this chapter's work started. Verifying claims against it meant the relationship graph had to exist first, which is where the actual "memory" in this system comes from.