The Photograph

Picture the photo tagged "street" again, sitting next to an unrelated photo somewhere else in the gallery. On the morning of 2026-06-28, I typed a search query expecting the street photo to come back. Nothing came back — not that photo, not any photo, for any query. Search had shipped that same morning and it was, functionally, silent.

The Problem

The search backend (Phase 5, first version) was built semantic-first: every query ran through KNN over photo embeddings, with small additive boosts (+0.03 to +0.05) if a query happened to also match a tag or mood exactly. It looked complete — vocabulary vocabulary, deterministic filter extraction, an embedding table with a vec0 index. What it didn't have was any actual embeddings to search against.

The Investigation

The diagnosis was almost embarrassingly simple once I went looking: 6 published photos, all AI-enriched, all with captions and tags and a mood — and 0 rows in photo_embedding_vec. buildPhotoEmbeddingText() worked. upsertPhotoEmbeddings.ts worked. refreshPhotoEmbeddings() worked, if you called it. Nothing did. enrichPhotoById() updated a photo's AI metadata and stopped — it never called upsertPhotoEmbedding(). The enrichment flow, laid out step by step, was actually:

1. Download image from Firebase ✓ 2. Call Claude Vision (enrichPhoto) ✓ 3. Update DB with AI metadata ✓ 4. Upsert embedding for search ✗ ← missing 5. Return result ✓

Step 4 simply didn't exist. Every photo I'd enriched over the previous two days had a caption and tags and no embedding at all — and nothing about the enrichment flow's return value or the admin UI would have told me that, short of running a query against the vec table directly.

I fixed it the same morning — re-fetch the photo after the metadata update, upsert its embedding, wrap it in try/catch so a failed embedding never rolls back a metadata write that already succeeded. Two days later, on 2026-06-30, with embeddings now actually populated, a second failure showed up: a photo tagged "street" with a semantic similarity of 0.3 lost a ranking contest to a completely unrelated photo with a similarity of 0.85. The small additive boost for an exact tag match (+0.03 to +0.05) wasn't nearly enough to overcome that gap. Semantic similarity was, quietly, running the entire ranking — and it had no idea what "street" meant to me, only what it statistically correlated with.

The Decision

For the missing-embeddings bug, the fix was mechanical once found — wire up the missing call, make it non-fatal, done. The harder decision was about ranking, because the honest fix wasn't a bigger boost, it was rejecting the entire premise that semantic similarity should be primary. I considered tuning the boost values up instead — 0.03 to something larger — and rejected it, for the same reason a bigger evidence cap wouldn't have fixed the cover-letter hallucination problem: a bigger number in the same broken formula just moves where it breaks, it doesn't stop it from breaking.

Application Documents had already answered this exact question for cover-letter evidence: deterministic retrieval provides precision, semantic retrieval provides recall, and exact intent always dominates fuzzy similarity. I adopted that philosophy directly rather than re-deriving a photography-specific one. The formula became additive with a hard cap: finalScore = lexicalScore + min(semanticScore × 2, 2). A single exact tag match is worth +5. The best possible semantic contribution, capped, is worth 2. A photo can no longer out-rank an exact match by being merely similar — that's not a tuning choice, it's an invariant, and the ranking decision doc explicitly marks the formula stable, not a phase in progress: any future change to it should be treated as reopening this exact regression, not a routine adjustment.

The Architecture

Query │ ▼ extractPhotoFilters() deterministic: tags, mood, location, color, orientation │ ├─────────────┐ ▼ ▼ Lexical Semantic Retrieval Retrieval (KNN over photo_embedding_vec — now actually populated) │ │ └──────┬──────┘ ▼ mergePhotoRetrieval() ID-based merge, union matchedFields, never duplicates ▼ rankPhotos() finalScore = lexicalScore + min(semanticScore × 2, 2) ▼ applyDiversity() × 2 ▼ Results

enrichPhotoById() now upserts the embedding immediately after the metadata write, in the same non-fatal try/catch pattern used for the vocabulary-embedding refresh added later — a failed embedding is logged, not swallowed silently, and never blocks the enrichment the admin actually asked for.

Lessons Learned

The embarrassing part isn't that the bug existed — pipelines have missing steps sometimes. It's that nothing in the system's own output would have told me it was missing. Six enriched photos, all looking correctly enriched in the admin UI, with a silent, invisible hole two steps later. I'd assumed a working admin UI meant a working pipeline; it meant the parts I could see were working.

The ranking bug taught me something more specific: I'd assumed additive boosts were a safe, conservative way to blend two signals. They're not, if the two signals operate on wildly different scales — a capped semantic contribution isn't a tuning parameter, it's a structural guarantee, and the difference between those two only shows up once you actually test a query where the two signals disagree.

Looking Forward

With lexical matching now authoritative, a new and much narrower problem appeared almost immediately: lexical matching only recognizes exact vocabulary. Someone searching "biking" gets nothing from a photo tagged "bike" — not because the ranking is wrong, but because the two words were never told they meant the same thing.