The Photograph

A photo in the gallery is tagged "bicycle". Search for "bicycle" and it appears instantly — an exact lexical match, worth +5, unbeatable by anything semantic. Search for "cyclist" instead, and by the ranking rules from the previous chapter, that photo should still win. It didn't. "cyclist" isn't the tag. It falls straight through to semantic retrieval, where it competes on a capped 0–2 scale against a photo that merely correlates with the word statistically.

The Problem

Hybrid Retrieval v2 fixed the ranking regression, but it created a precision requirement it couldn't satisfy on its own: lexical retrieval is only as good as exact string matches against a fixed vocabulary. A query for "biking" has no lexical path to the tag "bike" — canonicalization (case, plurals) doesn't help, because that's not a spelling mismatch, it's a genuinely different word for the same concept. The gap wasn't a bug in the new formula; it was the cost of making lexical matching strict enough to be trustworthy.

The Investigation

The fix I built was expandPhotoQuery() — not a synonym dictionary, but an embedding-based promotion step that runs before intent extraction. It embeds the actual controlled vocabulary currently in use (whatever tags, moods, and locations exist on published photos, plus the fixed color and orientation lists), and promotes a query token onto the nearest vocabulary term only if similarity clears a threshold. I shipped the first version with a threshold of 0.75, picked without testing it against real data.

It was too conservative. A manual search for "cyclist" still returned only semantic matches — no promotion — even though "bicycle" was a real tag in the vocabulary. I added a permanent diagnostic log and re-tested against real embeddings: "cyclist" → "bicycle" (tag) landed at similarity 0.604. "beautiful" → "serene and majestic" (mood) landed at 0.485. Neither cleared 0.75. I recalibrated to 0.55 — the midpoint of the two real observations, with margin on both sides — the same day, once actual numbers existed to calibrate against instead of a guess.

That single flat threshold held for three days, until a second, subtler problem showed up on 2026-07-03: "flowers" promoting to the tag "blossoms" at similarity 0.5527 was a good call — tags are an open vocabulary that tolerates synonym-like promotion. But "pictures" promoting to "portrait" (an orientation value) at 0.5781 — a nearly identical similarity — was a bad one. Orientation is a closed two-value set; a wrong promotion there doesn't add noise, it actively corrupts filter extraction. The same number was a correct decision in one category and a wrong one in another, at almost the same similarity score. There was no single threshold that could be right for both.

The Decision

I split the one flat threshold into five, one per vocabulary category, each set by how much a wrong promotion actually costs: tag 0.55 (open vocabulary, tolerates broader recall), mood 0.62 (more subjective, stricter), location 0.65 (avoid false-positive place matches), color 0.70 (few colors, easy to confuse), orientation 0.75 (closed two-value set, a false promotion strongly distorts search). These are still explicitly framed as conservative starting values calibrated from a handful of real observed log entries, not final ones — the same diagnostic logging that caught the first miscalibration stays in place specifically so the next one is measurable, not guessed at.

Before reaching for embeddings at all, I also added a cheaper, fully deterministic step: canonicalizeTerm() folds case, Unicode form, accents, and regular English plurals, so "mountain" and "mountains" collapse to one vocabulary entry with zero embedding calls. It's deliberately not a stemmer — it doesn't touch "biking" or "cyclist", and it's scoped to the tag category only, because blindly stripping a trailing "s" from a location name would mangle "Athens" into something wrong. Reserving the embedding layer for genuinely fuzzy cases, after the deterministic layer has already resolved the easy ones, keeps the probabilistic step focused on what actually needs it.

The Architecture

Query token │ ▼ Exact vocabulary match? → yes → use as-is, zero embedding calls │ no ▼ canonicalizeTerm() match? → yes → deterministic promotion, zero embedding calls │ no ▼ Generic photo-search stopword? → yes → leave token, still passed to semantic retrieval │ no ▼ Embed token → KNN-1 against photo_vocabulary_embedding_vec │ ▼ similarity ≥ getVocabularyPromotionThreshold(category)? │ yes │ no ▼ ▼ Promote to nearest term Leave token unchanged (indistinguishable from an exact (falls through to semantic typed match downstream) retrieval, as before)

Vocabulary embeddings live in their own tables (photo_vocabulary_embedding_metadata/_vec), independent from photo-content embeddings — a tag can disappear from the vocabulary entirely when the last photo carrying it is deleted, so refreshVocabularyEmbeddings() also deletes stale terms, something photo-content embeddings never need since a photoId is stable.

Lessons Learned

Both miscalibrations came from the same mistake: shipping a threshold that felt reasonable instead of one measured against real similarity scores. text-embedding-3-small runs noticeably "cooler" — lower cosine similarity — for related-but-different words than it does for near-duplicate sentences, which is what the model is primarily tuned for. A number that feels intuitively like "clearly similar" for prose is not the same number for two single words that mean roughly the same thing.

The second miscalibration taught me something the first one didn't: a correct threshold isn't a property of the embedding model, it's a property of what a wrong promotion costs in a specific category. I'd assumed one number could serve five categories because they all went through the same similarity function. They don't need the same threshold; they need the same diagnostic discipline — log the real number, don't guess, recalibrate when the data says to.

Looking Forward

With exact intent, canonical forms, and now vocabulary-aware synonyms all resolving to the same lexical fast path, search itself had gotten precise. But precise ranked results and an actual answer to a question are two different things — and the next problem wasn't about scoring photos anymore. It was about whether the system could explain, in its own words, why a set of photos belonged together at all.