The Beginning

Structured data answers "what's true." It doesn't answer "how much of it should a model see, and in what order" — and once every domain had real structure to draw from, that became the only question left, on two fronts that had nothing to do with each other: how photos get ranked against a search query, and how a job-description phrase gets matched to my own vocabulary.

The First Solution

For photo search, the earliest version of the ranking pipeline — internally called "Hybrid Retrieval v1" — treated semantic similarity as the primary signal, with lexical tag matches contributing only small additive boosts on top, +0.03 to +0.05. It felt reasonable: semantic similarity is supposed to be the sophisticated part, the part that understands meaning instead of just matching strings, so weighting it as primary looked like trusting the more capable signal.

For capability matching, the reasoning was similarly straightforward: my hand-maintained keyword dictionary was about 150 entries and obviously incomplete, so an embedding-based recovery pass made sense — if a job-description phrase is close enough to something already in my own vocabulary, promote it. I assumed a single similarity threshold would separate good promotions from bad ones cleanly enough to tune once and leave alone.

What Broke

Under Hybrid Retrieval v1, a photo tagged "street" with a similarity score of 0.3 could lose to a completely unrelated photo scoring 0.85 on semantic similarity alone. Explicit user intent — the exact tag someone searched for — was getting outranked by the "smarter" signal, precisely because that signal had more numeric room to dominate the sum. More weight on the sophisticated-looking number produced worse rankings, not better ones.

The capability threshold broke in a way that was harder to see coming. I calibrated it against five real job descriptions — 2,015 candidate phrases, 1,711 unique — expecting the similarity scores of confirmed-good and confirmed-bad promotions to cluster into two visibly separate ranges. They didn't. The lowest score among promotions I'd confirm as correct and the highest score among phrases that should never have been promoted sat at the exact same value: 0.550. There was no gap to draw a line in. Manual review of that overlapping band turned up real, specific false positives: "velocity" promoted to "vector" at 0.570, "documentation" to "database" at 0.587, and — the one that told me something structural, not just noisy — the English verb "architect" absorbed into the German noun "Architektur" at 0.567, because the vocabulary deliberately includes German terms so German-language job descriptions get the same expansion benefit as English ones, which meant English and German phrases were sharing embedding space in a way a single-language mental model of "similarity" hadn't accounted for.

Both failures had the same shape underneath the different math: a bigger or more heavily weighted signal wasn't more informative, it was more room for the wrong thing to look confident. Semantic similarity dominating the sum didn't make photo ranking smarter, it made irrelevant results score higher with more apparent certainty. A promotion threshold treated as a single clean cutoff didn't make capability matching more generous, it let "vector" and "velocity" sit in the same range as genuinely correct recoveries with no way to tell them apart by score alone.

The Decision

For ranking, I rejected weighting semantic similarity as primary, and rejected a plain weighted average of lexical and semantic scores too — a weighted average can still let a strong semantic match outrank a weak-but-exact lexical one under the right configuration, which means it only usually avoids the v1 failure rather than structurally preventing it. What I built instead makes the guarantee structural: finalScore = lexicalScore + min(semanticScore × 2, 2). A single exact tag match is worth +5, so it always outranks the maximum possible semantic contribution of +2 — not because I tuned the numbers to make that usually true, but because the cap makes it true by construction, in every configuration, permanently.

For capability matching, one threshold clearly wasn't doing two jobs well, so I split it into two. expandCapabilityVocabulary() still promotes at a loose 0.55, on purpose — that keeps the full diagnostic range visible so future recalibration has real data to look at, instead of already-discarded evidence. But only promotions scoring at or above 0.65 are ever trusted by buildExpandedCapabilitySignal() for actual retrieval. Everything between 0.55 and 0.65 gets computed, inspected, and then thrown away before it reaches evidence scoring — the exact band where "velocity → vector" and "architect → Architektur" were living.

The evidence pack itself carries a third version of the same discipline, in a different shape. It's capped at eight items, and a profile-alignment bonus that nudges tie-breaking toward themes matching my own positioning is itself capped at 0.3 — small enough that it can only ever break a near-tie, never override the primary relevance score outright. Ties are only even considered within a 0.02 delta window; outside that window, the ranking is authoritative, full stop. Three different subsystems, three different formulas, and all three land on the same principle: a secondary signal is allowed to nudge a decision at the margin, and never allowed to become the decision.

The New Architecture

Not this: raw signal → weight it heavily because it seems smart → trust the sum This: raw signal → normalize → cap or threshold it structurally → apply only within a narrow tie-break window → ranking outside that window stays authoritative

Concretely: min(semanticScore × 2, 2) caps a signal instead of weighting it. Two thresholds (0.55/0.65) separate "worth measuring" from "worth trusting." A 0.3-capped bonus inside a 0.02 tie-break window nudges without ever overriding.

Lessons Learned

Neither failure looked like a crash. Hybrid Retrieval v1 didn't error out — it returned photos, confidently, that just happened to be wrong ones. The single-threshold capability matcher didn't reject bad promotions — it accepted them with the same numeric confidence as good ones. That's the same shape of failure I'd already seen at the generation layer, in the cover letter that fills evidence gaps with something plausible instead of staying quiet — except here it was happening one layer earlier, in retrieval, before generation ever got involved. It told me the discipline of "don't let a secondary signal decide" has to be enforced twice: once over what a model is allowed to say, and separately, earlier, over what a model is even shown.

I'd also assumed, going into the calibration, that similarity scores would cluster the way I imagined language "should" behave — clean semantic neighborhoods with visible gaps between them. At the scale of 1,711 real phrases across two languages, they don't. I only found that out by calibrating against real data instead of picking a threshold that felt reasonable and moving on.

Looking Forward

Once ranking is genuinely a multi-stage funnel — normalize, cap, threshold, tie-break within a narrow window, and only then call it final — it stops looking like "a formula" and starts looking like a pipeline with named stages, each one a place a rule can be enforced. That's not a coincidence. It's the same shape retrieval always ends up taking once you stop trusting a single number to do the whole job, and it's the shape the next article makes explicit.

Takeaway

Every time I reached for "give the model more" — more weight on the smart-looking signal, more trust in a single number, more evidence just in case — the result got worse, not better, and every fix was some version of the same move: take a signal that wants to dominate and put a hard ceiling on how much of the decision it's structurally allowed to make.