The Photograph

Search for "street" by early July and you'd get back a ranked grid — the exact-tagged photo first, a few semantically related ones after it, each with a small badge explaining why it matched. That's a complete search feature. It's also, I realized looking at it, not actually what I wanted from a photo library that's supposed to feel like memory rather than a filing cabinet. A grid of results doesn't know that two of those photos were taken the same afternoon, or that three of them share a location I visited once.

The Problem

Hybrid retrieval, vocabulary expansion, canonicalization — everything up to this point made search better: more precise ranking, better recall, correct explanations for individual matches. None of it let the system reason about the set of photos it returned. Ask it "what did I photograph in Berlin" and it would hand back a pile of individually-correct results with no sense that they were connected, or in what order, or why they mattered together rather than separately.

A keyword search bar, however well-tuned underneath it, was never going to answer a question shaped like that. The interface itself was the wrong shape for what the retrieval pipeline had already become capable of feeding it.

The Investigation

The retrieval side didn't need to change at all — that was the first real finding. expandPhotoQuery() and its output, ExpandedPhotoQuery { originalQuery, normalizedQuery, promotedTerms, remainingSemanticTerms }, had already been shaped, without me planning it that way at the time, as an intent model something more conversational could consume directly. The question wasn't "how do I build query understanding for an assistant" — that already existed. The question was what to put on top of it, and how much autonomy to give a model once it was there.

The tempting version of "add an assistant" would have let Claude call search itself, decide what to look for, maybe run a second query if the first didn't seem to answer the question. I rejected that immediately, for the same reason cover-letter evidence selection stays outside Claude's control: an assistant that can decide what to retrieve is an assistant that can quietly decide what's relevant, and "relevant" drifting into "convenient" is exactly the failure mode this entire platform exists to prevent. Claude does not get to run its own search.

The Decision

The design that shipped (2026-07-02) calls the existing hybrid retrieval primitive exactly once per question, caps the result at 8 photos, and builds a fixed EvidencePack before Claude is ever invoked. Claude explains the pack; it does not choose the pack. buildEvidencePack() is a pure mapping over already-ranked results — it never re-scores, never re-sorts, never adds a photo that retrieval didn't already surface.

The harder decision came a day later, on 2026-07-03: whether the assistant should sit alongside the existing keyword search bar, or replace it outright. I chose to replace it — the page's SearchBar/SearchResults components were deleted, not deprecated, and PhotographyAssistant took over their exact screen position, restyled to match: single-line input, centered, Enter-to-submit. That's a real product bet, not just an engineering one: a visitor no longer picks between "search" and "ask," there's one box, and it happens to be grounded in the same deterministic retrieval either mode would have used. The gallery itself didn't disappear — it's still there as the idle state, rendered as the assistant's own children prop, the same component doing double duty as both "nothing asked yet" and "here's the whole collection."

The Architecture

User Question ↓ retrievePhotos() same shared primitive search already used — unmodified ↓ buildEvidencePack() deterministic: cap 8, preserve rank order, no image URLs ↓ [pack empty?] → fixed NO_RESULTS_ANSWER, no Claude call ↓ buildAssistantPrompt() evidence pack in retrieved_context (trust: derived), question in untrusted_input (trust: untrusted) ↓ structuredGenerate() Claude explains the pack; never computes it ↓ Assistant Response

EvidencePackPhoto deliberately carries no image URLs — it's Claude's context payload, not a render payload. The UI resolves which photos to actually show by joining citedPhotoIds against the gallery photos the page already fetched, so the backend schema never grew a new field just to support rendering.

Lessons Learned

I expected building "an assistant" to mean building a new intent-understanding layer. It didn't — the biggest surprise here was how much of a conversational interface's hard part had already been solved by treating search's own query-expansion output as a reusable intent model, months before I knew an assistant would consume it.

The product decision — delete the search bar rather than keep both — is the one I'm least certain about in hindsight. It works because the assistant is grounded in identical retrieval, so nothing is actually lost in capability. But it's a one-way door for anyone who specifically wants fast, keyword-shaped browsing rather than a question-and-answer interaction, and I don't yet have real usage data telling me whether that trade was right.

Looking Forward

Handing Claude a fixed evidence pack stops it from inventing which photos to discuss. It says nothing about what Claude is allowed to claim about the photos once it has them — and that gap, between "cited the right photos" and "said something true about them," turned out to need its own answer.