Why Every AI Feature Starts With Structured Data
Why the schema behind three unrelated AI features keeps looking the same, and what one JSON blob column taught me about how much structure is actually enough
Architecture·Advanced·8 min read · July 6, 2026
The Beginning
Once the call-site machinery was unified, the question that was left standing wasn't "how do I call Claude" anymore — it was "what do I hand it." And the answer to that question turned out to already be sitting in the schema, in decisions I'd made for reasons that had nothing to do with AI at all.
The experience table is the clearest example. It doesn't store a job as one paragraph. It's a row (company, duration, title_en/title_de) with four child tables hanging off it — experience_summary, experience_tech, experience_tech_icon, experience_work_sample — each one indexed on experience_id. I built that shape originally so the public Work section could render a clean list of bullet points and a tech-stack row per job. I wasn't thinking about retrieval when I wrote it that way.
The First Solution
The obvious way to answer "does this job mention Kubernetes" against a shape like that is the same regex-over-text approach I'd use against any blob of prose — scan the summary paragraphs for a keyword and hope the phrasing lines up. That's roughly what happens today in scoreCandidateEvidence(), except it isn't scanning one blob per job. It's scanning exp.tech_stack — the literal list from experience_tech — for an exact, case-insensitive, word-boundary match against a job description's keywords, and awarding it five points if it hits. It's the same operation I described above, but structure turned it from a fuzzy paragraph search into an exact lookup against a field that exists for exactly that value.
What Broke
Nothing broke in the sense of an incident — this is the one article in the season where the failure is a road not taken rather than a road I walked down and had to back out of. The counterfactual is visible in the codebase's own stated invariant: evidence selection has to stay deterministic, and it's never bypassed with AI ranking, full stop. That rule wouldn't need to be written down this explicitly if the alternative — asking Claude to judge relevance directly against loosely-shaped context — weren't a real, standing temptation. The reason it's not what happens is that a model asked "is this relevant" doesn't answer with a clean yes or no against ambiguous input; it answers with something persuasive, and persuasive is exactly the property that makes a wrong answer here dangerous instead of just unhelpful.
The place where I did watch this distinction matter in practice was in the shape of engineering_decision, which took a visibly different structural path than experience. It's not four normalized child tables — it's a title column next to a single decision_json blob. That's a deliberate, narrower kind of structure: enough to index and query the title, and a self-contained JSON document underneath it that Ask Vikram's retrieval reads by embedding it, not by joining against it. Structuring engineering_decision the way I structured experience — normalizing every field the JSON currently holds into its own column — would have been the wrong amount of structure for how it's actually consumed. Nobody joins across engineering decisions the way the Work section joins experience to its tech stack for rendering; the corpus is read by retrieval, not by a page template, so it didn't need the same shape.
The Decision
The choice, each time, was how much structure a piece of data actually needed, not whether to have any. experience needed real relational structure because two different consumers read it two different ways — a page renders it, scoreCandidateEvidence() scores it — and both needed to address specific fields directly. engineering_decision needed a queryable identity (the title) and an internally structured but relationally opaque payload, because its only consumer is a retrieval pipeline that already knows how to read structured JSON and doesn't need SQL to join into it.
Photography went a third direction, and for a reason that isn't about retrieval convenience at all — it's about what structure makes enforceable. photo, photo_exif, and photo_tag are three separate tables, with GPS coordinates isolated inside photo_exif specifically. That separation is what makes a rule like "GPS never leaves this server" possible to enforce at the type level: PublicPhotoExifSummary simply has no GPS fields, so there's no runtime check to forget, no field to accidentally serialize. You cannot build that guarantee against an unstructured blob — there's no type system checkpoint to put it in. Structure here isn't about making retrieval faster; it's what turns a privacy rule from "something I have to remember to redact" into "something that can't compile."
The New Architecture
Same tables, two consumers, no duplicated facts:
experience / experience_tech → Work section (render)
→ scoreCandidateEvidence() (score, +5 exact tech match)
engineering_decision.decision_json → Ask Vikram embeddings (retrieval, not joins)
photo / photo_exif (GPS isolated) → Public gallery (GPS-free type)
→ Admin-only enrichment (full EXIF)
Three different shapes, three different reasons — normalize where two consumers need to address the same field independently, keep a JSON document intact where only an embedding pipeline reads it, and split a table specifically where a privacy boundary needs to be provable rather than remembered.
Lessons Learned
I didn't design experience's four-table split with cover-letter scoring in mind — it existed for the Work section long before any of the AI pipelines did, and it happened to already be the right shape when scoreCandidateEvidence() needed it. That's a lucky accident, not a principle I can take credit for in advance. Where I don't have that luck is engineering_decision: a single JSON blob makes Ask Vikram lean on embedding similarity for everything, with none of the exact-match scoring that gives cover-letter evidence its +5 exact tech match clarity. That's a real trade-off I accepted for a table that, as I've already noted, doesn't yet have the authoring tooling the blog content just got — a table this structurally thin is easier to accept when the corpus is still small, and worth revisiting the moment it isn't.
Looking Forward
Getting the data shape right answers "do I have the right facts to work with." It says nothing about how many of those facts to hand a model at once, or in what order. experience_tech being queryable doesn't stop someone from handing every matched tech-stack item, every summary bullet, and every skill group to Claude in one shot "just to be safe." Structured data is necessary. It isn't sufficient — the next question is what a bounded, ranked selection of that data actually looks like.
Takeaway
Every AI feature in this platform starts with structured data not because structure is fashionable, but because each domain independently ran into the same wall: a field you can't address directly is a field you can't score, can't redact, and can't prove anything about. The shape of the fix was never "put everything in a table" — it was "put exactly as much structure into each table as its actual consumers need to address," and no domain here needed the same amount.