The Beginning

The first version of the cover-letter idea, in May, was about as simple as an idea can be: take a job description, take my work history, ask Claude to write a cover letter. I built it in an afternoon and it produced something that read fine.

The First Solution

"Read fine" was the whole bar at first. Hand the model the job description and enough context about my background, trust the prompt's instruction to "only use what's actually relevant," and let it write. It felt reasonable because it's the way most people describe using an LLM — give it context, ask for output.

What Broke

It broke in two separate ways, months apart, and both taught me the same lesson from a different angle.

The first was obvious once I saw it: a model asked to write persuasively doesn't leave gaps quiet. If the real evidence for a job requirement was thin, Claude filled it with something plausible-sounding instead of just not mentioning it. That's not a tone problem. A cover letter representing me to an employer can't contain a project I didn't do, and a CV reordering my actual dates for narrative flow is worse than an omission — it's a factual error with my name on it.

The second was subtler and showed up later, once I'd already built a deterministic evidence-scoring layer to fix the first problem. The keyword dictionary that layer used to match a job description against my experience was necessarily incomplete — about 150 entries, hand-maintained, missing however many synonyms and near-misses I hadn't thought of. So I added an embedding-based recovery pass: if a job-description phrase was close enough to something already in my own vocabulary, promote it. I assumed a single similarity threshold would do the job, tune it once, move on. It didn't. When I calibrated it against five real job descriptions — thousands of phrases — the similarity scores for confirmed-good and confirmed-bad promotions overlapped completely. There was no clean cutoff. Manual review turned up real false positives sitting right in the middle of that range: "velocity" promoted to "vector" at 0.570, "documentation" to "database" at 0.587, an English verb "architect" absorbed into the German noun "Architektur" at 0.567 — an artifact of the vocabulary deliberately including German terms so German-language job descriptions get the same benefit, which meant German and English phrases shared embedding space in ways I hadn't anticipated. I'd assumed similarity scores cluster into "clearly good" and "clearly bad." They don't, not at this scale, not with this data.

The Decision

For the first problem, I considered trusting the prompt — "use only what's relevant" — and rejected it, because I'd already watched the failure mode: a model handed ten pieces of evidence when three actually apply doesn't discard seven, it rationalizes using all ten. So evidence scoring became genuinely authoritative, not just a suggestion the model could override, and the pack handed to Claude is capped at eight items — not tuned for length, but because an uncapped pack is an open invitation to synthesize connections between things that were never meant to be read together.

For the second problem, one threshold clearly wasn't going to work, so I split it into two: a looser threshold (0.55) at the expansion stage that keeps the full diagnostic range for future recalibration, and a stricter one (0.65) at the point where a promotion is actually trusted for retrieval. Everything in between gets computed and then thrown away. I also had to decide whether to keep expanding the keyword dictionary by hand instead — I didn't, because a dictionary that has to anticipate every phrasing in advance was exactly the problem I was trying to get away from.

There was a third decision, smaller but one I'm glad I made deliberately instead of by default: when a shared structuredGenerate() wrapper emerged later for other Claude call sites, I considered migrating cover-letter generation onto it for consistency. I rejected that too. Cover-letter generation needs a retry loop that re-prompts specifically on the banned phrasing or opening-line violations it just committed, escalating with the exact offending text each time — a shape structuredGenerate() doesn't have. Forcing it in would have meant either complicating every other caller of that wrapper for one caller's benefit, or building a cover-letter-shaped escape hatch into a primitive whose whole value is being simple. CV summary generation, by contrast, had one small post-processing step that fit a general hook the wrapper already supported, so it migrated cleanly. Two different shapes staying different wasn't a compromise — it was just honest about what each call site actually needed.

The New Architecture

Job Description ↓ Deterministic Extraction (keywords + capability candidates) ↓ Capability Normalization (0.55 expand / 0.65 trust) ↓ Evidence Scoring [authoritative, capped at 8] ↓ Persona + Rhetorical Plan (framing only, never new content) ↓ Claude (prose only, retry-with-escalation on violations) ↓ PDF (Python/ReportLab, deterministic layout)

CV tailoring reuses the same pipeline output but narrows what Claude can touch to just the headline and summary — a validator runs after generation specifically to assert nothing else changed, because a CV is a factual record in a way cover-letter prose isn't.

Lessons Learned

The evidence-cap behavior surprised me the first time I saw it happen live — I'd assumed a well-written prompt was enough to make a model self-limit, and it just isn't, reliably. The threshold-calibration result surprised me more, because it meant an assumption I hadn't even noticed I was making — "similarity scores separate cleanly" — was simply wrong, and I only found that out by actually calibrating against real data instead of picking a number that felt right.

If I'm honest, I'd have calibrated the threshold before shipping the first version, not after finding false positives in production-adjacent testing. It worked out because I caught it early, but "pick a threshold that sounds reasonable" is a habit I'd like to break earlier next time.

Looking Forward

Behavior-profile generation is still on the older manual call chain and hasn't been migrated to structuredGenerate() — not because it can't be, but because nothing's forced the question yet. And I want a test that fails loudly if the evidence-pack cap is ever accidentally raised, rather than relying on the function itself being the only thing enforcing it.

Takeaway

Letting a model write persuasively was never the risky part. The risk was ever letting it decide what's true, what's relevant, or where the line sits between "close enough to promote" and "close enough to be wrong" — and every fix in this pipeline, in the end, is just that same boundary drawn one more time in a slightly different place.