Grep first, embed when it earns it

Retrieval shows up twice in our stack and the two cases got opposite architectures. Agent memory, a few hundred curated notes, runs on files and grep. User-facing search, tens of thousands of multilingual records, runs on hybrid embeddings. The dividing line is worth spelling out, because the industry default is to buy a vector database for both.

Where grep wins

Agent memory is small, curated, and read by something that can already reason. Each note carries a one-line description in an index file; the agent reads the 40-line index, decides what is relevant, and opens two or three files. Debugging is opening the exact file the agent read. Correcting is editing a paragraph. When we prototyped the same corpus behind embeddings, retrieval became a probability distribution we had to interrogate, updates required re-embedding, and nobody could answer the only question that matters in an incident: why did it read that?

Where embeddings earn it

Search across 40,000 records in mixed Korean, English and Vietnamese is a different animal. Users type "cheap spa near the river" and the record says "riverside massage, budget". Keywords miss; vectors catch. But pure dense retrieval lost to a hybrid on our own eval set, so hybrid is what runs:

const kw = bm25(query, 40);              // exact terms, names, numbers
const dn = cosine(embed(query), 40);     // paraphrase, cross-lingual
const pool = dedupe([...kw, ...dn]);
const ranked = await llmRerank(query, pool.slice(0, 30));  // one cheap model call
return ranked.slice(0, 8);               // what actually enters the context

On a 400-query labeled set built from real search logs, recall@20 was 0.71 for keyword alone, 0.76 for dense alone, 0.90 for the hybrid with rerank. The rerank call costs a fraction of a cent and fixed the last mile better than any embedding model swap we tried.

Chunking is where quality actually lives

Fixed 512-token windows sever tables from their headers and answers from their questions. We chunk by document structure, headings and list boundaries, and every chunk carries its source path and byte offsets, so anything the model quotes is one click from the sentence it came from. A retrieval system that cannot cite its sources is a rumor generator with good latency.