Your RAG doesn't need a better embedding model. It needs better chunking.

For the last week I've been running LongMemEval against Palace, a local memory store I build for coding agents. It's a single Rust binary with SQLite, BM25 and a small ONNX embedding model. No LLM sits anywhere in the write or recall path.

This post isn't about the final number. It's about what I learned getting there: which changes moved recall, which didn't, and how many of the improvements turned out to be plain bugs that only a benchmark run would surface.

Short version, so you can stop here if you want:

• Swapping embedding models did nothing on this dataset. Every difference was 1 to 4 questions.
• Chunking conversations the way the product actually stores them helped more than anything else.
• The ranking gains were mostly bug fixes. A substring match, an overeager phrase rule, and a chunker that silently dropped text.
• int8 models give different vectors depending on what else is in the batch. That one surprised me.

What's being measured

LongMemEval asks questions about a long chat history. In the s_cleaned split each question comes with a haystack of about 50 sessions (about 115k tokens), and somewhere in there are the sessions that hold the answer. 470 questions get scored; the 30 abstention questions are excluded, as usual.

I only measure retrieval: did the right session show up in the top K? The metric is recall_any@K at session granularity. That is not the same thing as the end-to-end QA accuracy many memory systems report, where an LLM reads the retrieved context and answers. Retrieval recall is an upper bound on what a reader model can do, and it's the part I control. So please don't line these numbers up against QA accuracy tables. They measure different things.

Setup for every run in this post:

• Windows, 16 threads, CPU only
• MiniLM-L6-v2 (384-dim) unless stated otherwise, 512-token cap
• Hybrid search: BM25 fused with cosine, plus the product's normal ranking boosts
• No LLM at any stage: no extraction at ingest, no query rewriting, no answer generation

One number worth keeping in your head the whole way through: one question is 0.0021. On 470 questions, a "+0.4 points" improvement is two questions. I'll say how many questions a change moved wherever I can, because decimals make small things look bigger than they are.

Starting point: the number had drifted

Back in May the README said R@5 0.981. When I reran the same configuration in October, I got 0.977.

My first instinct was "noise, run it again." But there's no noise here. The retrieval is deterministic: fixed ONNX model, BM25, cosine, no sampling. A rerun gives the same 0.977 every time. So two questions had genuinely moved, and something in the code had moved them.

Since May, the ranker had grown a few extra boosts on top of the plain BM25/cosine fusion: coding-agent intent boosts, preference matching, recency. I diffed the per-question results against the May baseline and found the first culprit:

The query contained the word "latest". The intent classifier saw "test" inside it, decided this was a question about running tests, and lifted an unrelated drawer into the top five.

The fix was to match intent terms only at word starts (inflections still match). R@5 went from 0.977 to 0.979. The coding-agent eval didn't change.

Lesson one: a benchmark number in your README is a claim about your current code. If the ranker changes, the number has to be re-earned. Reproducing an old number is a good way to find out what broke.

The model swap that changed nothing

I had just added support for switchable embedding models, so the obvious question was whether a better model would help. I ran five of them through the same per-session configuration:

Every recall difference in that table is between 1 and 4 questions. BGE got a slightly better R@1, mostly on single-session-user questions, and paid for it with worse R@5 and R@10 on multi-session ones. Nothing beat MiniLM in a way I'd be willing to call real.

What was real was the speed. The int8 MiniLM gave the same recall with about 1.4x faster ingest. That made it the interesting option, not because it's smarter but because it's cheaper for the same answer.

Lesson two: on a retrieval benchmark, the embedding model was not the bottleneck. I'd assumed it would be. Twelve-layer models cost twice as much per embedding as six-layer ones, and on this data I got nothing back for it.

A detour: batching was slower than not batching

While sizing the model work I measured embedding throughput on real drawers and found something backwards: embedding one text at a time did 94 drawers/s, and batching made it slower. 69/s at batch 16, 53/s at 32, 42/s at 64.

The cause was padding. A batch gets padded to its longest text, so one long drawer makes every short drawer in the batch pay for its length. Sorting texts by length and capping batches at 8 took it from 29/s (the old batched path) to about 110/s.

Another detour: int8 vectors depend on their neighbours

Later, when I made the int8 model the default for new palaces, an existing test started failing. It checks that embedding a text in a batch gives the same vector as embedding it alone.

For fp32 the worst cosine between "batched" and "alone" was 0.999999. For int8 it was 0.976.

Dynamically quantized ONNX models compute their activation scales over the whole input tensor. That means the whole batch. Put a text next to a long, unusual neighbour and its vector shifts. For a memory store that's a real problem: the same sentence could land in a slightly different place depending on what else happened to be mined in the same call.

The fix: int8 models embed one text per call. That costs some of the speedup, but the q8 model still indexes faster than fp32 (150 vs 132 drawers/s on real palace drawers, about 1.7x on longer synthetic ones), and now a text's vector depends only on the text.

I wouldn't have found this without the switch, and I wouldn't have switched without the benchmark telling me q8 was "free."

Chunking: measure what you ship

The usual LongMemEval setup stores each session as one document. That's not how Palace stores conversations. When it mines a chat log it splits it into exchange pairs: one chunk per user turn plus the reply to it.

So the benchmark was measuring a configuration no user ever runs. I added a --chunking flag to the runner with three modes:

• session: one drawer per session (the standard setup)
• text: the generic paragraph chunker used for project files
• exchange: each session goes through the exact normaliser and chunker that palace mine-convos uses

In the chunked modes a session ranks by its best chunk.

Exchange pairs won, and the biggest move was where I least expected it: single-session-preference R@1 went from 0.633 to 0.767. A preference (say, "I've been trying to cut down on caffeine") is usually one sentence buried in a long session about something else. As one big document, the session's embedding is dominated by everything else. As an exchange pair, the preference turn gets a vector of its own.

The text mode is a useful contrast. It has twice as many drawers as exchange and does worse. More, smaller chunks isn't the point. Chunks that follow the shape of the conversation are.

The chunker was dropping text

Wiring the real chunker into the benchmark turned up a bug in the product. The exchange chunker kept the user turn and the first 8 lines of the reply, and silently dropped the rest. Any long assistant answer lost everything after line 8.

Nobody had noticed because nothing ever compared mined conversations against the original. Running the real chunker over thousands of sessions with known answers did.

Replies are now kept whole. An exchange over about 2,000 characters continues in further chunks that each repeat the user turn, so the question context travels with every piece of the answer. Short sessions with fewer than three user turns fall back to paragraph chunks, the same as in the product.

Lesson three: benchmark the code path your users run, not a convenient stand-in. If I'd kept the one-document-per-session setup I'd have a slightly lower number and a chunker that was still eating replies.

Ranking: three general mechanisms and one bad rule

At this point I set myself a rule, which I'll quote because it shaped everything after it: "We're not tuning for the score, we're tuning for the output." LongMemEval is a proxy. Reading the misses to find general weaknesses is fine. Writing rules that fix specific questions isn't. Any change has to be a mechanism that would help a real palace, and has to be checked on data other than the data that inspired it before a number gets published.

With that in place, here's what changed in the ranker.

1. "should I" is not a convention question

Palace has an intent classifier, and questions that sound like project conventions ("should we use tabs?") get a boost toward convention-style drawers. The trigger included "should I".

One preference question in LongMemEval starts with "should I". The classifier filed it under conventions and boosted the wrong drawers. I narrowed the trigger. That question recovered, and R@1 on the coding-agent eval (a separate fixture, not LongMemEval) went up too.

2. Date-aware ranking

Temporal questions ("what did I buy two weeks ago?") were the category I most wanted to improve. In a live palace every drawer has a date, so a question that points at a time window should favour drawers from that window.

To make that testable the runner had to stop cheating in the other direction: every question is now asked on its own question_date, and every drawer is dated by its session. Relative dates like "10 days ago", "last Tuesday" or "in the past month" now resolve the way they would in a real palace.

The ranker parses those spans and gives drawers inside the window a small boost (weight 0.12). Two bugs came out while building it: spans like "two weeks" have to count in full rather than as a point, and "on Tuesday" in a sentence usually isn't a time reference at all, so that pattern is skipped.

Honest result: the temporal R@5 didn't move on s_cleaned. The three temporal misses that remain lose on topic, not on date. The gold chunk is from the right day and still scores below unrelated chunks from the same days. Date-awareness helps break ties; it can't rescue a chunk that doesn't look relevant. On larger haystacks, with many sessions per day, it's mostly a tie-breaker too.

I kept it anyway, because in a real palace "what did we decide last week" is a very common question, and the mechanism is right even when this dataset doesn't reward it much.

3. Sources that match in several places

When a session ranks by its best chunk, a session where three different exchanges match the question looks the same as a session where one does. That's throwing away evidence.

Now a drawer gets a small lift (weight 0.1) from the best other matching drawer from the same source. I picked the weight on a different split plus probes from a real palace, froze it, then ran s_cleaned once. 0.1 was best; 0.3 already hurt.

The bad rule I'm still carrying

In the interest of honesty: Palace's separate coding-agent eval is partly tuned. The ranker contains a handful of phrases copied from that eval's own questions. With them, the int8 model scores R@1 0.922 on it; without them, 0.902. The README says so and shows both. They have zero effect on LongMemEval and on real-palace probes, and I'm keeping them only until general mechanisms reach the same numbers.

I mention it because it's exactly the kind of thing that makes benchmark numbers meaningless if nobody discloses it, and it's very easy to do by accident.

Where it ended up

With exchange chunking, int8 MiniLM, and the ranking changes above, frozen before the run:

Per question type (exchange pairs, q8):

From the start of the week, exchange pairs went from 0.906/0.985/0.994 to 0.915/0.987/0.996. That's four more questions at R@1, one more at R@5. Per session, R@5 is back to 0.981, the May number, but now from code that's actually in the repo.

What's still weak

Preferences. Still the weakest category. "I prefer X" sentences rarely share words with the question that needs them ("what should I cook tonight?"), so BM25 can't help and a 384-dim embedding only partly can. Exchange chunking helped a lot. The rest probably needs something structural, not a new weight.

Temporal questions that lose on topic. As above. Not a date problem.

s_cleaned is near the ceiling. At R@5 0.987 the remaining misses are a handful of questions, and it's very easy to fit them by accident. That's why the source weight was picked elsewhere, and why I'm careful about reading much into the last few points. I've also been running the much larger m_cleaned split, where the numbers are lower and more informative. I'll write about it once I'm happy with what it says.

This is retrieval, not answers. The next honest step is running other memory systems through the same harness, so the comparison is like for like instead of across different metrics.

Things I'd tell myself a week ago

1. Rerun your old numbers before chasing new ones. The drift pointed straight at a bug.
2. Count questions, not decimals. One question is 0.0021 here. Most "improvements" are one or two.
3. Benchmark the path you ship. The product's chunker beat the standard setup, and testing it found a bug that had been quietly dropping text.
4. A bigger model isn't a free win. On this data it was slower and no better.
5. Quantization has side effects beyond accuracy. int8 vectors depended on batch neighbours. A boring equality test caught it.
6. Pick weights on data you won't report on, freeze them, then run once.
7. Disclose the tuned bits. Even small ones, even in a separate eval.

Reproducing it

Everything here runs locally from the repo. The benchmark runner, the exact commands for each row, and the per-question outputs are in the benchmarks docs:

https://github.com/AncientiCe/palace-rs

If you run it on different hardware or another split and get different numbers, I'd like to hear about it.

One question for you: has anyone else seen int8 or quantized embeddings drift depending on what else is in the batch? I'd love to know if it's just my setup.