Experimenting with a Decision Model as a RAG Chunker (Spoiler: It's Very Accurate, but Slow, with a Lot of Room to Grow!)

I had what felt like a very clever idea: instead of cutting documents every 256 tokens, or using embedding distance to guess where the topic changes, why not ask a decision model where each thought ends?

Feed it the current chunk and the next sentence. Make it choose APPEND or SPLIT. Repeat until the document is done.

It sounds smart. It works. In fact, it is surprisingly accurate. It is also painfully slow, because I used a four-billion-parameter general decision model for a job that should eventually belong to something much smaller and fine-tuned.

The short version: Imajev-4B retrieved 97.37% of the relevant evidence and matched the good cheap baselines without being fine-tuned for chunking. It also took 31 minutes to process a tiny corpus that semantic chunking handled in six seconds. The mechanism looks promising. The current engine is comically oversized.

Why I thought this might work

Chunking is one of those boring-looking choices that quietly controls an entire RAG system. If you split in the wrong place, the retriever may find the question but lose the answer sitting one sentence away.

Fixed-size chunking does not care about meaning. Semantic chunking is smarter: it embeds neighboring sentences and splits when the distance jumps. But vocabulary can change while the thought stays the same, and similar vocabulary can hide a completely new topic.

I wanted a model to make the actual decision:

current_chunk: everything kept together so far
next_sentence: the next piece of text

APPEND -> this continues, explains, or supports the current passage
SPLIT  -> this begins a different topic or self-contained section

I used Imajev-4B, a four-billion-parameter decision model with a trained choice readout and an explicit unknown option. I ran it locally through MLX. It never rewrote the source; it only chose the boundaries. A hard token cap could still force a split.

And yes, the basic behavior looked right:

A telescope sentence followed by another telescope sentence gets APPEND. A sudden jump to payroll gets SPLIT. These are sanity checks, not proof that the model understands every boundary.

This is not an entirely new direction. Meta-Chunking also explores merge/split probabilities. My test was narrower: can Imajev's decision head replace the usual chunk-boundary rule without touching the original text?

The test

I used the complete Chatlogs section of Chroma's chunking benchmark: 56 queries over 7,727 tokens.

I compared eight methods:

1. Imajev-4B decisions
2. Fixed 128-token chunks
3. Fixed 256-token chunks
4. Fixed 256 with overlap
5. Recursive 256 with overlap
6. Sentence packing up to 256 tokens
7. Semantic chunking at the 80th distance percentile
8. Semantic chunking at the 95th distance percentile

Every method used the same local BGE-small retriever. The important comparison used a shared maximum of 1,024 retrieved tokens, because comparing a fixed number of differently sized chunks is an easy way to fool yourself.

Recall tells us how much of the annotated evidence came back. IoU rewards relevant context and penalizes irrelevant or repeated context. Higher is better.

The result: accurate, but slow

Under the shared context budget, Imajev reached 97.37% recall and 7.06% IoU. That recall was the number that made me think there is something here: the decision model almost always kept the relevant evidence retrievable, despite never being trained specifically for this task.

Then the cheap methods walked in:

Plain sentence packing had slightly better IoU. Semantic chunking was effectively tied. Both were dramatically faster. So this is not a benchmark win, but it is a strong zero-shot result from a model doing a job it was never designed around.

At top five chunks, the decision model initially looked much better: 8.58% IoU, compared with 6.14% for semantic p95. Then I checked the chunk sizes. Imajev produced 87.5-token chunks on average; semantic p95 produced 180.5-token chunks. Five small chunks and five large chunks are not the same retrieval budget. Once I capped every method by tokens, most of the apparent win disappeared.

That is why the left and right sides of the graph tell different stories. The left side is flattering. The right side is fair.

Then I looked at the clock

On my Apple M4 with 32 GiB of RAM, Imajev made 289 model calls, processed 100,075 input tokens, and took 30.99 minutes.

Semantic p80 took 6.01 seconds. In this implementation, it was about 309 times faster.

This is ingestion time, not query time, so it would not slow every user question. But spending half an hour to produce roughly the same retrieval result as a six-second baseline is not a trade I would make.

The uncertainty intervals against semantic p80 and sentence packing crossed zero, so I cannot honestly claim a real improvement over either one. What I can claim is that the decision approach stayed competitive while using an unfine-tuned model and a very inefficient inference loop.

The system produced 89 lossless chunks, made reasonable boundary decisions, preserved every source character, passed 15 integrity tests, and survived an offline replay of all 305 boundaries and 1,792 retrieval records. It did exactly what I asked. The interesting question is how much of that quality can survive once the model becomes dramatically smaller and faster.

This is where it can get good

The experiment used a general four-billion-parameter decision model with a long instruction on every candidate boundary. It was not fine-tuned on chunking examples, did not reuse the repeated prompt prefix, and was called even when the boundary was obvious. I basically tested the most expensive version first.

That is why I see room to grow. A better version would use a much smaller model trained specifically for APPEND versus SPLIT. Give it the tail of the current chunk and the next sentence, fine-tune it on real boundary decisions, and make the training set heavy on difficult cases where embedding distance is misleading. A small classifier should be faster, easier to batch, and more consistent than repeatedly asking a general model to reason from a long instruction.

I would also stop calling the model for obvious boundaries. Let fixed rules handle headings and token limits. Let semantic distance handle the easy cases. Call the learned decision model only inside the uncertain middle, where the similarity score cannot confidently say whether the thought continued or changed.

That hybrid approach could keep the part I actually like: a learned boundary decision that sees the relationship between sentences. It would remove the absurd part: waking up a 4B model hundreds of times to answer a binary question.

To be clear, that is the next hypothesis, not a result from this benchmark. It still needs a labeled dataset, fine-tuning, and the same comparison against cheap baselines.

What I would build next

For production today, I would still use sentence packing or semantic chunking. They are fast, understandable, and scored just as well. But the next experiment is obvious enough that I want to build it.

I would turn the audited APPEND/SPLIT decisions into training data, add harder human-labeled boundaries, fine-tune a small model, batch the comparisons, and test it under the same token-budget benchmark. The target is not merely to preserve the 97.37% recall. It is to preserve or improve the retrieval quality while shrinking 31 minutes toward seconds.

The 4B run proved that a model can make useful sentence-boundary decisions. It also showed exactly where the waste lives. That is a much better starting point than a vague idea: I now have the labels, failure cases, baselines, graphs, and a number the smaller model has to beat.

The benchmark is small: one synthetic chat corpus, one retriever, no human boundary labels, and no answer-generation evaluation. I did not run the other four Chroma corpora or tune the prompt against the final scores. Also, the 97.37% figure is retrieval recall, not a direct measurement of boundary accuracy. The next model still has plenty to prove. At least now I know what to train it for.

Measured results · Audit summary · Rebuild the graphs · Method and evidence notes

Further reading: Is Semantic Chunking Worth the Computational Cost?, HiChunk / HiCBench.