We build a lot of systems that answer questions from a client’s own documents. The ones that hold up in serious settings all share a boring trait. When they give you an answer, they can tell you exactly where it came from: section 4(b), page 12, that specific clause. The ones that fail can’t. They hand you a confident paragraph and no way to check it, which in legal or compliance work is worse than no answer at all. Someone has to trust it, and nobody sensible will.

Most people assume the fix is a smarter model. Usually it’s something much less glamorous. It’s how the documents got sliced up before they were ever stored.

What retrieval actually does

Ask an AI a question about your files and it doesn’t read all of them. It pulls a few short passages out of a search index and writes an answer from those. “RAG” is the three-letter name for this. Whatever the model’s talent, the answer can only be as good as the passages it grabbed. And those passages were fixed the day you loaded the documents in.

The lazy way to chop up a document

Most tools slice by length. Every 500 words or so, cut, store the piece, keep going. Quick to build. Fine if you’re asking loose questions about a pile of blog posts.

It falls over the moment an answer has to hold up. A length-based cut has no idea where a section ends, so it saws straight through the seam between two clauses and staples half of each into one passage. Now that chunk blends two rules that were never meant to touch, and it can’t say where it came from, because the thing that made the cut was counting characters, not reading headings. “Somewhere in these forty pages” is not a citation. I’ve watched a demo come apart on that exact question.

Cut where the document already bends

Documents tell you where their joints are. A numbered section. A heading. A clause. A rule. Slice on those instead of every N words and each passage lines up with something a person would actually point at.

Then keep the address. Store the document, chapter, section and page right next to the text. It reads like filing-cabinet housekeeping and it’s the whole game. That metadata is what lets the system quote a source instead of vaguely waving at one. We work this way because our clients get read by people whose job is to catch the passage that can’t be sourced.

Do the slow work once, on your own hardware

Two more calls, both about not getting burned in production.

Load and index everything one time, on machines you control, rather than phoning an outside service on every question. Public legal sources are slow and heavily rate-limited, and now and then they’re simply down. Build on top of a live call and every lookup becomes a small bet against the network. Pay the storage cost once and the answer is sitting there when someone needs it.

Be polite to whatever you do pull from. Free tiers cap how fast you can fetch. Point a greedy fetcher at one and you’ll burn the daily quota in a few minutes, then get locked out. Ours crawls slowly and remembers where it stopped, so a big backfill grinds its way to done over a couple of days rather than dying at the halfway mark.

Three questions to size up any RAG tool

You don’t need to read the code. Ask it three things.

Where did that come from? A tool that chunks well will name the section. If the best it can do is “this PDF,” it’s cutting by length.

Show me the passage. You want a clean piece, one rule from start to finish, not a fragment that opens in the middle of a sentence.

Ask it again tomorrow. If it reaches out to the internet at question time, the answer will wobble with the connection. If it indexed once, it stays put.

If a tool flunks those, the model is almost never the culprit. The trouble sits upstream, in how the material got prepped, and that’s genuinely good news, because you can fix prep yourself without waiting on anyone’s next release. Put your attention on the chunking before you spend a dollar on a bigger model. If the vocabulary here is new, our AI & automation glossary walks through embeddings and vector search. And if you’re weighing a build like this, start with How to scope an automation project before you buy.