RAG for small-business knowledge bases: a builder's playbook

A retrieval-augmented assistant for a small business answers questions from the company's own documents instead of from the model's memory: it collects the help centre, policies and price list, cleans out stale pages, splits the text into chunks with metadata, retrieves the few chunks most relevant to a question, answers with citations, and says "that's not in our documents" when it isn't. The build is a week; the value is in the cleaning, the citations and the routing of anything uncertain to a person.

Why do businesses want this, and why do they distrust it?

Owners want the same twenty questions answered without a person, at night, in the customer's words. They distrust it because they've seen chatbots invent policies. RAG is the answer to both, if the assistant only says what the documents say and shows where it came from.

Step 1: collect the right documents

Help-centre pages, policies (returns, cancellations, privacy), FAQs, the price list, opening hours, service descriptions. Not: internal chat logs, old proposals, anything with personal data. Ask the owner which page they'd send a customer for each common question; those are the documents.

Step 2: clean before you index

Most RAG failures are source failures. Remove pages that are out of date, merge contradictions (two return policies from different years), and add missing facts the owner answers by phone every day. Give each document a title, a last-reviewed date and an owner. This step is the majority of the project and the part that makes the assistant trustworthy.

Step 3: chunk and embed

Split by heading, keep chunks a few hundred words, attach metadata (document, section, date, language). Hebrew and English documents index side by side; keep the language in metadata so retrieval can prefer the customer's language. Re-index on change, automatically, from the same source the business edits.

Step 4: retrieve with a threshold

Retrieve the top few chunks and require a minimum similarity. Below the threshold, the assistant does not answer from memory; it says the answer isn't in the documents and offers a person. This single rule prevents most invented answers.

Step 5: answer with citations, in the business's voice

The prompt: answer only from the provided chunks, cite the document and section, match the business's tone, keep it short, and when the chunks don't cover the question say so. Show the citation to the customer as a link where possible. Citations are what let the owner trust and correct the assistant.

Step 6: route what's uncertain

Low confidence, anything about money beyond the price list, complaints, medical or legal content: to a person, with the question and the retrieved chunks attached. The human-in-the-loop guide covers the patterns.

Testing before hand-over

The agent-testing guide gives the full method; the "not covered" set is the one owners care about most.

What to hand over

The document list with owners and review dates, the re-index procedure, the threshold and routing rules in plain words, and a monthly job: review ten unanswered questions and either add a document or a routing rule. The support plan should include it.

Best fit and not a good fit

Best fit: businesses with recurring questions and documents that already exist or can be written in a week. Not a good fit: businesses whose answers depend on judgement per customer, or whose documents are contradictory and nobody will fix them; clean first, or don't build.

What to do this week

Ask a client for the ten questions they answer most and the pages they'd point to. If the pages exist, you have a pilot; if they don't, writing them is the first deliverable.