RAG and a Knowledge Base: So Your Bot Answers From Your Data, Not Its Imagination

A chatbot that confidently quotes a client a nonexistent discount or a made-up warranty term is worse than no bot at all: you're the one answering for its words. This isn't "a glitch in one particular bot" but a fundamental property of language models — and it's exactly what RAG cures. Let's break down how a knowledge base for a chatbot works, what to put in it, how to verify the bot isn't making things up, and what it costs to do this properly.

What RAG is: an explanation without the academia

RAG (Retrieval-Augmented Generation) means generation reinforced by search. The mechanics are simple: before answering, the system searches your knowledge base for fragments relevant to the question, feeds them to the model in context, and gives it an instruction: answer only based on these fragments; if the answer isn't in them, honestly say you don't know.

In other words, the model stops being a "know-it-all from the internet" and becomes an interface to your documents: price lists, policies, service descriptions, contract terms. Every answer rests on a specific piece of your text — and that piece can be shown as the source.

Why a bot without RAG invents things

A language model is trained on internet texts up to a certain date. It knows nothing about your price list, the working hours of your branch on a specific street, or the return terms at your store specifically. At the same time, the model is built to always generate the most likely continuation of text — confidently and grammatically flawlessly, even when it has no facts. That's hallucination. And the less the model knows about a topic, the more confidently it makes things up — the client can't verify it without a source at hand, and a screenshot of an answer with a nonexistent discount lives in a dispute with you forever.

The practical consequence: a bot on a "clean model" with a prompt like "you're a consultant at store X" is doomed to invent things — the only question is when. AI on your own data is the only way to get a bot you can trust to talk with a client about prices, timelines, and terms.

How a knowledge base for a chatbot is built

Under the hood, RAG is a four-stage pipeline:

1. Document preparation. Price lists, FAQs, policies, and site pages are converted into clean text. Garbage in means garbage in the answers, so this isn't a formality: broken tables and unrecognized scans kill quality before you even start.
2. Chunking. Documents are split into fragments of 200–800 tokens — not mechanically by length, but by meaning: so a term and its exception don't end up in different chunks.
3. Embeddings. Each fragment is turned into a numeric vector that captures its meaning. Texts close in meaning get close vectors — which is why the search finds "how much does delivery cost" even when the client asked "do I have to pay for bringing it?"
4. Search and answer. The client's question is vectorized, the system pulls the 3–10 most relevant fragments (often with reranking for precision), and the model forms an answer strictly based on them — with a link to the source.

Metadata is attached to each fragment: document type, update date, product, city. This allows filtering: for a question about the Kyiv branch, the bot won't pull the Lviv branch's hours.

What to put in the knowledge base: a checklist

A practical minimum for a support or sales bot:

• the price list and payment terms — the most frequent topic and the most expensive for hallucinations;
• a FAQ from real client questions — pull six months of chat history, the structure is already there;
• delivery, return, and warranty terms — verify the wording with a lawyer, because the bot will quote it verbatim;
• product and service descriptions with specs;
• procedures for typical situations: an order didn't arrive, the size didn't fit, how to reschedule an appointment.

What NOT to put in: outdated versions of documents next to current ones (the bot doesn't know which is right — version conflicts cause the most insidious errors), internal documents with data clients shouldn't see, and "everything just in case" — excess volume worsens search precision. A good volume benchmark to start is 30–100 pages of current documents: that's enough for 80% of client questions, and such a volume is realistic to keep up to date.

How to verify the bot isn't inventing: tests and metrics

"We clicked around and it seems to answer fine" isn't acceptance. The working process looks like this:

• Golden set. 50–100 real questions with reference answers, including tricky ones: questions outside the base, questions with double meaning, provocations for a discount that doesn't exist.
• Groundedness. A groundedness metric: whether every statement in the answer is backed by a fragment from the base. It's checked automatically by a second judge model across the whole golden set after every change.
• "I don't know" as a correct answer. A bot that honestly says "I'll check with a manager" to an out-of-base question and hands the dialogue to a human is a success, not a flaw. The worst result is a confident answer from nowhere.
• Source citations. In admin mode, every answer shows which fragments it was assembled from. This speeds up error analysis many times over: you immediately see whether the problem is in the search or in the document itself.

The target figures we ship bots to production with: 95%+ groundedness on the golden set and zero confident answers to questions the base has no answer for. The golden set lives alongside the bot: every error found in production becomes a new test case, and subsequent changes can no longer break what was fixed once.

Common RAG implementation mistakes

• Dumping documents in a pile and waiting for magic. A knowledge base is a product with structure, an owner, and an update cycle — not a folder of PDFs.
• Scans without recognition. A PDF scan of a price list is, to the system, an image. Without OCR, this data simply doesn't exist.
• Populating the base and forgetting it. Prices changed — the bot answers with the old ones. You need a process: who updates the base, and how, with every change to the price list or terms. Technically the update takes minutes; organizationally it's the most common point of failure.
• No handoff to a human. Even perfect RAG doesn't cover everything. The path "bot → live human" with the conversation's context must be designed from day one.

RAG or fine-tuning the model: which to choose

A frequent client question: maybe it's simpler to "train the model on our data"? For business bots the answer is almost always RAG, and here's why:

Criterion	RAG	Fine-tuning
Updating data	minutes: replace the document in the base	a new training cycle each time
Answer source	visible which fragment it came from	invisible, baked into the model weights
Hallucinations	controlled by instruction and verification	don't go anywhere
Startup cost	from $2,000	many times more, both upfront and in maintenance

Fine-tuning makes sense for the style and format of answers, not for facts: facts change faster than you can retrain a model. The working formula for business: facts in the knowledge base, tone and behavior rules in the prompt.

What a bot on your own data costs

A turnkey RAG chatbot — from $2,000–5,000 one-time: document preparation and chunking, search setup, prompts, a golden set, integration with your site or messenger. Operating costs — $60–300 a month depending on traffic. Timeline — 3–5 weeks with testing, of which a good third goes to working with your documents, and that's normal: the quality of the base determines the quality of the bot.

Details are on the chatbots page. If the bot needs not just to answer but to act — create requests, check order status, book appointments — that's already AI agents: RAG remains the knowledge layer in them, on top of which actions in your systems appear. We've gathered the broader picture of applying AI on a company's own data on the AI for business page.

Want to assess whether your documents are ready for a knowledge base and what a bot for your task would cost? Book a free 30-minute AI audit — we'll go through your data sources, sketch the architecture, and give an honest budget estimate: maxicolabs.com/contact.