RAG explained: building an AI knowledge base for SMEs

What RAG is and where the term comes from

A language model only knows its training data, which does not include your price lists, work instructions or maintenance contracts. RAG closes this gap: before answering, a search step finds matching passages in your documents and hands them to the model with the question. The model answers from them and can cite the source. The Datenschutzkonferenz (DSK), the joint body of German data protection authorities, describes the ideal case: all factual knowledge comes from the reference documents; the model only supplies the language.

The term comes from a research paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, published in the NeurIPS 2020 proceedings. They linked a language model to a search index over Wikipedia, split into around 21 million 100-word passages. Two of their arguments still hold: answers can be traced to sources, and knowledge can be updated by swapping the index instead of retraining the model. Unlike the original, which trained retrieval and model jointly, RAG systems as the DSK describes them leave the model unchanged.

How RAG works, step by step

The DSK assumes a system of four building blocks: embedding model, vector database, retrieval component (retriever) and language model. The process in six steps:

• 1. Prepare the documents: extract text from PDF, Word, wiki or ticket system. The DSK advises removing headers, footers and page numbers first. Scans also need text recognition (OCR).
• 2. Split into passages: documents are cut into shorter passages (chunks). Fixed lengths can tear related content apart; ideally each chunk holds one complete thought. Chunk size and overlap affect quality.
• 3. Compute embeddings: an embedding model turns each passage into a numeric vector approximating its meaning; similar texts end up close together. Vector and text are stored together. For German documents, the DSK recommends a model trained on German texts.
• 4. Search by meaning: the same embedding model converts the question into a vector, and the database returns the closest passages.
• 5. Add keyword search: meaning-based search can miss exact matches such as article numbers or error codes, so it is combined with keyword search such as BM25 (hybrid search), and reranking re-sorts the hits. In Anthropic’s tests, this plus added context per passage cut missed passages by 67 percent.
• 6. Answer with the source: the model receives question and passages with the instruction to answer only from them and cite the source, a system instruction the DSK also mentions. If nothing suitable is found, the assistant says so instead of guessing.

When RAG fits and when it does not

There are three basic ways to give a language model your knowledge. Which fits depends mainly on how large and changeable your collection is:

RAG and fine-tuning are not mutually exclusive: a study cited by the DSK observed that each can improve results independently. For an SME knowledge assistant, RAG is still the obvious start, because documents change and answers should be traceable. To calculate, query stock levels or trigger processes, you need defined interfaces rather than a knowledge assistant, as used by AI agents.

Permissions: who may see which documents?

The language model itself cannot control who sees which information, as the DSK also notes. Permissions must therefore apply in the search, before a passage reaches the model. The 2025 OWASP list of top risks for language model applications explicitly names inadequate access controls in vector databases as a cause of unauthorised access to sensitive content.

• Take permissions from the source: each passage stores its source document and who may open it. A question only searches passages the signed-in person could see in the original.
• Filter technically: Qdrant separates data in a shared collection via a tenant or group field that every search filters on. PostgreSQL row-level security restricts per user which rows a query returns. With pgvector’s approximate indexes, filters apply only after the index scan; since version 0.8.0, iterative index scans can make up for missing hits.
• Separate collections where needed: for clearly separate purposes, such as departments with their own files, the DSK names tenant or functional separation and a permissions and roles concept.
• Protect vectors like plain text: according to OWASP, significant amounts of source information can be recovered from embeddings. The vector database needs the same protection as the documents, plus immutable logs of retrieval activity.

Data protection: what the DSK guidance on RAG says

In October 2025 the DSK published dedicated guidance on generative AI systems using RAG (version 1.0). It covers systems with embeddings and a vector database and assumes the entire system runs on the controller’s own premises. Key points:

• Legal basis: even processing personal data from the documents with the embedding model and storing it in the vector database requires a legal basis.
• Accuracy: RAG can reduce hallucinations and inaccurate output but not rule them out. Outdated or incomplete documents produce wrong answers, so the controller must review the collection regularly. An added web search regularly fails the requirements for verified data.
• Data minimisation and erasure: unneeded personal data is removed or anonymised before ingestion. Vector database entries can be deleted selectively; standard deletion after retention periods works here too.
• Data subject rights: access, rectification and erasure can usually be implemented for inputs, outputs, documents and the vector database, but largely unsolved within the model itself.
• Purpose limitation: personal data from your documents can become linked inside the model with personal data from its training, without this showing in the output. Countermeasures belong in the design phase.
• The model stays what it is: RAG does not make an unlawfully trained model lawful. But because the model needs less factual knowledge, more models come into question, including smaller ones on your own premises.

If the model or the vector database runs with a cloud provider, processing agreements, processing location and possibly third-country transfers come into play. See the guide Using ChatGPT and co. in line with the GDPR and, for own hardware versus cloud, the local AI versus cloud AI cost comparison. For chats that people interact with directly, Art. 50(1) EU AI Act has also applied since 2 August 2026: people must be told they are dealing with an AI system unless this is obvious. The duty lies with the provider, which can already be a company running its own chatbot on a third-party model. More in The EU AI Act for SMEs.

Typical sources of error and how to catch them

• Outdated and duplicate documents: with three versions of a price list in a folder, the system cannot tell which applies. Remedy: ingest only approved folders, archive old versions, store the validity date.
• Poor scans and layouts: without text recognition, a scanned PDF stays empty for the search; multi-column pages and tables get jumbled. Tools such as Docling detect reading order and table structure and include OCR for scans. Spot checks remain essential.
• Passages without context: “the notice period is three months” is worthless without knowing which contract is meant, so passages should carry document title and heading. In Anthropic’s tests, added context per passage alone cut missed passages in meaning-based search by 35 percent.
• Wrong passages: the search returns a similar-sounding but unsuitable passage, such as another customer’s maintenance contract. Hybrid search, filters on customer or validity and a visible source citation make this rarer and easier to spot.
• Hidden instructions: OWASP describes a CV with white-on-white text instructing a screening system to recommend the candidate. According to the DSK, documents from unknown or untrustworthy sources should not be used anyway.
• Hallucinations despite RAG: the model can mix in knowledge from its training and, on contradictions, follow what it learned rather than the documents. The DSK explicitly names this faithfulness to context as a remaining challenge.

Measuring quality: test questions, not gut feeling

Whether an assistant is any good shows not in the first demo but in a fixed set of test questions rerun after every change. Here is how:

• Collect real questions: from the inbox, phone notes and the team, each with expected answer and source. Add questions the collection cannot answer; here “I cannot find anything on this” is correct.
• Assess search and answer separately: first whether the right passage was retrieved, then whether the answer rests on it. The open-source tool Ragas calls these metrics context recall and faithfulness. Anthropic measures retrieval failures as the share of relevant passages missing from the top 20 hits.
• Repeat after every change: new documents, another model or a different passage size shift results. All test questions rerun before the change goes live.
• Adjust in operation: evaluate ratings and unanswered questions. They reveal gaps in collection or search and extend the test set.

Building blocks and hosting: examples with checked licences

A RAG system can be assembled from open components. The following are checked examples, not a complete market overview:

• Everything on your own premises: documents, vector database, embedding and language model run on your own hardware, exactly the scenario the DSK assumes. Details on the page on local AI.
• Models in an EU cloud: according to IONOS, its AI Model Hub runs inference only in German data centres, does not store inputs or outputs and offers several embedding models; its built-in vector database was discontinued on 31 August 2026. Mistral AI says it hosts data in the EU by default and also offers embedding models; depending on the feature, data may temporarily leave the EU.
• Mixed: documents and vector database stay in-house; only the question and retrieved passages go to a cloud model. These passages leave the building too, so processing agreements and location must be clarified as well.

The path to your own knowledge assistant

• 1. Define the goal: internal assistant or customer chat, and which questions it should answer.
• 2. Tidy the collection: choose approved folders, remove old versions, check scans, name content owners.
• 3. Settle permissions, data protection and hosting: permissions model, legal basis, deletion concept, own hardware, EU cloud or a mix.
• 4. Prototype: a subset of documents, real test questions, source citations and permissions from the start.
• 5. Pilot with one group: AI labelling, handover to a person when unsure, feedback evaluation.
• 6. Operation: reload changed documents automatically, run test questions regularly, maintain the collection. An assistant is only as good as its documents.

How I build such an assistant, from initial review through prototype to operation, is described on the page AI chatbot and knowledge assistant. For more areas of use, see the article on AI use cases for SMEs.