Two years ago, I set up Enterprise Delivery Excellence (EDE). One of the visions I set for it sounded simple and turned out to be hard: consolidate and centralise our information. Pull the scattered documents, policies and playbooks out of inboxes, shared drives and people’s heads, and into one organised place. The immediate case was delivery excellence. People shouldn’t lose hours hunting for the right document, and no two teams should work from a different version of the truth.
But that vision was always aimed further ahead. A single, organised source of information is the foundation you build everything else on. And it is exactly what AI needs to be genuinely useful inside a business. Get the knowledge consolidated first, and you are ready to put AI on top of it. The technique that does that, i.e., turning your organised information into answers, is called Retrieval Augmented Generation (RAG). It is the bridge between a clever general-purpose model and your own organisation’s knowledge, and it is well worth understanding.
To see why the foundation matters so much, start with what a language model does badly on its own.
Ask a large language model a question about your own company, and you’ll often get a confident, well-written, completely wrong answer. Ask it what discount your sales team can offer on a big deal, and it will happily invent a number. It’s not because it’s broken, but because it has never read your price book. It’s answering from memory, and your price book was never in its memory.
This is the core limitation of a language model on its own. It knows an enormous amount about the world in general, frozen at the moment it was trained, and it knows nothing about your documents, your customers, or anything that happened after its training cut-off. Worse, it doesn’t know what it doesn’t know, so it fills the gap with a plausible guess.
RAG is the technique that fixes this. And the simplest way to understand it is an exam you’ve sat many times.
The Open-Book Exam
A language model answering from memory is a student sitting a closed-book exam. Bright, articulate, well-read, but forced to answer from whatever they happened to remember. When memory runs out, they bluff. Sometimes the bluff is right. Sometimes it’s a confident, fluent mistake.
RAG turns it into an open-book exam. Before the model answers, it goes and finds the relevant pages, e.g., from your documents, your database, your knowledge base, and puts them on the desk in front of itself. Then it answers the question using those pages. Same clever student, but now they’re reading from the actual textbook instead of guessing from memory.
That’s the whole idea in one sentence: RAG retrieves the right information first, then lets the model generate its answer from that information. The “retrieval” is fetching the pages. The “augmented generation” is writing the answer with those pages in hand.
How It Actually Works
Under the hood there are four steps. None of them is as complicated as the jargon makes them sound.
- Prepare the library (indexing). First, your documents are broken into small passages, a process called chunking. Because you want to retrieve the precise paragraph that answers a question, not a whole 90-page handbook. Each passage is then converted into an embedding: a list of numbers that captures its meaning. Think of it as giving every passage a coordinate, so that passages about similar things end up near each other, the way books on the same topic sit together on a shelf. These coordinates are stored in a vector database, i.e., a library organised by meaning rather than by title.
- Understand the question (retrieval). When someone asks a question, the question itself is turned into the same kind of coordinate. The system then looks for the passages whose coordinates sit closest to it. This is why RAG is powerful: it matches on meaning, not just keywords. Someone can ask “how long do I get off when I have a baby?” and the system will find the paragraph headed “Parental Leave Entitlement,” even though the words don’t match, because the meanings do.
- Hand over the pages (augmentation). The passages it found are slipped into the prompt, alongside the user’s question, as background for the model to read.
- Write the answer (generation). The model now answers the question using the passages it was given. In a well-built system, it cites which passage each fact came from, so the answer is traceable back to a real source.
One thing worth being clear about: the model doesn’t learn your documents permanently in this process. It reads them at the moment of answering and forgets them afterwards, the way a student reads a page during the exam without memorising the whole book. That’s a feature, not a flaw. It means the moment you update the document, the next answer is up to date, with no retraining required.
A Worked Example
Say your sales team keeps getting stuck during quotation queries. A customer asks a specific question about pricing, discounts or contract terms, and the rep has to stop, dig through folders, and ping three colleagues before they can answer. So the business builds an assistant that answers these questions from the company’s own current documents, e.g., the price book, the discount policy, and the standard contract terms. A rep types:
“The customer wants a 3-year deal for 500 licences. What discount can I offer, and what are the payment terms?”
Here is what happens behind the scenes. The question is turned into a coordinate and matched against the sales library. It pulls back the passages that matter: the volume discount table, the rule for multi-year commitments, and the standard payment terms. Those passages are handed to the model along with the question, and it answers something like:
“For 500 licences you’re in the 250–999 volume band, which carries a 15% discount. A 3-year commitment adds a further 5%, for 20% in total. Standard terms are annual payment in advance, though 3-year deals qualify for quarterly billing on request. Anything beyond 20% needs sign-off from a sales director. (Source: Price Book v4, Discount Policy sections 2 and 5.)”
Notice what just happened. The answer is specific to your pricing, not a generic guess. It correctly flags the approval threshold, so the rep doesn’t promise a discount they can’t give. It’s current, because it read this quarter’s price book rather than last year’s. It cites its source, so the rep or their manager can check the figure before it goes to the customer. And when pricing changes next quarter, the assistant is correct the moment the new price book is loaded, with nobody retraining anything. That is RAG earning its keep, a faster answer in front of the customer, and less risk of quoting the wrong number.
Why Organisations Reach for It
RAG has become the default way to put AI on top of real business information, for a few practical reasons.
It answers from current and private data, e.g., your contracts, your product manuals, last night’s support tickets, none of which the base model has ever seen. It cuts down hallucination, because the model is answering from supplied facts rather than reaching into memory; when the right passage is on the desk, the temptation to guess drops sharply. It makes answers traceable, because every claim can point back to a source document, which matters enormously in regulated settings. And it’s far cheaper and faster than the alternative, e.g., retraining or fine-tuning a model every time a document changes would be slow, expensive, and out of date almost immediately. With RAG, you just update the document.
You’ll have used RAG already without knowing it. A customer-support chatbot that answers from a company’s actual help centre. A legal assistant that answers questions about a specific contract. A bank’s assistant that quotes today’s product terms. A doctor’s tool that draws on current clinical guidelines rather than a model’s fuzzy recollection. An internal “ask-me-anything” assistant sitting in Microsoft Teams, answering staff questions from the company’s own knowledge base. All of these are RAG: a general-purpose model, given an open book of specific, trusted information, answering from it.
What RAG Does Not Fix
RAG is powerful, but it is not magic, and it’s worth being honest about where it strains. Most disappointing RAG projects fail on these points, not on the clever parts.
- Bad retrieval means bad answers. If the system fetches the wrong passages, the model will write a confident answer grounded in the wrong pages. Garbage in, garbage out, except the garbage now comes with fluent prose and a citation, which can make it more convincing, not less.
- It can still hallucinate. Handing the model the right passage makes a good answer likely, not certain. A model can still misread the page, blend it with its own assumptions, or answer a slightly different question than the one you asked.
- The library has to be maintained. RAG is only as current as the documents behind it. Point it at a stale, contradictory or badly organised knowledge base and it will faithfully serve up stale, contradictory answers.
- It sees passages, not the whole picture. RAG retrieves the handful of chunks closest to a question. It’s excellent at “what does the policy say about X,” and much weaker at “read all 4,000 tickets and tell me the three biggest themes”. That isn’t what the retrieval step is built to do.
The Governance Angle
There’s one more point that sits right at the heart of the work I write about here. RAG doesn’t just make AI more accurate, it makes it more governable. And it introduces a governance risk of its own.
On the positive side, an answer that cites its source is an answer you can audit. When the model says “section 3.4,” you can check section 3.4. That traceability is exactly the kind of thing that turns an AI system from an unaccountable black box into something you can actually govern. You can see the evidence behind the action.
But the open book cuts both ways. A RAG system will retrieve from whatever library you point it at, which means if you don’t govern who is allowed to see which passages, it will happily surface a salary spreadsheet or a confidential contract to someone who should never have seen it. The retrieval step needs the same access controls as the documents themselves. Giving AI an open book is a genuine step forward. Just make sure it’s only open to the pages that particular reader is allowed to turn.
That, in the end, is the pattern across all of this. The technology gives you a powerful new capability. Whether it’s trustworthy depends on how you govern the thing it actually does.