RAG vs Fine-Tuning: How to Customize an LLM
Use retrieval-augmented generation (RAG) when the AI needs to know your facts: policies, product details, documents that change. It looks up relevant passages at answer time and can cite them. Use fine-tuning when you need to change how the model behaves, such as a consistent format, tone or narrow classification task, and you have many high-quality examples. Most business projects should start with RAG and good prompts, and add fine-tuning only if testing shows a clear gap.
- RAG supplies knowledge at answer time; fine-tuning changes the model's behavior through extra training.
- RAG handles changing information well: update the documents and answers change immediately.
- Fine-tuning needs a large set of high-quality examples and repeated evaluation.
- Fine-tuning does not replace retrieval for current facts, as Microsoft's own documentation notes.
- Fine-tuning availability varies by vendor and model, so check current support before planning around it.
At a Glance
| Factor | RAG (retrieval) | Fine-tuning |
|---|---|---|
| What it changes | What the model knows for each answer | How the model tends to respond |
| Best for | Facts, policies, documents, product data | Format, tone, narrow tasks, smaller cheaper models |
| Keeping current | Update documents; no retraining | Retrain when behavior or data changes |
| Citations | Can point to the source passage | No built-in way to cite sources |
| Data needed | Your existing documents, cleaned and organized | Many curated input and output examples |
| Cost profile | Search index plus longer prompts | Training runs, hosting and evaluation |
| Access control | Can filter documents by user permissions | Anything trained in is available to all users |
| Main risk | Poor retrieval returns the wrong passage | Overfitting, cost and stale behavior |
Choose RAG if the AI must answer from your company's information, especially information that changes or must be cited. Choose fine-tuning if you have a stable, repetitive task with plenty of good examples and prompting alone cannot get the format or behavior right. For many production systems, the answer is RAG first, with fine-tuning as a later optimization.
Two Techniques for Two Different Jobs
Business owners often ask to "train the AI on our data". That phrase hides two different goals. One is knowledge: the AI should know your return policy, your service menu, last quarter's price list. The other is behavior: the AI should always reply in a certain structure, classify tickets into your categories or write in your brand voice.
Retrieval-augmented generation addresses knowledge. The term comes from a 2020 research paper by Lewis and colleagues, which combined a language model with a searchable document index. In practice, your documents are split into passages and indexed. When someone asks a question, the system retrieves the most relevant passages and gives them to the model along with the question, so it answers from your material. Our explainer on what RAG is covers the mechanics.
Fine-tuning addresses behavior. You train an existing model further on examples of inputs and the outputs you want, which adjusts its weights. Microsoft's documentation describes fine-tuning as useful for consistent style and structure, better tool use, and specializing a smaller model, while noting that it does not replace retrieval for current information.
Why Most Projects Start With RAG
- Your information changes. Prices, staff, policies and inventory change weekly. With RAG you update a document and the next answer reflects it. A fine-tuned model would need retraining.
- Answers can be traced. RAG systems can show which document an answer came from, which makes reviewing and correcting much easier.
- Permissions can be respected. Retrieval can filter documents by who is asking, so a front-desk employee does not see payroll files. Information trained into a model cannot be hidden from some users.
- It uses data you already have. Your existing documents, help articles and FAQs are the starting point, after cleanup.
RAG has its own failure mode: if retrieval returns the wrong passage, the model confidently answers from the wrong material. Most RAG quality work goes into chunking documents sensibly, adding context to each passage, combining keyword and semantic search, and testing with real questions. Anthropic's write-up on contextual retrieval is a good technical example of those improvements.
When Fine-Tuning Is Worth It
Fine-tuning earns its cost when you have a stable, repetitive task and plenty of good examples. Typical cases are classifying thousands of incoming messages into your categories, extracting fields from a consistent document type, producing output in a strict format, or getting a smaller, cheaper model to match a larger one on one narrow task.
It also comes with real overhead. You need curated training data, separate test data, repeated training runs, and evaluation to confirm the tuned model is actually better than a well-prompted base model. When the base model is retired, you may need to retrain. Vendor support also shifts: OpenAI's documentation states it is winding down its fine-tuning platform for new users, while Microsoft Foundry, Google Cloud and open-weight models continue to support tuning. Confirm current availability for the model you plan to use.
Combining Both Approaches
The two are not mutually exclusive. A mature system might use RAG to supply current facts and a fine-tuned model that is better at using retrieved context and following your output format. Microsoft's fine-tuning documentation specifically lists training models to use retrieved context more effectively as one use case. For a small business, though, that combination is usually a second-phase optimization, not a starting point.
A concrete example: a property management company wants an assistant that answers tenant questions. Lease terms, building rules and maintenance procedures belong in a RAG index, because they differ by property and change over time. If the company later wants every reply to follow a strict template and route into one of fifteen ticket categories, and prompting cannot achieve that reliably at volume, a tuned model for the classification step might be justified. The knowledge still comes from retrieval.
A Practical Order of Operations
- Prompting first. Clear instructions and a few examples solve more problems than people expect.
- Add RAG when the AI needs your specific information. Clean the source documents first; see how to prepare business data for AI.
- Build an evaluation set of real questions with correct answers, and score every change against it.
- Consider fine-tuning only if evaluation shows a consistent behavior gap that prompting and retrieval cannot close.
This order keeps costs low and gives you evidence before each step up in complexity. If you want help scoping a knowledge assistant, our AI automation services start with exactly this kind of assessment.
Sources and Further Reading
We link to the primary documentation so you can verify the details yourself.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), arXiv
- Fine-tuning in Microsoft Foundry, Microsoft Learn
- Model optimization, OpenAI
- Introducing Contextual Retrieval, Anthropic
- RAG solution design and evaluation guide, Microsoft Learn
Related Questions
Does RAG mean my documents are sent to the AI provider?
The retrieved passages are sent to the model with each question, so the provider processes them. Use a business or API plan whose terms exclude training on your data, check retention settings, and keep highly sensitive documents out of the index unless your setup is approved for them.
How many examples do I need to fine-tune a model?
It depends on the task and model, but think in hundreds to thousands of high-quality, consistent examples, not a few dozen. The quality and consistency of examples matters more than raw volume, and you need a separate test set to prove improvement.
Can fine-tuning stop an AI from making things up?
Not reliably. Fine-tuning shapes behavior but does not give the model a dependable source of truth. Grounding answers in retrieved documents, telling the model to decline when the answer is missing, and testing are more effective against hallucination.
Is RAG expensive to run?
Usually not for small-business volumes. Costs come from the search index or vector database, embedding your documents and the longer prompts that include retrieved passages. Those costs are typically modest compared with the build and testing work.