RAG vs fine-tuning: when retrieval, training or a better prompt wins
Retrieval changes what a model can see; fine-tuning changes how it behaves. A comparison of cost, updates, citations and upkeep, with a decision table and a prompt test to run first.
Treating RAG vs fine-tuning as two ways to teach a model your facts leads to the wrong choice. Retrieval-augmented generation (RAG) changes what the model can see when it answers. It searches your documents and passes the relevant passages into the prompt. Fine-tuning changes how the model behaves. It retrains the model on examples until a format, tone or set of labels becomes habit. If the problem is missing or changing facts, use retrieval. If the problem is unsteady behaviour in a task you repeat at volume, consider fine-tuning. And test a better prompt before you pay for either.
RAG vs fine-tuning: what each one changes
The term RAG comes from a 2020 research paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. It paired a language model, whose knowledge sits in its trained weights, with a searchable index of Wikipedia that the model read at answer time. Business RAG systems apply the same idea to your own policies, contracts and manuals.
Fine-tuning works on the weights themselves. You supply example inputs with the outputs you want, and training adjusts the model towards them. The table compares both with the cheapest option, a better prompt.
| Question | Better prompt | RAG | Fine-tuning |
|---|---|---|---|
| What changes | The instructions and examples the model reads | The documents the model reads at answer time | The model's weights |
| Updating a fact | Edit the prompt | Replace the document and re-index it | Retrain on new examples |
| Can an answer show its source? | Only if the source is in the prompt | Yes, if the system shows the passages it retrieved | No, the knowledge is not traceable to a document |
| Removing a piece of data | Delete it from the prompt | Delete it from the index | Retrain without it |
| Upfront work | Writing and testing instructions | Cleaning documents, indexing, access rules, testing | Collecting and labelling examples, training runs, testing |
| Main running cost | Prompt length | Search plus longer prompts | Hosting or calling the custom model |
| Skills needed | A workflow owner who writes clear instructions | Data engineering and search tuning | Machine learning and evaluation |
The table shows why accuracy is not the only test. Retrieval keeps knowledge outside the model, where you can read it, change it and delete it. Fine-tuning moves knowledge into the model, where you can do none of these without another training run.
Why to test a better prompt first
We recommend writing an evaluation set and a careful prompt before you build either. OpenAI's model optimization guide sets out the same order. It starts with evals that measure output, then prompting, and treats fine-tuning as useful for some use cases. A prompt costs little to change and shows quickly whether the model can do the task at all.
A fair prompt test has four parts: the role, the rules, three to five worked examples and a sample of real documents pasted in. Run it against 30 to 50 real questions with agreed answers. If it passes, you may need neither method, only a shared prompt the team uses. If it fails because facts are missing or out of date, that points to retrieval. If it fails because the format or labels drift, that points to fine-tuning.
Our guide to Arabic LLM evaluation explains how to build that question set for Arabic and mixed Arabic and English work. The same set later tells you whether RAG or fine-tuning improved anything. Our Agent Evaluation Sets service prepares questions with expected answers that check an assistant before it goes live and after every change.
When retrieval is the right answer
Choose RAG when answers depend on documents that change, when users need to see where an answer came from, or when different staff may see different documents. Policies, procedures, product terms and regulator circulars all fit this description.
An illustrative example: a Jordanian insurer wants call-centre staff to answer questions on policy wording in Arabic. Policy terms change each renewal season, and supervisors need to check which clause an answer relied on. Fine-tuning would freeze last season's wording into the model. Retrieval lets the insurer swap the documents, show the clause next to each answer, and limit each agent to the products they sell.
Retrieval has its own costs. Scanned Arabic documents need OCR checks, spelling variants need normalising, and someone has to keep the index current and the access rules in line with your systems. Our step-by-step guide to Arabic RAG covers that work. Where knowledge is locked in old document stores or spreadsheets, retrieval can be the first useful layer on top of legacy systems, before anything is replaced.
When fine-tuning earns its cost
Fine-tuning pays off when the task is narrow, repeated at high volume, and a tested prompt still cannot hold the behaviour steady. OpenAI's guide, which OpenAI is moving to legacy documentation as it winds down its own fine-tuning platform, lists the gains: more consistent formatting, more examples than fit in one prompt, shorter prompts, and a smaller, cheaper model for one task. Classification, extraction into a fixed structure and a house writing style are typical candidates.
An illustrative example: a UAE logistics company sorts incoming customer emails, written in Arabic, English or a mix of both, into a fixed set of request types. The categories are stable, the volume is high and a misrouted email delays a shipment. After a prompt test, the labels still drift on mixed-language emails. Here a smaller fine-tuned model, trained on labelled emails, can be cheaper per call and steadier than a large model with a long prompt. The facts do not change; the behaviour is what needs work.
Fine-tuning does not always mean retraining the whole model. Parameter-efficient methods such as LoRA (Hu and colleagues, 2021) train small added matrices instead. The LoRA paper reports that, compared with full fine-tuning of GPT-3 175B, it can reduce trainable parameters by 10,000 times and GPU memory needs by three times. That lowers the training bill. It does not lower the cost of collecting good examples or retesting each release.
The maintenance cost of fine-tuning
A fine-tuned model is tied to its base model and to the platform that trained it. When either changes, you retrain or lose the model. Two current examples show the risk.
- OpenAI's deprecations page records that from 7 May 2026, organisations that had never run fine-tuning could no longer start. From 2 July 2026, job creation ended for organisations with no inference on a fine-tuned model in the previous 60 days. From 6 January 2027, active existing customers will no longer be able to create new fine-tuning jobs. Existing fine-tuned models keep running until their base model is shut down; for models built on GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini, OpenAI has set that shutdown for 14 January 2027.
- Amazon Bedrock's fine-tuning page, checked on 31 October 2026, still lists Claude 3 Haiku as the only Anthropic model that can be fine-tuned there, in the US West (Oregon) region. Bedrock's model lifecycle page shows that model reached end of life in that region on 10 September 2026, so the one Anthropic option on that list can no longer be used. The other models in that table are from Amazon (Nova and Titan) and Meta (Llama). Every region in the fine-tuning table is in the United States. For a Saudi or UAE organisation with rules on where its data is processed, that matters before any training data leaves the country.
How retrieval ages instead
Retrieval ages differently. When a better model appears, a RAG system can switch its answering model and rerun its evaluation set, while the documents and index stay. Changing the embedding model, the one that turns text into searchable vectors, is a bigger job, because the index must be rebuilt.
Deleting data follows the same pattern. If personal data sits in a training set, removing it means a new training run. In a RAG system, you delete the document from the index and from any caches, logs or stored copies, and it stops appearing in answers. This is general information, not legal advice; your obligations depend on your sector and the data involved.
RAG vs fine-tuning: a decision table
| Your need | Start with |
|---|---|
| Answers must follow documents that change monthly or faster | RAG |
| Users or auditors must see the source of each answer | RAG |
| Different staff may see different documents | RAG, with access rules applied at search time |
| A fixed format or label set drifts despite a tested prompt | Fine-tuning |
| One narrow task at high volume, where cost per call matters | Fine-tuning a smaller model |
| A house tone the prompt cannot hold | Fine-tuning, after a prompt test |
| Personal data you may have to delete on request | RAG, with that data kept out of training sets |
| The model does the task well with a good prompt | Neither; standardise the prompt |
Some systems need both: a fine-tuned model that keeps your output format, fed by retrieval that supplies current facts. We recommend adding the second method only after the first is measured, so you know which one moved the results. If you are weighing RAG vs fine-tuning for a single assistant, start with retrieval, because it is easier to inspect and to undo.
How to start
Pick one use case and write 30 to 50 real questions with agreed answers. Run a careful prompt against them, then decide from the failures rather than from a vendor demo. Our Transformation Lab builds retrieval platforms and custom applications under Custom AI Applications & Modernisation. If you are not sure the use case is ready, the AI Exposure Scan is a free 45-minute conversation that ends in a written next step.
Frequently asked questions
Is RAG cheaper than fine-tuning?
It depends on where the cost falls. RAG needs document cleaning, an index and someone to keep it current, and each call carries longer prompts. Fine-tuning needs labelled examples, training runs and retraining when the base model changes, but a smaller tuned model can cost less per call. For answers that depend on changing documents, RAG tends to cost less over time because updates do not need a training run.
Can you use RAG and fine-tuning together?
Yes. A fine-tuned model can hold a fixed output format or house style while retrieval supplies current facts from your documents. We recommend building one method first, measuring it against an agreed set of real questions, and adding the second only if the results show a gap it can close. Otherwise you cannot tell which change improved the answers.
Does fine-tuning stop a model from making things up?
No. Fine-tuning changes how a model behaves, but it does not tie answers to a document you can check. A RAG system can show the passage behind each answer and can be told to refuse when no passage supports the question. Neither method removes errors entirely, so test both against real questions with agreed answers before staff rely on the output.
Which is better for Arabic documents, RAG or fine-tuning?
For questions answered from Arabic policies, manuals or circulars, start with RAG, because the documents change and users need to see the source. The hard work is in the pipeline: OCR quality on scanned pages, spelling normalisation and search that handles Arabic and English together. Fine-tuning fits narrower Arabic tasks, such as sorting messages into fixed categories, once a prompt test has failed.
Get new insights by email
One email when we publish something useful. No spam; leave any time.
Services in this article
Related lab: Transformation Lab ↗
