Fine-Tuning vs RAG vs Prompting: Which Lever When
The decision framework for customizing model behavior, with honest costs for each path.
Prompting, retrieval-augmented generation (RAG) and fine-tuning are the three levers you can pull to make an LLM (large language model) do what you need. Prompting changes what you send the model, without changing the model. RAG looks up relevant passages from your own documents and adds them to the prompt. Fine-tuning trains the model further on your own examples, so its weights change. The right choice depends on what is actually going wrong: is the model missing knowledge, or is it behaving the wrong way?
Most production systems combine these levers. The usual order is a good default: prompt first, add RAG for missing knowledge, and fine-tune only for behavior that prompts cannot hold. This playbook shows how to tell which failure you have, what each fix costs, and walks through an example. For how each lever works inside, it links to the articles that teach them.
Think of an LLM as a skilled general contractor. Prompting is the instruction you give on the job site: what to build and how. RAG is handing them the right blueprints and regulations for this job. Fine-tuning is sending them on a course so that your company's way of working becomes habit. Many reliable projects use more than one.
How does prompting steer a fixed model?
The model's weights never change when you prompt it. Your instructions and examples simply give it more context to predict from. Few-shot examples work because the model has seen countless documents in which a pattern is shown and then continued, so a few input and output pairs in the prompt make the next output follow the same pattern. Few-Shot: Examples Beat Instructions goes deeper.
Prompting works best when the model already knows how to do the task and you only need to steer it: tone, output format, the steps to follow. It has two hard limits. First, it cannot add knowledge the model never saw. If your refund policy was never in its training data, the best prompt in the world produces a plausible guess, not your policy. Second, everything you add costs tokens on every request: long instructions and many examples add cost and waiting time, although prompt caching makes a repeated prefix much cheaper, as Why RAG Exists: The Context Problem explains. Very long prompts can also be fragile, with a small wording change shifting the behavior.
How does fine-tuning reshape the model's behavior?
Fine-tuning continues training on your own examples of inputs and ideal outputs, so the desired behavior sticks without long prompts. The common cheap method, LoRA (low-rank adaptation), keeps the original weights frozen and trains two thin matrices whose product is a small correction added to them; only those thin matrices are trained (Hu et al. 2021, LoRA: Low-Rank Adaptation of Large Language Models). Fine-Tuning, Explained: Teaching an Old Model New Tricks covers how it works, what it costs, and the forgetting it can cause.
Fine-tuning is the right lever when the gap is behavior or style rather than facts, and when prompts cannot hold that behavior consistently. It can also let a smaller model do a narrow job that would otherwise need a larger one. In one study, a fine-tuned T5 model of 770 million parameters outperformed the 540 billion parameter PaLM prompted with a few examples, on a benchmark, using 80 percent of the available training data (Hsieh et al. 2023, Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes).
The trade-off is maintenance and access. Facts baked into weights go stale, so changing knowledge belongs in RAG, not in a training run. And access to fine-tuning closed models has shrunk: as of October 2026, OpenAI is winding down its fine-tuning platform and no longer accepts new users (OpenAI's supervised fine-tuning guide), Anthropic's API does not offer fine-tuning (Anthropic's glossary), and Google offers supervised tuning for some Gemini models on its cloud platform rather than in the Gemini API (Google Cloud's Gemini tuning page). Most fine-tuning today happens on open-weight models.
How does RAG inject external knowledge at inference?
RAG leaves the model untouched. For each question, a search step finds the most relevant passages in your documents and adds them to the prompt, and the model answers from them. In the original RAG paper, the answer was generated from the question plus the top few retrieved passages, as a weighted mix over those passages (Lewis et al. 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks).
A typical pipeline has five stages: split documents into chunks and embed them ahead of time, embed the question, retrieve the closest chunks, add them to the prompt with their sources, and generate the answer. RAG, Properly builds every stage in code, from chunking to an evaluation that catches regressions.
RAG keeps knowledge separate from the model. When a document changes, you re-process that document, not the model, which makes RAG the natural lever for up-to-date, private or verifiable information. Because each answer can cite the passage it came from, it also makes answers checkable.
The cost is latency and complexity. Each question needs an embedding of the question and a search, plus a re-ranking step if you add one. On a laptop CPU, we measured about 13 milliseconds to embed a question with a small open model and about 4 milliseconds to search 50,000 chunks exactly, while re-ranking 20 candidates with a small cross-encoder took about 0.2 seconds. Indexing ran at about 110 chunks a second, so re-embedding tens of thousands of chunks takes minutes. Your numbers will differ, so measure them. And retrieval can fail: if the right passage is not retrieved, no prompt recovers it.
How do prompting, fine-tuning, and RAG compare across the dimensions that matter?
| Dimension | Prompting | Fine-Tuning | RAG |
|---|---|---|---|
| What changes | The input you send | The model's weights, or an adapter added to them | Passages from your documents added to the input |
| Knowledge recency | Fixed at the model's training cutoff | Fixed at the time of the last training run | As fresh as your index |
| Latency added | Only the extra prompt tokens | None from training; the model runs as before | An embedding call and a search, plus re-ranking if used |
| Data needed | None beyond the prompt and a few examples | Tens of examples to start; OpenAI's guide requires at least 10 and suggests starting with about 50 | Any document collection; if it is small enough to fit in the prompt, you may not need retrieval at all |
| Update cadence | Instant: change the prompt | A new training run | Re-process the changed documents |
| Best for | Steering a model that already knows the task: format, tone, steps | A stable behavior or style that prompts cannot hold; replacing long prompts; a smaller model for a narrow job | Facts that change, private documents, answers that cite sources |
| Failure mode | Fragile long prompts; knowledge the model lacks | Stale facts; forgetting other skills; data that does not match real inputs | Retrieval misses; misleading passages; answers that stray from the sources |
When should you combine them?
Most reliable systems layer them. The prompt sets the task, the rules and the output format. RAG supplies fresh evidence for each question. Fine-tuning, when it is needed at all, makes a behavior consistent so the prompt can stay short.
OpenAI's guide to optimizing LLM accuracy frames this as a matrix with two directions (OpenAI's guide to optimizing LLM accuracy). If the model lacks knowledge it needs, you optimize the context, which is RAG's job. If it produces inconsistent results or the wrong format, you optimize the model, which is where fine-tuning comes in. A typical project starts with prompt engineering and a set of evaluation questions with known answers, finds where the failures fall, applies the matching lever, and evaluates again.
Hybrid patterns are common. A model fine-tuned to follow a strict answer format can sit on top of a RAG pipeline that supplies the facts. A re-ranker fine-tuned on your documents can improve retrieval while the main model stays untouched. The levers work together.
The trap to avoid is using fine-tuning as a substitute for retrieval. A model fine-tuned on a snapshot of your knowledge base will drift out of date, and it cannot point to a source. Let RAG carry the facts and, where needed, let fine-tuning carry the behavior: how to read, summarize and format those facts.
A worked example: one support bot, three failures
An online store's support bot gets three complaints in its first week:
- It told a customer they could return a sale item within 30 days. The policy changed last month: sale items are final.
- Its replies are supposed to be JSON for the website to display, but about one reply in twenty is broken JSON.
- Its tone is curt, while the store wants warm and brief.
Sort each by kind. The first is missing knowledge: the model cannot know a policy that changed last month. Fix it with RAG over the current policy documents, and cite the policy in the answer so support staff can check it. No prompt or fine-tune will keep up with policy changes.
The second is a format behavior. First use the API's structured output feature, which constrains the reply to your schema, as Structured Outputs: JSON You Can Actually Parse explains, and validate the JSON in code with a retry. Fine-tuning would be a later resort if a model you must use cannot follow the format any other way.
The third is behavior too, but cheap to try with a prompt: describe the voice in the system prompt and show two or three example replies. If, after an evaluation set of a few dozen real questions, the tone still drifts and the prompt has grown long, that is when fine-tuning on a few hundred replies in the right voice starts to pay. With closed-model fine-tuning now scarce, that may mean an open-weight model.
Frequently Asked Questions
Q: Can I skip fine-tuning entirely and rely on RAG plus prompting? For most teams, yes. RAG handles knowledge, and good prompts plus structured outputs handle most behavior. You may need longer prompts to enforce a style, and consistency across very different phrasings can be harder, so measure on your own questions before deciding you need fine-tuning.
Q: When does fine-tuning reduce inference cost? When it lets you remove long instructions and examples from every request, or switch to a smaller model that still does the job. Weigh that against the cost of training, evaluating and serving your own model version.
Q: How many examples do I need for fine-tuning? It depends on the task. OpenAI's guide requires at least 10 examples, recommends starting with about 50 well-crafted ones and reports improvements from 50 to 100. Teaching general instruction following takes more: the LIMA study used 1,000 carefully chosen examples (Zhou et al. 2023, LIMA: Less Is More for Alignment). Quality and variety matter more than raw count.
Q: Why does RAG sometimes give a wrong answer despite retrieving the correct document? The model may not use the passage well, especially if many passages are joined together and the right one sits in the middle. It may also summarize inaccurately or mix in what it remembers from training. Tell it to use only the retrieved passages, label each passage with an ID, and check its citations in code, as Grounding and Citations: Answers You Can Check shows.
Q: Can I use LoRA adapters alongside RAG without conflicts? Yes. A LoRA adapter typically controls style or format, while RAG provides the facts. The adapter sees the full prompt, retrieved passages included, so make sure its training examples include prompts with retrieved passages; otherwise it learns from inputs that look different from what it sees in use.
Test yourself
You run an internal assistant for a hospital's HR department. Three problems appear: (1) it quotes last year's shift allowance rates, which changed in January; (2) it answers in long paragraphs when staff want two-line answers on their phones; (3) it sometimes calls a policy "mandatory" when the document says "recommended". For each, name the lever you would try first and why.
Answer: (1) Outdated rates are missing knowledge. Use RAG over the current HR documents, re-index when policies change, and cite the source passage so staff can check it. Fine-tuning would freeze this year's rates into the weights and go stale again. (2) Long answers are a behavior problem, and the cheapest fix is the prompt: ask for two lines, show a couple of examples, and cap the output length. Fine-tune only if an evaluation shows the prompt cannot hold it. (3) Turning "recommended" into "mandatory" is a grounding failure: the model strays from its source. Ask for a verbatim quote with each claim and check the quotes in code, so a claim the document does not support is caught before staff see it. If retrieval is pulling the wrong passage, improve retrieval too.
If you found this deep dive on how to actually choose between prompting, fine-tuning, and RAG useful, not just the usual platitudes, you’ll want to subscribe to Internals Decoded. Every week we break down real systems internals, from vector databases to LLM training loops, with the same level of detail. Join at internalsdecoded.com.
Sources
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al. 2021)
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (Hsieh et al. 2023)
- Supervised fine-tuning (OpenAI, read October 2026)
- Glossary (Anthropic)
- Supervised fine-tuning for Gemini models (Google Cloud, read October 2026)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al. 2020)
- Optimizing LLM accuracy (OpenAI, read October 2026)
- LIMA: Less Is More for Alignment (Zhou et al. 2023)
- LoRA (Hugging Face PEFT documentation)
