RAG vs fine-tuning vs MCP: which one do you need?
"Should we fine-tune?" is usually the wrong first question. The right first question is: what is the model missing? Knowledge, behavior, or access? Each has a different fix.
The three problems#
- It does not know the facts. (Your docs, last week's data.) → give it knowledge: RAG.
- It knows what to do but does it inconsistently. (Tone, format, a narrow skill.) → change behavior: fine-tuning.
- It must look something up live or do something. (Query a DB, create a ticket.) → give it access: tools via MCP.
A decision flow#
Side by side#
| RAG | Fine-tuning | MCP / tools | |
|---|---|---|---|
| Solves | Missing knowledge | Inconsistent behavior | Missing access or actions |
| How | Retrieve text into the prompt | Train on examples, change weights | Model calls functions on servers |
| Data freshness | Update the index, instant | Retrain to update | Live every call |
| Setup cost | Medium | High (data, training, evals) | Low to medium |
| Cost per update | Low | High | None |
| Cites sources | Yes, easily | No | Depends on the tool |
| Typical failure | Wrong chunk retrieved | Overfit, stale knowledge | Wrong tool, unsafe action |
| Reversible | Yes | Retrain | Yes |
Common misconceptions#
- "Fine-tuning teaches the model our docs." It can absorb patterns, but it is an unreliable way to store facts and impossible to update cheaply. Use RAG for facts.
- "RAG replaces tools." RAG reads static text. For a live balance or creating a record, you need a tool.
- "MCP is a different thing from tools." MCP standardizes how tool servers are described and called. See MCP vs API.
- "We need all three." Most products need one or two.
Combinations that work#
- Support bot: RAG over help articles + a tool to look up the customer's order.
- Coding agent: repo search and file reading as tools, rules in a CLAUDE.md, no fine-tuning.
- Brand-voice writer: a prompt with examples first; fine-tune only if that is not consistent enough.
- Data analyst: SQL tools over MCP, the schema as retrieved context.
What to do first#
- Write 20 real test cases.
- Try the best prompt you can, with examples.
- See which cases fail and why (missing fact, wrong style, no access).
- Add the matching piece and re-run the cases.
That loop is evals, and it keeps you from fine-tuning to fix a retrieval bug. Details on the pieces: what is RAG and tool use explained.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG retrieves documents at question time and puts them in the prompt, so it adds up-to-date knowledge. Fine-tuning changes the model's weights with training examples, so it shapes behavior, style and format rather than adding facts reliably.
Is MCP an alternative to RAG?
No. MCP is a protocol for connecting a model to tools and data sources. A RAG search can be exposed as an MCP tool, so the two work together.
When should I fine-tune an LLM?
When prompting and examples cannot give you the consistent format, tone or narrow skill you need, you have good training data, and the task is stable enough to justify the cost.
Should I start with RAG or fine-tuning?
Start with prompting, then add retrieval or tools. Fine-tuning is usually the last step because it is the most expensive to build and to change.
Prefer plain text? Read this page as Markdown.