Fine-Tuning vs RAG vs Prompt Engineering: When to Use Which

A practical decision framework for choosing the right LLM customization approach.

Three main ways to adapt LLMs to your needs. Each has different cost, complexity, and effectiveness. This guide helps you choose.

Crafting input instructions to guide model behavior without changing the model.

# Zero-shot " Write a SQL query for... " # Few-shot " Examples: \n Input:... \n Output:... \n\n Now:... " # Chain-of-thought " Think step by step:... " # Structured output " Return ONLY valid JSON: {...} " # Role + Context " You are a senior engineer. Context:... Task:... " Enter fullscreen mode Exit fullscreen mode Pros/Cons ✅ Pros ❌ Cons Instant iteration Limited by context window No training cost Can't learn new facts Works on any model Inconsistent on complex tasks Version controllable Prompt injection risk Cost API calls only: ~$0.001-0.01 per request No infrastructure Approach 2: RAG (Retrieval-Augmented Generation) What It Is Ground model responses in your documents by retrieving relevant chunks at query time.

Query → Embed → Vector Search → Top-K Chunks → LLM + Context → Answer Enter fullscreen mode Exit fullscreen mode When to Use Large document collections (100+ docs) Frequently changing data Need citations/sources Regulatory compliance (audit trail) Multiple domains in one system Implementation Levels Level 1: Basic (Chroma + OpenAI) # 50 lines, runs locally import chromadb from chromadb.utils import embedding_functions ef = embedding_functions. SentenceTransformerEmbeddingFunction ( " all-MiniLM-L6-v2 " ) client = chromadb. PersistentClient ( "./db " ) col = client. create_collection ( " docs ", embedding_function = ef ) col. add ( documents = texts, metadatas = metas, ids = ids ) results = col. query ( query_texts = [ question ], n_results = 5 ) Enter fullscreen mode Exit fullscreen mode Level 2: Production (Qdrant + Reranker) # Hybrid search + cross-encoder reranking from qdrant_client import QdrantClient from sentence_transformers import CrossEncoder client = QdrantClient ( url = "...

", api_key = "... " ) reranker = CrossEncoder ( " cross-encoder/ms-marco-MiniLM-L-6-v2 " ) # Retrieve 20, rerank to 5 results = client. query_points ( collection, query, limit = 20 ) pairs = [( q, r. payload [ " text " ]) for r in results ] scores = reranker. predict ( pairs ) top5 = sorted ( zip ( results, scores ), key = lambda x: x [ 1 ], reverse = True )[: 5 ] Enter fullscreen mode Exit fullscreen mode Level 3: Advanced (Agentic RAG) # Multi-hop reasoning, tool use # Query → Decompose → Retrieve per subquery → Synthesize Enter fullscreen mode Exit fullscreen mode Pros/Cons ✅ Pros ❌ Cons Grounded in your data Added latency (retrieval) Easy updates (re-index) Chunking strategy matters Citations built-in Retrieval can miss context No training needed Embedding quality critical Works with any LLM Context window limits Cost Component Monthly (1M vectors) Vector DB (Qdrant Cloud) $0-150 Embeddings (OpenRouter) $0-20 LLM calls $10-100 Total $10-270 Approach 3: Fine-Tuning What It Is Update model weights on your data to bake in knowledge, style, or behavior.

// Alpaca format (instruction tuning) { "instruction": "Write a Python function", "input": "calculate fibonacci", "output": "def fib(n):..." } // Chat format { "messages": [ { "role": "system", "content": "You are a Python expert" }, { "role": "user", "content": "Write a fibonacci function" }, { "role": "assistant", "content": "def fib(n):..." } ]} // Preference pairs (DPO) { "prompt": "...", "chosen": "better response", "rejected": "worse response" } Enter fullscreen mode Exit fullscreen mode Minimums:

# 1. Prepare data python prepare_data.py --input raw.jsonl --output train.jsonl --val-size 100 # 2. Train (4-bit, 7B model, 1x A100 40GB) python train.py \ --model meta-llama/Llama-3.1-8B-Instruct \ --data train.jsonl \ --output./lora-adapter \ --lora-r 64 --lora-alpha 16 \ --batch-size 2 --grad-accum 4 \ --lr 2e-4 --steps 500 \ --quantization 4bit # 3.

Merge & export python merge.py --base meta-llama/Llama-3.1-8B-Instruct \ --adapter./lora-adapter --output./merged-model # 4. Convert to GGUF python convert-hf-to-gguf.py./merged-model --outfile model.q4_k_m.gguf Enter fullscreen mode Exit fullscreen mode Pros/Cons ✅ Pros ❌ Cons Best task performance High upfront cost No retrieval latency Hard to update Works offline Catastrophic forgetting risk Bakes in style/format Needs ML expertise Single artifact to deploy GPU required Cost Scale Hardware Time Est. Cost 7B QLoRA 1× A100 40GB 2-4 hrs $10-30 13B QLoRA 1× A100 80GB 4-8 hrs $40-80 70B QLoRA 4× A100 80GB 8-24 hrs $300-1000 7B Full FT 8× A100 80GB 12-48 hrs $1000-5000 Decision Framework START: What's your primary need? │ ├─→ "Control output format/style" │ └─→ **Prompt Engineering** (try first, 15 min) │ ├─→ "Answer from my documents/data" │ ├─→ Data changes frequently? │ │ ├─→ Yes → **RAG** (re-index on change) │ │ └─→ No → Continue │ ├─→ Need citations? │ │ └─→ Yes → **RAG** │ ├─→ < 100 docs? │ │ └─→ Yes → **RAG** (Chroma local) │ └─→ > 10K docs, high query volume? │ └─→ **RAG** (Qdrant/Milvus) or **Fine-tune** if style critical │ ├─→ "Model must know domain deeply" │ ├─→ Can you get 500+ quality examples? │ │ ├─→ Yes → **Fine-tune (QLoRA)** │ │ └─→ No → **RAG + few-shot prompts** │ ├─→ Need offline/air-gapped? │ │ └─→ Yes → **Fine-tune** (deploy GGUF) │ └─→ High volume (>10K req/day)? │ └─→ **Fine-tune** (cheaper at scale) │ └─→ "Reduce hallucinations on facts" ├─→ Facts in documents? │ └─→ Yes → **RAG** └─→ Facts = training knowledge? └─→ **Fine-tune** or better prompts Enter fullscreen mode Exit fullscreen mode Hybrid Approaches (Real World) 1.

RAG + Fine-Tuned Embedder # Fine-tune embedder on your domain, use with RAG # Better retrieval → better answers Enter fullscreen mode Exit fullscreen mode 2. RAG + Fine-Tuned Generator # Fine-tune small model (7B) for style/format # Use RAG for knowledge # Best of both worlds Enter fullscreen mode Exit fullscreen mode 3. Prompt → RAG → Fine-Tune (Progression) Week 1: Prompt engineering (validate concept) Week 2: Add RAG (ground in data) Week 3: Fine-tune if needed (optimize) Enter fullscreen mode Exit fullscreen mode 4.

Router Pattern # Route queries to best approach def route ( query ): if is_factual ( query ): return rag_chain elif is_creative ( query ): return prompt_chain elif is_coding ( query ): return finetuned_coder else: return default_chain Enter fullscreen mode Exit fullscreen mode Cost Comparison at Scale Approach 1K req/mo 100K req/mo 1M req/mo Prompt (API) $5 $500 $5,000 RAG (managed) $30 $300 $2,000 Fine-tuned (self-host) $100* $100* $100* *Fixed GPU cost (A100) (A100) (4×A100) Break-even: Fine-tuning pays off at ~50K req/mo if you own GPUs.

Start with prompt engineering — 15 minutes, validates the task.

Add RAG when you have documents and need grounded answers — hours to set up.

Fine-tune only when: you have 500+ examples, need consistent style/format, high volume, or offline deployment.

Most production systems use hybrids: RAG for knowledge + fine-tuned model for behavior.

Part of the LLM Customization series on Dev.to