Advanced12 min readAI for Business

Choosing between prompting, RAG, and fine-tuning (and when to combine)

Diagnose whether an LLM task needs clearer instructions, current source context or a training experiment. Compare quality, access requirements and total operating cost before combining methods.

What you should be able to do

Prompting changes what the model is asked. RAG supplies retrievable evidence at inference time. Fine-tuning adapts model behavior on a measured task; it is not a maintainable factual database. Choose from the failure you can demonstrate, and combine methods only when evaluation justifies the added system.

Saved only in this browser.
In this article

A team gets the brief: “Make our AI better at handling our specific use case.” They have a few choices on how to do it. They can write better prompts. They can build a RAG system to feed the model relevant data. They can fine-tune the model on examples of their domain.

These are not interchangeable. They solve different problems. Done wrong, you spend three months and a budget fine-tuning when the answer was a better prompt. Or building elaborate RAG infrastructure when fine-tuning would have been simpler. Or stuck on prompts when the model fundamentally can’t do what you need.

Here is a framework for choosing and, more importantly, for running a fair comparison. It does not publish universal project costs or accuracy gains. Primary references include the original RAG paper, OpenAI’s current fine-tuning guide, and Hugging Face PEFT documentation.

What each actually does

A clean distinction:

Prompting changes what the model is asked. You give the model better instructions, examples, format requirements, context. The model itself doesn’t change; the input does.

RAG changes what data the model sees. You retrieve relevant information at query time and include it in the prompt. The model has fresh, specific, dynamic data without being trained on it.

Fine-tuning changes model weights to adapt behavior on a measured task. It may affect style, format, task performance, and memorized associations. It is not a reliable or maintainable way to install a current factual database.

These solve different problems:

  • Instructional gap: a clearer task contract may help. → Test prompting.
  • Evidence gap: the request lacks needed information. → Test direct context, retrieval or an authoritative tool.
  • Remaining task-performance gap: the model fails despite the tested prompt and context. → Compare another model, tool, task redesign or fine-tuning when appropriate.

Knowing which gap you have is half the battle.

Diagnosing the gap

Do not infer model capability from what a capable person could do with the same prompt. Inspect failures on a representative set, then intervene on one factor. A better instruction, supplied reference or different model is an experiment, not a guaranteed fix.

Consider an invented returns assistant. Current policy P2 permits returns within 14 days but excludes personalised goods. Historical examples used an obsolete 30-day policy. Three failures suggest different tests:

Observed failureControlled testWhat the result would establish
Answers 30 days when P2 is absentSupply current P2 directly, with model and prompt otherwise fixedWhether the missing evidence explains this case
Says all items qualify despite P2’s exclusionKeep P2; revise extraction/output contractWhether clearer handling of conditions helps
Changes confirmed policy content while applying the desired toneCompare prompt/style examples with a tuned candidate on frozen factsWhether adaptation improves style without factual regression

If direct P2 context works but retrieval selects P1, investigate the source/version retrieval path. If both fail, retrieval alone has not solved the problem. Neither result yet proves a fine-tune is needed. An incorrect access decision belongs in application authorisation, not in a training dataset as the sole remedy.

Prompting: the underrated lever

A prompt baseline can be inexpensive to construct when the model already has the needed context and tools. Measure the complete evaluation cost rather than assuming prompting always wins.

A few things you can do with prompting alone:

  • Change tone, format, length.
  • Apply reasoning patterns (chain-of-thought, self-critique).
  • Encode constraints (do X, don’t do Y).
  • Encode policies and guardrails.
  • Adapt to specific use cases (different prompts for different features).
  • Improve consistency through few-shot examples.

What you can’t do with prompting alone:

  • Create reliable evidence absent from both supplied context and available sources. Prompting can supply new facts, but cannot make missing evidence appear.
  • Guarantee a capability the model cannot demonstrate on the task.
  • Guarantee factuality or authorisation through wording alone.

Shorter inputs and outputs can change latency and cost, and good prompts can change style substantially. Compare those effects rather than treating them as impossible without training.

A reasonable first experiment is the lowest-cost prompt baseline. Time-box it by number of evaluated variants and a stopping rule, not by an arbitrary week. Move on when additional prompt changes no longer improve the held-out metric or violate another requirement.

Prompt engineering effort

Run prompt variants against the same train/development split and preserve a held-out test set. Change one design dimension at a time (task contract, examples, context, output schema, or tool set) and record quality, latency, and cost. Stop when the predeclared acceptance bar is met or the next change fails to improve the development set. A calendar duration cannot prove that prompting is exhausted.

What good prompts look like

Candidate prompt elements to test include:

  • Clear role and task.
  • Specific format requirements.
  • Representative examples when an eval shows they help.
  • Explicit constraints (what to do, what not to do).
  • Edge case handling.
  • Output schema.

Use the shortest prompt that preserves the measured behavior and necessary constraints. Paragraph count is not a quality metric.

RAG: the knowledge fix

RAG is a candidate when:

  • The model needs factual information it doesn’t have.
  • The information changes (live data, recent events, account-specific data).
  • The information is specific to your domain or organization.
  • You need inspectable sources and can test whether answers are actually supported by them.

It’s the wrong tool when:

  • The problem is instructional, not knowledge.
  • A small, permitted and current source pack can be supplied directly with acceptable cost and quality. Test that simpler baseline first; selective retrieval may still be justified by permissions, freshness or request-specific scope.
  • You need the model to do something differently, not just know something different.

The realistic cost

A RAG system is real engineering:

  • Build: ingestion, permissions, chunking, indexing, retrieval, evaluation, and operations (estimate from your sources and acceptance criteria).
  • Operate: ongoing: keeping the index updated, monitoring quality, fixing issues.
  • Infrastructure: parsing/OCR, storage, embedding, retrieval, reranking, generation, backups, and telemetry (price them from dated quotes and measured volumes).
  • Per-query cost: may add query embedding, retrieval, reranking, and context tokens, but can also reduce wasted context or enable a cheaper generator. Measure the complete path.

Approve the investment only when the evaluated retrieval path improves the defined outcome enough to justify ingestion, permission, freshness, latency, cost, and operational work.

RAG quality is a journey

There is no transferable “week-one quality” percentage. Baseline lexical, vector, and hybrid retrieval on a labeled set; report recall and final-answer metrics by slice; then change one stage at a time. Ship only after the business-specific error and permission checks pass.

Fine-tuning: when prompting and RAG aren’t enough

Fine-tuning is the right tool when:

  • You have a clear capability gap; the model can’t reliably do the task even with good prompts and context.
  • You have enough representative, licensed, privacy-reviewed training data to improve held-out performance. Determine sufficiency with a learning curve, not a universal example count.
  • You need consistent, narrow behavior (a specific style, a specific output format, a specific domain).
  • Inference cost / latency matters (a fine-tuned smaller model can be cheaper than a generic larger one).

It’s the wrong tool when:

  • Your data is in flux (the fine-tuned model will be stale fast).
  • You have not built the relevant prompt, constrained-output, retrieval, or tool baseline for comparison.
  • You don’t have good evals (you can’t tell if fine-tuning helped).
  • The task needs very current information (fine-tuning is a snapshot).
  • The task primarily needs current, attributable facts; retrieval or authoritative tools are easier to update and cite than weight changes, although they still require evaluation.

Types of fine-tuning

Full fine-tuning: all model weights are eligible for update. It has the largest trainable surface and compute/storage burden, but does not inherently produce the best held-out result.

LoRA (Low-Rank Adaptation): trains low-rank adapters while the base weights remain frozen, reducing trainable parameters and commonly reducing optimizer-memory needs. Compare quality and serving support with full tuning for the actual task.

QLoRA: backpropagates through a frozen quantized base model into LoRA adapters. It can reduce memory requirements; quality is empirical, not a fixed “lower at scale” rule.

Prompt tuning / prefix tuning: trains continuous prompt or prefix parameters. Hardware, provider support, task quality, and serving integration determine whether the smaller trainable surface is useful.

Instruction tuning: supervised adaptation on instruction-response examples. It is common in model development and can also be an application-team experiment when the model licence, data, infrastructure, and evals support it.

RLHF / DPO / KTO: different preference-optimization families; they are not interchangeable and do not all consume identical data. Use the pinned implementation’s primary paper and documentation, then test reward/preference quality and safety regressions.

Parameter-efficient tuning is a useful candidate when full tuning does not fit the hardware or iteration budget. This article has no representative survey proving what most teams use or a universal best balance.

The realistic cost

Fine-tuning cost and duration depend on base model, sequence lengths, tokens, epochs, hardware, distributed strategy, failures, and evaluation. Build the estimate from a small instrumented run:

  • Data prep: provenance, permission, deduplication, privacy review, formatting, and quality labeling.
  • Training: planned training tokens divided by measured training tokens per second gives estimated training duration. Multiply duration by the hardware rate, then add checkpointing, failed runs, storage and other overhead.
  • Evaluation: base versus tuned model on held-out target, safety, and general-capability sets.
  • Iteration: budget only after defining what result would justify another run.
  • Deployment: provider availability, adapter serving, rollback, monitoring, and retention obligations.
  • Maintenance: retraining when data updates, when the base model updates, when the use case shifts.

Include engineering and reviewer time; compute alone is not total cost. Proceed only when the measured improvement is worth the continuing serving and retraining burden.

When fine-tuning shines

Scenarios where a fine-tuning experiment may be justified:

Stable task behavior. Fine-tuning may improve format or style on a particular distribution, but constrained decoding can already guarantee supported schema shapes. Compare prompt-only, constrained-output, and tuned baselines rather than promising 95% versus 99%.

Specialized language or syntax. Medical terminology, legal phrasing, or an internal DSL may improve on held-out examples, but high-stakes correctness still needs external validation, current evidence, and qualified review.

Style/voice. A tuned model may improve a scored style rubric on the target distribution. It does not “bake in” perfect consistency; measure content quality, safety, and drift as well as voice.

Latency/cost experiment. A smaller tuned model may clear the same acceptance bar with lower measured serving cost or latency. Include training, evaluation, capacity, reliability, and maintenance before claiming a payoff.

Behavioral adaptation. Safety tuning may improve a measured refusal behavior, but it can also over-refuse, under-refuse, or regress other capabilities. It must remain one layer beside deterministic authorization, policy enforcement, monitoring, and human gates.

When fine-tuning fails

Common ways fine-tuning disappoints:

Insufficient or unrepresentative data. Small, narrow datasets sometimes help and large noisy datasets sometimes hurt. Plot held-out performance as training data increases and inspect slice coverage before buying more compute.

Bad data. Garbage in, garbage out. Inconsistent, low-quality examples produce inconsistent, low-quality models.

Capability regression. Tuning can degrade abilities outside the target task. Test the relevant non-target and safety slices rather than assuming every capability changes in the same direction.

Stale learned knowledge. Updating information embedded in model weights may require another training run. A tuned model can still receive current facts through prompts, retrieval or tools; use those paths when they fit the task rather than retraining for every factual update.

Base model improvements outpace fine-tuning. The base model improved enough that the fine-tune is no longer better. You’re now maintaining a fine-tune of an outdated base.

Evaluation problems. Without a held-out baseline and leakage controls, apparent gains may reflect memorisation or changes to the test rather than improvement on the target task.

The combination patterns

Prompted retrieval is a candidate for current, attributable knowledge. Adding a fine-tune may help a demonstrated behaviour such as style or extraction, while sources remain externally governed. Multiple tuned components also multiply deployment, evaluation and retirement work. Require an incremental comparison for each added component; a plausible architecture diagram does not establish a benefit.

The decision framework

A practical decision flow:

Question 1: Is the problem solvable with the current model and a good prompt?

If yes: build and evaluate a prompt baseline. Ship only if the acceptance and safety criteria pass.

If no, go to Question 2.

Question 2: Does the problem involve knowledge the model doesn’t have?

If yes: compare direct context, lexical retrieval, vector retrieval, and hybrid retrieval as appropriate. Estimate the implementation after source and permission discovery.

If no, go to Question 3.

Question 3: Is the problem about consistent format, narrow domain, or specific behavior?

If yes, and you have representative data with the necessary rights and privacy controls: run a small fine-tuning experiment and compare it with the unchanged baseline.

If you don’t have the examples: invest in collecting them, OR try better prompting / RAG further before fine-tuning.

Question 4: Have you done the eval work to know which approach actually helps?

This question applies at every step. Without evals, you’re guessing.

Cost and effort comparison

A comparison worksheet: fill it with measured or quoted values for the same acceptance bar:

ApproachEffortCost (one-time)Cost (per query)Maintenance
Promptingvariants × eval runtime + reviewmodel calls + reviewer timemeasured tokens/toolsprompt/model/eval updates
RAGsource discovery + ACL + pipeline + evalingestion, index, engineeringretrieval + rerank + generationsource sync, permissions, eval
Fine-tuning (LoRA)data + training + eval + servingrights review, compute, engineeringserving hardware/providerdata, base model, safety eval
Prompting + RAGcombined critical pathcombined, less shared workmeasured full traceprompt, sources, retrieval, eval
All threecombined critical pathcombined, less shared workroute-specifichighest operational surface

The right choice depends on the measured gap and the complete operating model. Prompting plus retrieval is common for dynamic knowledge, but it is not a universal sweet spot.

Pick the gap, then the lever

Use the adaptation decision record to preserve the failed cases, baseline configuration and one-factor comparisons. Include source permissions, freshness, critical-slice failures, complete operating cost and rollback. Keep “no model-based automation” available if none of the candidates meets the required bar.

Read next

Continue through the same learning path with the next practical articles.