Chapter 4 of 8
ML and LLM theory with judgment
The theory round has a trap built into it. It sounds like it is testing whether you can define terms, and if you answer by reciting definitions you fail. It is actually testing judgment: whether you know when a technique is the wrong choice. The most common form is the classic triage question, "You need the model to do X, would you use RAG, fine-tuning, or prompting, and why?" This chapter's lab encodes that decision as real code you can defend. The senior answer follows a clear rule. Prompting first, because it is the cheapest and fastest and a capable base model plus a few examples handles a surprising amount. RAG when the task needs knowledge the model does not have, especially fresh, private, or frequently-changing facts, because you retrieve them at query time instead of baking them into weights. Fine-tuning when the task needs a new behavior, format, or tone (not new facts) and you have enough labeled examples to teach it. A real follow-up they use to catch reciters: "When would you NOT fine-tune?" The answer: when the thing you need is current knowledge (fine-tuning bakes in a snapshot that goes stale), or when you have only a few dozen examples (too little to teach without overfitting). Being able to say when NOT to reach for a tool is the whole signal this round is looking for.
The lab: read it, then run it
#!/usr/bin/env python3
"""
LAB IP4: RAG vs fine-tune vs prompt, the decision drill.
The theory round rarely asks you to recite definitions. It asks for judgment:
"Here is a scenario. Would you use RAG, fine-tuning, or just prompting, and
why?" Reciting what each one IS is a failure signal. Knowing WHEN to reach for
each is the senior answer. The rule of thumb the strong candidates give:
- PROMPT when the task is achievable with instructions and a few examples
on a capable base model. Cheapest, fastest, try it first.
- RAG when the task needs knowledge the model was not trained on,
especially knowledge that is fresh, private, or changes often.
You retrieve facts at query time instead of baking them in.
- FINE-TUNE when the task needs a new BEHAVIOR, format, tone, or narrow
skill (not new facts), and you have enough labeled examples to
teach it. You change the model's weights, not its context.
You encode that judgment as a decision function and PROVE it routes a table of
real scenarios the way an interviewer would expect.
Run: python3 modules/academy-content/labs/interview-prep/ip4-rag-vs-finetune-decision.py
"""
import sys
def decide(scenario):
"""Return 'prompt', 'rag', or 'finetune' for a scenario described by flags.
Priority order matters and reflects real practice: reach for the cheapest
tool that actually fits. Knowledge needs point to RAG; behavior/style needs
with training data point to fine-tuning; everything else is prompting."""
needs_fresh_or_private_facts = scenario.get("needs_fresh_or_private_facts", False)
needs_new_behavior_or_format = scenario.get("needs_new_behavior_or_format", False)
labeled_examples = scenario.get("labeled_examples", 0)
simple_instruction_task = scenario.get("simple_instruction_task", False)
# 1. Fresh, private, or changing FACTS is the classic RAG signal. You cannot
# fine-tune knowledge that changes daily, and prompting cannot hold a
# whole corpus, so you retrieve.
if needs_fresh_or_private_facts:
return "rag"
# 2. A new BEHAVIOR/format/style with enough labeled data is the fine-tune
# signal. Fine-tuning teaches how to act, not what is true today.
if needs_new_behavior_or_format and labeled_examples >= 500:
return "finetune"
# 3. A behavior need with too little data still cannot fine-tune well, so
# fall back to prompting with examples rather than overfit a tiny set.
if needs_new_behavior_or_format and labeled_examples < 500:
return "prompt"
# 4. Anything a clear instruction and a few examples can do: just prompt.
if simple_instruction_task:
return "prompt"
return "prompt" # default to the cheapest tool
# Each row: a scenario an interviewer would actually pose, and the answer that
# gets you the offer.
cases = [
({"needs_fresh_or_private_facts": True,
"simple_instruction_task": False},
"rag", "Answer questions over the company's private, frequently-updated docs"),
({"needs_new_behavior_or_format": True, "labeled_examples": 5000},
"finetune", "Force a strict house JSON style, 5000 labeled examples on hand"),
({"needs_new_behavior_or_format": True, "labeled_examples": 40},
"prompt", "Want a specific tone but only 40 examples, too few to fine-tune"),
({"simple_instruction_task": True},
"prompt", "Summarize a pasted article into three bullets"),
({"needs_fresh_or_private_facts": True,
"needs_new_behavior_or_format": True, "labeled_examples": 9000},
"rag", "Needs both fresh facts AND style: facts win, retrieve then shape"),
]
print("STEP 1: route each scenario")
all_ok = True
for scenario, expected, desc in cases:
got = decide(scenario)
ok = (got == expected)
all_ok = all_ok and ok
mark = "ok " if ok else "BAD"
print(f" [{mark}] {got:8s} (want {expected:8s}) {desc}")
print("")
print(f"STEP 2: every scenario routed as an interviewer expects : {all_ok}")
print("")
print(f"RAG VS FINETUNE VS PROMPT ROUTING MATCHES EXPECTED: {'YES' if all_ok else 'NO'}")
if not all_ok:
sys.exit(1)
print("Judgment beats recitation. Next round: LLM system design.")
Runnable lab
ip4-rag-vs-finetune-decision.pyEncode the RAG-vs-finetune-vs-prompt judgment as a decision function and prove it routes real scenarios correctly.
Proves: RAG VS FINETUNE VS PROMPT ROUTING MATCHES EXPECTED: YES
Runs in your browser via Pyodide. First run loads the runtime once; no install, no server.
The decision function routed every scenario the way an interviewer would expect: private, frequently-updated docs to RAG; a strict house format with thousands of labeled examples to fine-tuning; a desired tone with only forty examples back to prompting because that is too little data to fine-tune well; and the mixed case where fresh facts and style are both needed resolves to RAG first, because you cannot fine-tune knowledge that changes daily and you can always shape the retrieved answer with a prompt. Practice giving these answers in under two minutes each, and always attach the "when not to" clause, because that is what distinguishes judgment from memorization. Keep a two-minute explanation ready for BPE, attention, and the KV cache too, since those are the three internals this round asks you to explain cold. Next you scale up to designing a whole system.
Check your understanding
1. What is the theory round actually testing?
2. When should you reach for RAG over fine-tuning?
3. When is fine-tuning the wrong tool?