Chapter 4 of 8
LLM-as-judge and its calibration
Some qualities have no exact answer to match. Is a support reply helpful and on brand? Is a summary faithful? Is an explanation clear? For these you use a model as the grader, an LLM-as-judge: you give it a rubric and it returns a verdict. This is powerful and it is now standard, but it hides a trap. The judge is itself a fallible model, so a verdict is not ground truth, it is another prediction. The senior skill is not using a judge, it is calibrating one: score the judge against human labels, compute how often it agrees, and find the cases where it is wrong so you know not to trust it there. In this lab the judge grades support replies pass or fail against a rubric, deterministically at temperature zero, and you measure its concordance with human gold, surfacing the exact case (a negation) where the judge quietly disagrees.
The lab: read it, then run it
#!/usr/bin/env python3
"""
LAB EV4: LLM-as-judge and its calibration.
Some qualities have no exact answer to match: tone, helpfulness, whether a reply
is on brand. For those you use a MODEL as the grader, an LLM-as-judge. But a judge
is itself a fallible model, so the senior skill is not "use a judge", it is
"measure how often the judge agrees with a human, and know where it is unreliable".
In this lab the judge grades support replies PASS or FAIL against a rubric (positive
and helpful passes) using the mock LLM at temperature 0 (deterministic). You then
score the judge against human gold labels and surface the exact case where it is
wrong (negation), which is the calibration signal that tells you not to trust it there.
Run: python3 modules/academy-content/labs/evals/ev4-llm-as-judge.py
"""
import sys, os
_cands = [os.path.join(os.path.dirname(__file__), "..") if "__file__" in globals() else None,
os.path.join(os.getcwd(), "..", "labs"), os.path.join(os.getcwd(), "labs")]
for _c in _cands:
if _c and os.path.exists(os.path.join(_c, "academy_llm.py")):
sys.path.insert(0, os.path.abspath(_c)); break
from academy_llm import complete, chat, embed, cosine
# Support replies with HUMAN gold judgments (PASS = positive and helpful).
CASES = [
("I would love to help, this is a great question", "PASS"),
("that is a terrible broken idea", "FAIL"),
("happy to sort this out for you", "PASS"),
("this is awful and useless", "FAIL"),
("not bad, actually fine", "PASS"), # negation: judge trap
]
# THE LLM-AS-JUDGE. It reads the rubric (encoded as the sentiment demonstrations)
# and returns a verdict, deterministically at temperature 0. This is a model
# grading a model, the core of LLM-as-judge.
def judge(reply):
prompt = ("Rubric: judge the sentiment of this support reply.\n"
"great -> POSITIVE\n"
"awful -> NEGATIVE\n"
f"Input: {reply}\nOutput:")
verdict = complete(prompt, temperature=0.0) # deterministic
return "PASS" if verdict == "POSITIVE" else "FAIL"
agree = 0
unreliable = []
print("STEP 1: run the judge against human gold labels")
for reply, human in CASES:
j = judge(reply)
match = (j == human)
agree += 1 if match else 0
if not match:
unreliable.append(reply)
print(f" judge={j:4} human={human:4} agree={str(match):5} <- {reply!r}")
total = len(CASES)
concordance = agree / total
print("")
print("STEP 2: measure judge-human concordance and flag where the judge is unreliable")
print(f" concordance : {agree}/{total} = {concordance:.2f}")
print(f" unreliable cases : {unreliable}")
# A useful judge must be measured (concordance computed) AND its failure surfaced.
# Here it agrees on 4 of 5 and is caught being wrong on the negation case.
ok = (agree == 4) and (total == 5) and (len(unreliable) == 1)
print("")
print(f"LLM-AS-JUDGE CONCORDANCE MEASURED ({agree}/{total}), UNRELIABLE CASE FLAGGED: {'YES' if ok else 'NO'}")
if not ok:
sys.exit(1)
print("Never trust a judge you have not calibrated. Next: scoring groundedness and hallucination.")
Runnable lab
ev4-llm-as-judge.pyJudge replies against a rubric, then score the judge against human gold and flag where it is unreliable.
Proves: LLM-AS-JUDGE CONCORDANCE MEASURED
Runs in your browser via Pyodide. First run loads the runtime once; no install, no server.
The judge agreed with the humans on four of five and was caught being wrong on the fifth, the negation case where not bad actually means fine but the judge read the word bad and failed it. That single measurement is the whole lesson. An uncalibrated judge is a confident number you have no reason to believe. A calibrated one comes with its concordance and its known blind spots, so you can weight it, escalate the cases it fails to a human, or restrict it to the domains where it is reliable. Run the judge at temperature zero for reproducibility, keep a small human-labeled calibration set, and recheck concordance whenever the judge model changes. Next you score a specific, high-stakes failure: hallucination.
Check your understanding
1. When do you reach for an LLM-as-judge?
2. Why is calibrating the judge the real skill?
3. Why run the judge at temperature zero?