Chapter 5 of 8
Domain 2: prompt engineering and structured output
Domain 2 has one rule the exam tests repeatedly: few-shot examples beat descriptive instructions for ambiguous scenarios. A vague instruction like "be more precise" can be read many ways, while a worked example shows format, decision logic, and edge-case handling unambiguously, and the model generalizes the pattern. The second theme is schema design for structured output. Using tool_use with a JSON schema is the most reliable way to get structured output, but it only guarantees valid syntax, not semantic correctness. Mark a field required only if it is always present in the source, because marking an optional field required forces the model to fabricate a value when the data does not exist. Use a nullable type for fields that may be absent so the model can return null honestly. Always include other and unclear in enums, with a companion detail field, so data outside your categories is never silently dropped, and remember an honest unclear is better than a confident wrong category. When extraction fails validation, you retry with specific context: the original document, the previous incorrect extraction, and the exact error. Retry helps for format and structural errors, but no number of retries produces a due date that is genuinely absent from the invoice. Finally the Batch API saves up to 50 percent but adds up to 24 hours of latency, so it is correct only when nobody is waiting. In this lab you extract JSON with the shared mock model, prove nullable prevents fabrication where required forces it, run a validation-retry loop, and route Batch versus synchronous.
The lab: read it, then run it
#!/usr/bin/env python3
"""
LAB CF5: Prompt engineering and structured output (Domain 2, 20%).
Two exam rules dominate this domain. Nullable fields prevent hallucination
while required fields force it when the data is absent. And retries help for
format and structural errors but never conjure information that is genuinely
missing from the source. This lab uses the shared offline mock LLM to extract
JSON for real, then drills the schema decisions and a validation-retry loop, and
finishes with the Batch-vs-synchronous API router.
Deterministic, offline. Imports the shared academy_llm mock (standard library).
Run: python3 modules/academy-content/labs/cca-f/cf5-structured-output.py
"""
import sys, os, json
_cands = [os.path.join(os.path.dirname(__file__), "..") if "__file__" in globals() else None,
os.path.join(os.getcwd(), "..", "labs"), os.path.join(os.getcwd(), "labs")]
for _c in _cands:
if _c and os.path.exists(os.path.join(_c, "academy_llm.py")):
sys.path.insert(0, os.path.abspath(_c)); break
from academy_llm import complete
# STEP 1: structured output is parseable data, not prose. Ask the mock to
# extract JSON and load it with a real parser.
doc = "Extract JSON: the invoice name is Acme and the number is 5000"
raw = complete(doc)
obj = json.loads(raw) # would raise if the output were not valid JSON
print("STEP 1: JSON extraction that a program can load")
print(f" model output : {raw}")
print(f" parsed name : {obj.get('name')} parsed number: {obj.get('number')}")
json_ok = obj.get("name") == "Acme" and obj.get("number") == 5000
# STEP 2: required vs nullable. Marking an ABSENT field as required forces the
# model to fabricate; a nullable field lets it return null honestly.
schema_required = {"required": ["due_date"]} # bad: due_date absent in source
schema_nullable = {"nullable": ["due_date"]} # good: allows null
source_has_due_date = False
def extract_field(present: bool, field: str, mode: str):
if present:
return "2025-01-15"
if mode == "nullable":
return None # honest: not in the source
return "FABRICATED-DATE" # required forces a made-up value
req_val = extract_field(source_has_due_date, "due_date", "required")
null_val = extract_field(source_has_due_date, "due_date", "nullable")
print("")
print("STEP 2: required forces fabrication, nullable allows honest null")
print(f" required due_date -> {req_val!r} (fabricated, wrong)")
print(f" nullable due_date -> {null_val!r} (honest null, correct)")
nullable_ok = (req_val == "FABRICATED-DATE" and null_val is None)
# STEP 3: enums must carry 'other' and 'unclear' so data is never silently
# dropped.
ENUM = ["invoice", "receipt", "credit_note", "other", "unclear"]
def categorize(kind: str) -> str:
return kind if kind in ENUM else "other"
enum_ok = ("other" in ENUM and "unclear" in ENUM and categorize("packing_slip") == "other")
print("")
print(f"STEP 3: enum {ENUM}")
print(f" unknown 'packing_slip' -> {categorize('packing_slip')} (captured, not dropped)")
# STEP 4: validation-retry. Retry fixes a fixable format error; it cannot invent
# a value that is not in the source.
def validate(value):
if value is None:
return "value absent from source"
if "/" in value:
return "date not in ISO 8601 format"
return None
def extract_with_retry(initial, source_present, max_retries=2):
value = initial
for _ in range(max_retries):
err = validate(value)
if not err:
return value, "passed"
if "format" in err:
value = value.replace("03/05/2025", "2025-03-05") # fixable, retry helps
else:
if not source_present:
return value, "unfixable: data absent" # retries cannot help
return value, "exhausted"
fixed, status_fix = extract_with_retry("03/05/2025", source_present=True)
absent, status_absent = extract_with_retry(None, source_present=False)
print("")
print("STEP 4: validation-retry loop")
print(f" bad-format date -> {fixed} ({status_fix})")
print(f" absent field -> {absent} ({status_absent})")
retry_ok = (fixed == "2025-03-05" and status_fix == "passed"
and status_absent == "unfixable: data absent")
# STEP 5: Batch API vs synchronous. If anyone is waiting, it is synchronous.
def api_for(scenario: str) -> str:
waiting = scenario in ("pre-merge code review", "interactive support",
"ci/cd merge gate")
return "synchronous" if waiting else "batch"
api_ok = (api_for("pre-merge code review") == "synchronous"
and api_for("weekly security audit") == "batch")
print("")
print("STEP 5: Batch vs synchronous routing")
print(f" pre-merge code review -> {api_for('pre-merge code review')}")
print(f" weekly security audit -> {api_for('weekly security audit')}")
ok = json_ok and nullable_ok and enum_ok and retry_ok and api_ok
print("")
print(f" JSON parses into named fields : {json_ok}")
print(f" nullable prevents fabrication : {nullable_ok}")
print(f" enum captures the unknown : {enum_ok}")
print(f" retry fixes format, not absence : {retry_ok}")
print(f" batch only when nobody is waiting : {api_ok}")
print("")
print(f"STRUCTURED OUTPUT RULES HOLD: {'YES' if ok else 'NO'}")
if not ok:
sys.exit(1)
print("Nullable over required, honest unclear over a wrong guess, sync when someone waits. Next: tool design and MCP.")
Runnable lab
cf5-structured-output.pyExtract JSON with the shared mock, prove nullable prevents fabrication, run a validation-retry loop, and route Batch versus synchronous.
Proves: STRUCTURED OUTPUT RULES HOLD: YES
Runs in your browser via Pyodide. First run loads the runtime once; no install, no server.
The lab made the schema rules physical. A field marked required forced a fabricated date, while the nullable field returned an honest null, which is the difference between clean data and a quiet lie in your pipeline. The retry loop fixed a badly formatted date but could not conjure a value that was never in the source, which is exactly the boundary the exam tests. And the Batch router sent the pre-merge review that a developer is waiting on to the synchronous API, and the weekly audit to Batch. Carry two habits: build your eval test set from real cases, not cases your prompt already passes, and add explicit normalization rules for values that appear in many formats so the JSON is not just valid but consistent. Next you move to Domain 4 and the tools themselves.
Check your understanding
1. For an ambiguous categorization task, what steers the model best?
2. An invoice may or may not contain a due date. How should the schema treat that field?
3. When does retrying an extraction NOT help?
4. When is the Batch API the correct choice?