Don't Try to Make an Agent Deterministic

A serious agent needs a playbook, a harness, hard boundaries, and an operator who owns the decisions the system cannot score.

Date: 2026-08-31 | Classification: TLP:CLEAR

Why this matters. Stop treating a human approval button as the architecture. Decide what the agent may explore, what the harness must block, what evidence it must collect, and when an operator must take responsibility.

The Operator's Take

The argument that prompting ends and coding begins is too neat. A model does not become reliable because you moved its prompt into a Markdown playbook or wrapped it in code. It is still a probabilistic system making judgments from incomplete information.

The useful shift is not prompt to code. It is prototype to operation. A real agent needs a playbook that says what good work looks like, a harness that gives it bounded ways to act, and an operator who owns the decisions the system cannot score for itself.

That is not a retreat from autonomy. It is the only version of autonomy worth buying. Let the machine run the repeatable work hard. Put a skilled person at the points where evidence is ambiguous, the mission changes, or a wrong action carries consequences nobody can undo.

What This Is

Post eleven closes the Agentic AI Red Team foundations run.
The model remains non-deterministic. No prompt, framework, or API changes that fact.
A playbook tells the agent how to pursue an objective. A harness limits what it can do while pursuing it.
Evidence, not fluent output, is what turns an agent action into a finding or a completed task.
The operator is not there to approve every click. The operator owns judgment, escalation, and the consequences of a material decision.
The target is not one hundred percent autonomy. The target is the highest autonomy that can still be explained, stopped, and stood behind.

Prompting does not end. It matures into a playbook.

A one-shot prompt is useful for discovery. Give an agent a bounded task, watch where it hesitates, and learn which instructions actually matter. That is how you find the shape of the work.

The mistake is treating that first prompt as an operating model. It tells the agent what you want, but it rarely answers the questions that make a long-running system safe and useful:

What phase is this run in?
What evidence changes the plan?
What counts as progress?
What counts as done?
What must never happen without an escalation?

Those answers are the playbook. A playbook is not a giant prompt that tries to predict every branch. It is a sequence of objectives, evidence standards, stop conditions, and decision rights. It lets the agent adapt inside the mission without inventing a new mission halfway through it.

For a red team engagement, that might mean reconnaissance can enumerate and prioritize, validation can test a suspected path only against an authorized target, and reporting can promote a finding only when the evidence package is complete. The agent still chooses its next probe. It does not get to redefine the scope, call an unproven observation a compromise, or keep spending after the plan has failed.

The harness does not make the model deterministic

This distinction matters because people keep promising a result code cannot deliver. A harness does not turn a model into a calculator. It makes the system accountable for the model's actions.

Agent. Owns hypotheses, prioritization, and the next best action. It must not decide scope, irreversible impact, or whether its own claim is true.

Harness. Owns tool access, state, budgets, logs, and retry limits. It must not make novel judgments about mission intent.

Operator. Owns escalation, ambiguous evidence, exceptions, and material decisions. The operator should not spend time on routine work the playbook already scores.

The agent is the creative layer. It reads a strange response, forms a theory, and chooses where to look next. That is why it is valuable.

The harness is the hard layer. It checks the target before a tool fires, keeps an evidence trail, denies actions outside the allowed set, caps the run, and preserves state between stages. Those are not intelligent decisions. They are rules that must hold even when the model is confused, compromised, or confidently wrong.

MITRE ATLAS now catalogs agent-specific paths such as context poisoning, tool invocation, configuration modification, and host escape. That is the practical reason to put the boundary outside the model. A system that can read hostile content and invoke tools needs controls that hostile content cannot talk around. MITRE ATLAS

A human approval button is not a control plane

If the only control is a person approving a giant stream of opaque actions, the person becomes a rubber stamp and the agent becomes a queue generator. If the operator has no evidence, no authority to change the plan, and no clear escalation condition, calling that human-in-the-loop is a polite name for hoping.

The operator is part of the service, not the emergency brake

The wrong picture is a customer watching an agent work and clicking approve every time a modal appears. That is slow, exhausting, and unsafe. It turns the person into the least informed component in the loop.

The better picture is an operator who understands the playbook and owns the moments where the playbook reaches its limit. The operator does four jobs.

Interprets ambiguity. A model can notice that two pieces of evidence conflict. It cannot reliably know whether that conflict is noise, a new attack path, a broken integration, or a reason to stop.

Changes mission intent. The customer may care about speed, stealth, proof, business continuity, or a newly discovered constraint. Those are priorities, not tool results. An operator translates them into a changed plan.

Approves material exceptions. Some actions are reversible and contained. Others can expose data, disrupt a service, create legal consequences, or change the engagement itself. The difference is blast radius, not the number of steps the agent took to get there.

Judges evidence. An agent can assemble requests, logs, screenshots, and test results. The operator decides whether the package proves the claim strongly enough to act on it.

NIST's AI Risk Management Framework is deliberately system-level: risk management applies to the design, development, use, and evaluation of AI systems, not simply to the model choice. That is the right lens. You are operating a socio-technical system with a model inside it, not buying a vending machine that dispenses decisions. NIST AI RMF

What should run free, and what should stop?

Do not draw the line at "AI" versus "human." Draw it at whether the system can score the action and contain the mistake.

Let the agent run

Repeatable reconnaissance inside a verified scope.
Ranking hypotheses against a stated objective.
Gathering the artifacts a playbook requires.
Retrying a bounded action under a fixed cap.
Summarizing evidence for review.

Require an operator

A new target, changed scope, or a conflict in authorization.
A claim that will be reported as a material finding.
An exception to a policy, budget, or stop condition.
Any action with irreversible or external consequences.
A decision where evidence remains contradictory.

This is why percentage arguments are usually a distraction. There is no universal split between machine and human work. A mature process may automate most of its volume while requiring expert attention at only a handful of high-consequence moments. Another process may need constant oversight because it cannot score the result. The ratio follows the work. It is not a maturity badge.

The real test: can you explain why the agent stopped?

People ask whether an agent can complete a task. That is only half the question. Ask four more:

What evidence made it believe it was making progress?
What hard rule could have stopped the next action?
Who could change the plan when the evidence did not fit the playbook?
Can an operator reconstruct why the system did what it did?

If those questions have no clean answer, the problem is not that the model needs a better prompt. The system has no operational ownership.

The most dangerous form of automation is not a system that makes an obvious error. It is a system that takes plausible actions for opaque reasons, then leaves a person to explain the consequences afterward. A playbook, a harness, and an operator turn that into a system where autonomy has a boundary and accountability has a name.

What To Do

Build the operating model before expanding autonomy

Write the mission in phases: objective, allowed actions, required evidence, stop condition, and escalation condition for each.
List every action the agent can take, then mark the ones that are irreversible, externally visible, or scope-changing.
Put those boundaries in the harness, not in a sentence the model is asked to remember.
Define the artifact that proves each material claim before the agent starts looking for it.

Give the operator a real job

Name who can change the mission and who can authorize an exception.
Make the operator review evidence packages, not endless raw tool calls.
Log the agent's plan changes, blocked actions, evidence, and operator decisions in one trace.
Measure how often the operator changes the agent's conclusion. That is where the next playbook improvement lives.