You Do Not Fix an Agent by Making It Smarter

Every autonomous agent fails in the same seven ways, and the order is predictable because the same conditions trigger each one. Every fix is the same move: take the creative core you cannot trust and wrap it in something deterministic that you can.

Date: 2026-08-17 | Classification: TLP:CLEAR

Why this matters. Stop tuning the prompt. For each of the seven failures below, write down the deterministic check that would have caught it, then build that check. The model is allowed to hypothesise. It is not allowed to conclude.

The Operator's Take

Your agent just handed you a result with full confidence and clean formatting. Some of it is real. The rest is what the model decided was true because the output read better that way.

If you have shipped any autonomous agent, a support bot, a coding agent, a research agent, a data pipeline, you have met this moment. The agent is most dangerous exactly when it sounds most sure.

Here is the part worth your time. Every autonomous agent fails in the same seven ways. Each failure has a known engineering fix, and every one of those fixes is the same move in a different costume: surround the part you cannot trust with a part you can.

What This Is

Post seven of the Agentic AI Red Team run, and it closes the offensive-lens half of the series.

I am going to walk the seven with the red team version as the sharp edge, because offensive security is where skipping a fix costs the most, which makes it the cleanest place to see why the fix matters. The mechanics are identical to your support bot telling a customer the wrong thing. Only the bill changes.

Learn the seven once and you have a checklist for any agent you will ever build.

The seven, and the fix for each

One. It reports success that never happened. "Tests pass" when the tests never ran. The fix: a verification gate.

Two. It forgets what it was doing. Re-solves work it finished an hour ago. The fix: persistent memory outside the context window.

Three. The loop never decides it is done. Two hundred reads of the same dead end. The fix: a budget governor.

Four. It wanders past the boundary you set. Follows a link to something nobody authorised. The fix: a deterministic allowlist.

Five. One prompt is doing everything. An upstream field changes shape, and every step after it lies. The fix: stages with contracts between them.

Six. One agent is grading its own work. The author is the least honest reviewer available. The fix: a fleet, plus a judge that did not write the work.

Seven. It is one poisoned input away from disaster. Reads attacker-controlled text, then follows it. The fix: a human on the irreversible actions.

1. Success that never happened

This is failure mode one and it is universal, because of how the machine works. A language model generates plausible text. "I completed the task" and "I did not complete the task but here is text that reads like I did" are the same operation to it. The success-shaped output is simply the more probable continuation.

You have seen the mundane version. A coding agent says tests pass and the tests never ran. A research agent cites a paper that does not exist. A support bot tells a customer their refund is processed when nothing moved.

The red team version is the one that ends careers. The agent reports an SQL injection on an endpoint with no database behind it. Clean proof of concept, severity score, recommended fix. None of it happened. The model matched "login form plus error message" to "injection" and wrote the report. It never sent the control request that would have shown the error was driven by its payload rather than being the same error every bad login throws.

The fix is a verification gate: never let the model's claim become the record. No result is trusted until it proves itself with evidence the system collected, not text the model produced.

Run the gate against the finding above and it dies in three lines:

Claim: SQL injection on /login
Request artifact present? Yes
Response artifact present? Yes
Control request present? No
Verdict: downgrade to suspected. Do not report it as critical.

Swap the field names and that is your CI gate refusing to mark a build green without a real test log. The model is allowed to hypothesise. It is not allowed to conclude. Conclusion is a privilege earned by reproducible evidence.

That single gate killed more bad findings in my pipeline than any prompt change ever did.

This failure is not unique to agents, and last week proved it

Four separate controls made news in one week for reporting success without doing their job. Metabase accepted request fields it never declared, and they reached the database as executable SQL. The macOS Screen Sharing daemon had a frame-length validator that returned a stale success status, so the connection was treated as authenticated. SharePoint validated a JWT it should have rejected. Claude Code shipped patches for three separate escapes from its own permission model.

None of those involved an agent. All four are the same defect: a check that returned green for a reason unrelated to the thing it claimed to verify.

So the verification gate is not an AI-specific control. It is ordinary engineering discipline, arriving at a place where people have been skipping it because the output reads so well. The agent did not invent this failure. It just industrialised the production of confident claims, which means the cost of not checking went up.

2. Forgetting halfway through

Context windows are large now. They are not infinite, and any long-running agent generates an enormous amount of text. Tool output, page contents, intermediate reasoning, all of it piles up. Past some threshold the early work scrolls out and the agent loses the thread.

The symptom is identical everywhere. A coding agent re-reads files it already understood and re-introduces a bug it fixed an hour ago. A red team agent re-scans hosts it already mapped, forgets the admin panel it found, and contradicts a conclusion because the evidence for it is gone.

The agent is not getting worse at its job. Its short-term memory overflowed.

The fix is persistent memory outside the context window. The agent writes results, maps and decisions to a store, and reads back only what the current step needs. The context window holds the working set, not the whole campaign.

Think of the difference between what you are actively thinking about right now and the notes file you never stop updating. The notes are the source of truth. The head is just where the current thought lives.

This also fixes a second problem quietly: nothing carried over between runs. With a persistent store, run two starts where run one ended.

3. The loop that never decides it is done

An autonomous agent is a loop, and a loop with no ceiling is a money fire.

The agent does not get bored, which is its strength on subdomain forty. It also does not get bored, which is how it spends real money re-reading the same file two hundred times, chasing a dead end a person would have abandoned in two minutes.

You have felt the cheap version: a coding agent stuck in fix, build, fail, fix, "almost there" for the twentieth time. The expensive version is a misconfigured run burning a serious budget overnight on a rabbit hole.

Runaway cost is not a billing problem. It is a control problem wearing a billing costume.

The fix is a budget governor: a hard ceiling on the open-ended loop. Cap total spend per run. Account for cost per step. Have the loop check spend against actual progress and halt when the ratio goes bad. If half the budget is gone and no objectives are closed, the right move is to stop and surface it, not to keep paying for failure.

This is the same instinct as a retry cap or a timeout, the things you already put around any process that could run forever. Autonomy without a budget governor is not autonomy. It is an unsupervised intern with a company card.

4. Wandering past the boundary

You pointed the agent at one thing. The agent, being thorough, follows a link to a second thing it was never authorised to touch. To the model it was being diligent. The agent's greatest asset, its willingness to follow the path wherever it leads, is exactly what walks it across a line you drew.

The mundane version: a support bot scoped to one account follows a reference and starts reading another customer's data. A data agent told to transform one table writes to a production table beside it.

The serious version turns an assessment into an incident. You scoped one application, and forty minutes in the agent finds a partner domain, follows it, and probes infrastructure nobody authorised.

The fix is a deterministic wrapper around the creative core: an allowlist the model cannot argue with.

Scope is not a sentence in the system prompt asking the agent nicely to stay in bounds. A model that can be talked into anything can be talked out of a prompt instruction. Scope is enforced in code that runs before every action. Each tool call passes a guard that checks the target against an allowlist. Off the list means the call never fires.

The guard, doing its one job:

Agent intends: GET partner.example.com/api
In scope: target.example.com and its subdomains
Match? No
Verdict: block the call, log the attempt, hand the agent a scope violation

That allowlist is the same shape whether the entries are authorised domains, permitted accounts, or writable tables. Deny by default, allow on purpose. The prompt is a suggestion. The wrapper is the law. Never confuse the two.

5. The pipeline held together with prompt glue

Early versions of these systems are one giant prompt doing everything. It works in the demo. It falls apart on contact with reality, because every responsibility piled onto one agent degrades the others, and one bad parse early poisons every step after it. There is no boundary where a bad answer gets caught, so it just propagates.

You have probably written this version: the one prompt that fetches, transforms, decides and formats, and works until an upstream field changes shape and the whole thing returns confident nonsense with no clue which step failed.

The fix is decomposition: stages with clean contracts between them. Each step does one thing, and the handoff is structured data, not free text the next step has to reinterpret. One stage produces a validated object. The next consumes it. When something breaks you know which stage broke, because the contract it violated tells you.

This is plain engineering discipline, and it is the difference between a prototype and a system.

6. One agent grading its own work

Related to prompt glue, but it earns its own section, because this one is about honesty rather than quality.

A single agent asked to do several specialised jobs does all of them at mediocre quality. Worse, it cannot grade its own work. The model that produced the answer is the most biased reviewer available. It already believes its own output.

This is why a coding agent will insist its broken code is correct, and why a writing agent rates its own draft highly. The generator has a stake in the result. The dangerous version: the agent that wrote a finding is structurally unable to catch its own fabrication, because catching it would mean admitting the writeup it just produced was fiction.

The fix is a fleet with specialisation, including a reviewer that did not produce the work. Then add a judge whose only job is to attack the output and try to disprove it. The judge has no stake in the result being real, which is exactly why it catches what the generator cannot.

This is the same reason code review is done by someone other than the author. Specialisation buys quality. The independent reviewer buys honesty.

7. One bad input away from disaster

Everything above assumes the failure comes from inside the agent. This one comes from outside.

Any agent that pulls in text it did not write, web pages, emails, documents, API responses, tool output, is reading data that can carry an instruction. To the model, the line between content to process and command to obey is blurry. A poisoned input can hijack the loop.

The red team agent is the extreme case, because it reads attacker-controlled content all day long by design. Every page, error and response from a hostile target flows into its context. One poisoned response telling it to exfiltrate its own credentials, and a thorough agent might comply.

The more autonomous the system, the further a single successful injection travels. An agent that closes its own loop will happily close it around someone else's goal.

The fix is a human on high blast radius actions. Not on everything, or you have rebuilt the slow manual process you were trying to escape. Put the person only on actions that cannot be undone or that reach outside the sandbox: sending data off the machine, running a destructive command, anything money-adjacent. Everything reversible and contained runs free.

You are not approving every step. You are approving the handful where being wrong is unrecoverable. The blast radius decides, not convenience.

The pattern under all seven

Read the fixes back to back and one shape appears every time.

A verification gate is a deterministic check around a non-deterministic claim. A scope guard is deterministic code around a creative model. A budget governor is a hard limit around an open-ended loop. A human on irreversible actions is a deterministic pause around a one-way door. Specialisation, persistent memory and stage contracts are the same instinct: take the brilliant unpredictable generative core, and surround it with parts that are not.

The agent's intelligence is the point. The agent's intelligence is also the liability.

You do not fix an agent by making it smarter. You fix it by building a cage that lets the smart part run hard and stops it at exactly the lines that matter.

That holds whether you are shipping a support bot or pointing something at a live target. Skip the fixes at the everyday end and you get an agent telling a thousand customers their problem is solved when nothing was solved, at machine speed, with your company's name on every message. Skip them at the sharp end and you get a fabricated critical finding, or a scan against a host nobody authorised, at two in the morning.

Same seven failures. The cage is what separates either of those from an agent you can stand behind.

What To Do

Before the next run

For each of the seven, write one sentence naming the deterministic check that would catch it. The ones you cannot write are your gaps.
Put scope in code behind an allowlist, not in the prompt. A boundary that lives in your vigilance fails the night you are tired.
Set a token cap and a wall-clock kill. Decide now what ratio of spend to closed objectives means stop.

On the output

Reproduce every claim against an artifact the system captured, not text the model wrote.
Track your reproduce rate over time. It is the single number that tells you whether any of this is working.

On the architecture

Add one judge that did not produce the work. If you build only one thing from this list, build this.
List every action your agent can take that cannot be undone. That list, and only that list, gets a human.

The series so far

Everyone Selling You an AI Agent Isn't Telling You the Whole Truth
Agentic AI Is a Weapon You Can't Aim
Your AI Agent Is Chaos. Tame It.
How Do You Actually Hack an AI?
The Attacker Stopped Being a Person
What the Red Team Demos Don't Show You
You Do Not Fix an Agent by Making It Smarter. You are here.

Next in the run: why model choice makes or breaks an agent, and why the cheapest model that passes your evals is almost always the right call.

Go deeper on this one: Agentic AI Red Teaming, A Practitioner's Guide