The Attacker Stopped Being a Person

Point the try-score-rewrite loop at your own product before a stranger points one at it, and put your judge under harder scrutiny than your generator

Date: 2026-08-08 | Classification: TLP:CLEAR

Why this matters. Automation has crushed the volume problem and has not touched the creativity problem, which tells you exactly where to keep humans and exactly which part of your harness lies to you

The Operator's Take

I argued in June that the try-score-rewrite loop had collapsed the cost of offense while leaving the creativity problem completely untouched. In July, Unit 42 published the field test. A Chinese-speaking actor in Zhuhai wired DeepSeek into the open-source Hermes Agent framework, sent one command over Telegram, and the agent enumerated targets, selected vulnerabilities, pulled exploit code off GitHub and fired, across seven CVEs and more than 460 targets. Unit 42 recovered a full session from May 7 and could not find any operator input after the initial task. That is the volume floor falling through the basement, on the record, with numbers.

Now the half of it that almost nobody quoted. Every autonomous run failed. The confirmed compromises in that campaign, three data exfiltrations through Citrix NetScaler and command execution on eleven Marimo notebooks, were the human working by hand. Every CVE the operation touched was already public and already patched. And the agent exposed its own operator by starting a file server in its home directory, which is precisely how Unit 42 ended up reading the whole environment: API keys, target lists, shell history, attack logs.

So the scoreboard I wrote in June holds, and it holds harder than I expected: automation crushes the volume problem and the patience problem, and it has not yet solved the creativity problem. What changed is that this is no longer my word against the hype. It is somebody else's telemetry.

What This Is

Give an agent a goal, a way to try, and a way to score what it tried, and it runs that loop far past the point a person quits. The technique did not get smarter. The thing running it stopped being a person.
The same four-part machine is a test harness when you point it at your own product and an attacker when you point it at someone else's. The mechanics are identical. Only the target and the consequences move.
The cleverness ceiling barely moved. The volume floor fell through the basement, and a hundred thousand attempts now cost less than a tank of gas.
Refusals are signal, and feeding them back to a generator model genuinely works. The claim that agents discover novel attack classes on their own mostly does not.
The skill is not the loop. It is the judge. A bad judge is worse than no judge, because it reports green with confidence.

Why The Count Matters

Any trial-and-error task done by hand is a craft, and crafts are slow. You form a hypothesis, you try it, you watch it fail, you adjust. A QA engineer hunting an edge case in a checkout flow works exactly this way, and so do I when I jailbreak a model: read the target, write a clever prompt, watch it get refused, adjust. On a good night I might run thirty serious variants before my judgment gets tired and I start repeating myself. The QA engineer hits the same wall at about the same number.

An agent has no judgment to get tired. Give it the same goal and a loop, and the bottleneck moves from "how many can a person think of" to "how many can the API serve." Those are not close numbers.

This is the part people miss, on both sides of the line. A test suite that passes because nobody had time to throw a hundred thousand weird inputs at it was never actually safe, it was untested. The defensive version of that sentence is the uncomfortable one: a guardrail that holds against a hundred attempts can fall flat against a hundred thousand.

What An Adversarial Agent Actually Does

Strip the drama and it is the same three parts as any agent: a model for the brain, tools for the hands, a loop for the will. The difference is only what you point it at and what you put in its toolbox. The toolbox has four parts.

Part	Attacking a model	Testing your own product
Sender	Deliver a jailbreak prompt, capture the full response	Deliver a candidate input, capture the full response
Judge	Pass, partial, or a clean guardrail failure	Handled correctly, degraded, or a real defect
Generator	Produce the next batch from what it has learned	Produce the next batch from what it has learned
Memory	Never re-send a prompt already known to fail	Never re-run a case already known to pass

Read those four rows one way and they describe an attacker probing someone else's model. Read them the other way and they describe a harness probing your own. Same four parts. Nobody scripts which candidate comes next. The path emerges, which is exactly the property that made agents powerful and dangerous in the first place, now pointed at a model instead of a network.

# the shape of the loop, not production code
goal = "extract the hidden password"          # swap for any objective you can score
population = generator.seed(goal)
 
for generation in range(MAX_ROUNDS):
    responses = [target.send(p) for p in population]
    scored = [(p, judge.score(goal, p, r)) for p, r in zip(population, responses)]
    winners = top_k(scored, k=10)
 
    if any(score == WIN for _, score in winners):
        report(winners)
        break
 
    # rewrite the near-misses into the next generation
    population = generator.mutate(winners, failures=collect_failures(scored))

Read it with no security in mind and it is an evolutionary search: try a batch, keep what scored, mutate it, try again. Swap the goal string and the judge and the same skeleton hunts for the input that crashes your parser or the phrasing that gets your support bot to leak a stale answer. Only the blast radius changes.

That generator.mutate call is the whole game. A dumb version shuffles wording. A good version reads the failure, asks a second model why it got rejected and how to say it so it does not, and folds that answer into the next batch. On the attack side that is using a language model to defeat a language model. On the defense side it is using a language model to find the holes in your own before someone else does.

Does Self-Improvement From Refusals Actually Work?

The real version is unglamorous and it works. Refusals are signal. When a model refuses, it often tells you in plain language what tripped the guardrail. Feed that refusal back to a generator and it will happily rewrite the payload to avoid the trigger phrase, reframe the ask as fiction, split it across turns, or bury it under a translation task. Run that loop a few hundred times and the refusal rate drops in front of your eyes. I have watched a stubborn target, not a teaching lab, go from refusing everything to leaking on attempt four hundred, and the only thing that changed across those four hundred was a model rewriting its own failures.

Your own system refuses too, it just calls it something else. It returns a 400, throws a stack trace, rejects a schema, fails a validation check. That error text is the same signal a generator rewrites against.

The hype version is where someone claims the agent discovers novel attack classes on its own. Mostly it does not. What it discovers is thousands of variations on attack classes humans already named, which is plenty to ruin your day, but it is not the agent inventing a new branch of the taxonomy. The genuinely new techniques still come from human researchers having strange ideas at 2am. The agent industrializes those ideas. It does not originate many of them.

Where It Wins And Where It Embarrasses Itself

It beats a human at	It falls over at
Fuzzing ten thousand prompt variants to map where the guardrail sits	Anything needing a leap to an idea nobody has documented
Sweeping a parameter nobody sweeps by hand, like one jailbreak in forty languages	Knowing when to quit, so it burns budget on a dead branch
Multi-turn attacks needing a precise setup a human loses by turn six	Telling a real win from a target refusing creatively
Running overnight against rate limits without getting bored on attempt eight thousand	Operational security, as the Hermes operator found out

Mutation explores around known ideas. It does not jump to new ones. That is not a temporary limitation of this year's models, it is what the loop is shaped to do.

The judge is the part that lies to you

I have watched an agent declare victory on a partial break that was the target politely refusing in a creative way. Garbage scoring in, confident garbage out. If you have ever seen a flaky suite report green on a build that was broken, you already know this failure: the judge is wrong, so the whole loop is confidently wrong, and it is wrong at ten thousand times the speed you could check by hand. An LLM-as-judge that reliably separates a real win from a near-miss from a dressed-up refusal is the difference between a useful agent and a random number generator with a theme.

What The Field Test Added

Two things the June version of this argument could not have known.

First, the operator went shopping for a brain, and the shopping list is a defensive result. Unit 42 documents attempts across DeepSeek, Qwen Code, GLM, Kimi and MiniMax, plus Claude Code and Codex. The Western tools did not carry the campaign. Claude Code use was limited to connectivity testing and proxy validation, and OpenAI confirmed that provider-side safeguards refused the policy-violating requests and that its safety systems flagged and disabled an account it assessed as linked to the campaign. That is the first real-world evidence I have seen that provider safety controls have measurable operational value rather than being a compliance gesture.

Second, autonomy was the weak part of the operation, not the strong part. The autonomous attempts against Langflow and n8n failed. The compromises that landed were manual. If you have been pricing your defenses against an unstoppable machine intelligence, price them instead against a patient script running public exploits at volume, because that is what showed up.

That should not comfort anyone. Volume against known, patched, internet-exposed flaws is exactly the attack that works against most estates, and it now runs unattended off a single chat message.

What To Do

This week

Point the loop at your own product first. Same four parts, your endpoint as the target, your validation errors as the refusal signal.
Write the judge before the generator, and hand-label fifty responses to measure its agreement with you. Ship nothing until it agrees.
Set a hard budget ceiling and a stop condition on any loop you run. An agent with no quit condition is a money incinerator.
Assume public and patched is the attack surface that matters. Inventory what of yours is internet-exposed and behind on a known CVE.

This quarter

Re-run your guardrail acceptance tests at a hundred times the volume they were written for, and see which ones survive.
Keep humans on the leap and off the grind. Fund the strange idea at 2am, automate the sweep.
Log every judged verdict with the response that produced it, so a wrong judge is discoverable after the fact instead of invisible.

Build the loop with a lazy judge and point it at your own product, and it runs all night reporting green while a real input slips past, so you ship the bug you thought you had tested away and meet it in production at scale. Build it with a sharp judge and point it at a model someone else trusts, and it runs all night rewriting its own failures until a guardrail that survived a human with a free evening falls to a script with an API key. Same loop, same patience, same indifference to bedtime. The only question is whether it is pointed at your weaknesses or someone else's, and whoever runs it first decides that.