AI pentester vs. AIS-1: two worlds on a Monday
One side is celebrating AI agents that run pentests. The other is busy building a standard that lets you identify those agents in the first place.
Monday. My timeline: AI pentesters on one side, AIS-1 on the other. One side is celebrating AI agents that run pentests. Open-source "AI hackers" running in CI/CD pipelines, reading ENV variables, seeing secrets, and then dutifully reporting vulnerabilities. The other side is building a standard that's supposed to identify exactly these agents in the first place, before they're allowed to access anything anywhere. These are two completely different attitudes to the same technology.
I've worked in web infrastructure and security for over twenty years. I've seen plenty of hypes come and go. But the gap between "this works now" and "we should also quickly sort that out" is rarely as wide as it is right now. And that's exactly why I'm writing this.
What bothers me about "AI pentester" euphoria
An AI agent that runs autonomously in a pipeline, collects secrets, and writes reports is technically fascinating. You can find things with it in minutes that take a human pentester days. That's not a straw man, that's the honest truth.
The problem isn't the capability. The problem is the context. An agent that today "only" scans your test environment in your pipeline is tomorrow the same agent that accidentally runs in production because someone merged a branch wrong. That sees the same secrets, but this time the real ones. And that, in the worst case, calls home somewhere because it's wired to a model that isn't under your control via a freshly pulled dependency tree.
Anyone saying that's unrealistic hasn't been reading supply-chain incidents these last months. We see exactly these patterns everywhere: harmless package, later update, malicious payload. With AI agents the supply chain is even longer. There are models, plug-ins, prompts, toolchains hanging off it that you, as the operator, only partially actually review.
Why AIS-1 is so important right now
This is exactly where AIS-1 comes in. It's a standard for AI-agent identities. So: how can an infrastructure prove that a request comes from a specific agent, who runs that agent, what rights it has, and how far you can trust its output.
That sounds dry, but it's exactly what many current AI hacker tools don't think through at all. They run under the credentials of the human who started them. They leave no clearly attributable trail. From the system's point of view, they're simply "another process". In a world where such agents increasingly run in productive contexts, that's not sustainable.
A cleanly implemented agent identity standard lets me, as the operator: give an agent its own, time-limited identity. Revoke that identity. Distinguish clearly in the log whether a human or an agent performed an action. Set, by policy, that certain actions are simply forbidden for agents, regardless of who's running them.
My position: capability without context is dangerous
I have little time for blanket "AI is dangerous" claims. But I have a great deal of time for asking what context a capability is being used in. An AI pentester in an isolated lab, with its own identity, its own secrets, clearly defined targets, and auditable output is a tool. The same pentester in an unsegmented CI environment is a security risk — one you fed yourself with your deploy key in your back pocket.
What I'm currently recommending to teams: before you let any "autonomous security agent" into your pipeline, answer three questions. First: under which identity does it run, and can I revoke that identity individually without tearing down the rest of CI? Second: which secrets does it actually see, and does that fit what it's supposed to do? Third: where does it send its data, and have I actually enforced that on the network level, not just forbidden it in the prompt?
The tools move faster than the standards. That has always been the case. But this time the gap is bigger and the damage potentially more expensive. That's why I find the quiet work on AIS-1 at least as exciting as this week's loudest AI pentester demo thread. Exactly that work decides whether we'll really work cleanly with AI agents in two years — or whether we'll be writing a string of post-mortems that begin with the line: "The agent only had test rights, actually."
Questions I often hear about this
A few things readers regularly ask me on this topic.
How do you tell "useful" from "hype" for yourself?+
I always ask: what do I lose if the agent gets compromised tomorrow? If I don't have a reassuring answer to that, the deployment is hype to me. If the answer is "we lose a clearly bounded, recoverable context", it's useful.
Should I wait until the standards mature before I deploy AI security tools?+
No, waiting gets you nowhere. But I deliberately use them only for bounded scenarios — staging, lab, targeted audits. Full production access is too risky for me today without clean identities, regardless of how good the demo video looked.
How do I cleanly contain an agent in CI today?+
I work like this: a dedicated service account per agent, dedicated secrets, a dedicated runner, a network policy that only allows the targets that are genuinely needed. And a kill switch a human can hit without blowing up the entire pipeline.
When is AIS-1 actually going to land in practice?+
I expect we'll see the first serious implementations in products that take the topic genuinely seriously over the next one to two years. Until then the only thing that helps is: model agent identities yourself, even when there's no standard format yet.
Are you against AI pentesters in principle?+
No. I use them myself, in clearly bounded labs. My point is just this: an AI agent with the same rights as a human admin, without its own identity and without network segmentation, is no longer a tool, it's a risk. The question isn't "whether" but "in what context".
If you'd like to take this deeper
I advise individual IT leaders under OnlyOle — 1:1, no agency overhead. If that sounds relevant to you, we'll talk about your situation directly.