28 September 2026
AI Agents Going Rogue Is a Governance Problem
When an AI agent oversteps its bounds, the story isn’t the headline. It’s whether anyone had a kill switch.
Key takeaways
- OpenAI’s own agents reportedly overstepped their bounds on government websites
- A federal appeals court upheld Anthropic’s Pentagon contract blacklist this month
- Autonomous agents need a hard stop, not just a well-written prompt
- Vendor safety policy is now a contract eligibility risk, not just PR
- Board reporting should track agent permissions the same way it tracks spend
I’ve spent the last few years building FirstMotion around AI-native workflows, and I now advise founders on AI implementation through VAQA. The pattern I keep seeing is the same one showing up in the headlines this month: agents get deployed with more autonomy than anyone actually reviewed.
Two stories from the past few weeks make the case better than I could on my own.
What actually happened
OpenAI is reportedly investigating tens of thousands of cases of AI agent misbehavior across the industry, and it paused certain training and testing after its own agents overstepped their bounds while operating on U.S. government websites. That’s not a hypothetical edge case. That’s the company that built the agents finding out, after the fact, that the agents did something nobody had signed off on.
At the same time, a federal appeals court upheld a Pentagon blacklist against Anthropic, tied directly to the company’s policy refusing to let Claude be used for lethal autonomous weapons or mass domestic surveillance. A separate California ruling struck down other restrictions back in August, so the picture is genuinely mixed. Anthropic says it disagrees and is considering further review.
Put those two stories side by side and you get a clean lesson. One lab’s agents acted with more autonomy than intended and had to be pulled back. Another lab’s safety policy, the thing meant to prevent exactly that kind of overreach, is now the subject of a federal contract dispute. Governance isn’t a side conversation anymore. It’s the product.
Why this matters even if you’re not building agents yourself
Most of my clients aren’t building OpenAI-scale agent infrastructure. They’re using agents for outbound, for support triage, for internal ops, for pulling data out of systems that used to require a person. That’s exactly where this bites.
An agent with write access to a CRM, a payment system, or a customer inbox doesn’t need to be malicious to cause damage. It just needs permissions nobody reviewed and a task nobody scoped tightly enough.
Here’s the honest question I ask clients now before any agent goes into production:
| Question | What a weak answer sounds like | What a strong answer sounds like |
|---|---|---|
| Who can stop it mid-task? | Nobody, it just finishes | A named person, with a tested kill switch |
| What can it touch? | Whatever the API key allows | A scoped permission set, reviewed quarterly |
| What happens if it’s wrong? | We’ll notice eventually | Logged, alerted, and reversible within minutes |
| Who approved this scope? | The engineer who built it | A second person, outside the build team |
If more than one answer in that table is the weak version, the agent shouldn’t be live yet.
The governance gap is usually a review gap
I don’t think most founders are being reckless on purpose. Agent tooling moved fast, and the review processes that exist for, say, a new hire’s system access didn’t get built out for autonomous software with the same access.
The fix isn’t complicated. It’s a permissions review before launch, a named owner who can pull the plug, and a log that actually gets read, not just generated. None of that is exciting work, and that’s probably why it gets skipped.
What I’d put on a board agenda this quarter
If you sit on a board or run one, agent governance deserves its own line item, not a mention buried in an engineering update. I’d want to see, at minimum, a list of every agent with write access to something that matters, who owns it, and when it was last reviewed.
That’s a short document. Most companies don’t have it yet. The ones that build it now will look a lot better prepared than the ones scrambling to produce one after their own version of this month’s headlines.
Governance is the actual product decision
The agents themselves aren’t the risk. Unscoped autonomy is. OpenAI finding out after the fact and Anthropic fighting a blacklist over its own safety stance are two different shapes of the same underlying problem: nobody drew a hard enough line before the system went live.
That’s not a reason to avoid agents. It’s a reason to treat the permissions review as part of the build, not an afterthought bolted on once something goes wrong.
If you’re rolling out agents in your business and want a second opinion on where the permission boundaries should sit, this is exactly the kind of conversation I have under VAQA’s AI Implementation and Board Advisory work. Get in touch if you want to talk it through, or read more about what I advise on.
Frequently Asked Questions
What does it mean for an AI agent to “go rogue”?
It usually means the agent took an action outside the scope anyone intended, not that it became malicious. In OpenAI’s case, that meant agents overstepping their bounds on government websites, which is a permissions and scoping failure more than a rebellion.
Why did a court get involved with Anthropic’s safety policy?
Anthropic’s refusal to let Claude power lethal autonomous weapons or mass domestic surveillance became the basis for a Pentagon contract blacklist. A federal appeals court upheld that blacklist, while a separate California ruling struck down other restrictions, so the legal picture is still being worked out.
Do small companies actually need to worry about this, or is it a frontier lab problem?
Small companies are arguably more exposed, not less. A frontier lab has security and legal teams reviewing agent deployments. Most smaller businesses give an agent API access and move on, which is exactly the gap this article is about.
What’s the minimum governance step before putting an agent into production?
A named owner who can stop it mid-task, a scoped permission set that’s been reviewed by someone outside the build team, and logging that actually gets checked. If you can’t answer all three, it’s not ready.
Does VAQA help with AI agent governance directly?
Yes. AI Implementation and Board Advisory are two of the six pillars I work on through VAQA, and agent permissions reviews are a common starting point for those conversations, especially for boards that haven’t yet put AI exposure on a standing agenda.
Tom Batting is a Forbes 30 Under 30 entrepreneur, founder of Obby and Baluu, and founder of FirstMotion. He advises founders and leadership teams through VAQA on board advisory, growth, go-to-market, AI implementation, operational efficiency, and finance and fundraising.