Agentic AI security: What actually stops an AI agent from going rogue

There’s a slightly uncomfortable truth about agentic AI security: A smarter model is not necessarily a safer one.

The risk starts when intelligence gets paired with authority. Give a model access to a browser, an inbox, a CRM or a file system, and it can move from suggesting what should happen to actually making things happen. That changes the security problem considerably.

Gartner recently predicted that by 2029, at least 70% of organizations running agentic AI in IT infrastructure and operations will experience a material service, security or cost incident linked partly to insufficient runtime controls. Its point is a useful one: written policies may not be able to stop an agent from taking the wrong action at machine speed.

Which brings us to the question I hear most often when we talk to firms evaluating agents: What actually stops an agent from going rogue? The answer is (less dramatic than the question) mostly good architecture. 

An AI agent stays in bounds (and doesn't go rogue) when four things are controlled by design, not by prompt:

• the tools available to it
• the systems and data those tools can access
• the decisions it's allowed to make for itself
• the points where a person or another control has to step in

None of those ideas is particularly novel in enterprise security. Agents just make them more important.

Start with the agent's blast radius

A useful way to think about agent security is to start with the question: what’s the worst thing an agent could reasonably do with the access it has?

Take a research agent. Suppose it searches the web for information about a prospective client and, during that process, retrieves a page containing instructions designed to manipulate an AI system. This is an indirect prompt injection: malicious instructions arrive through content the model has been asked to process rather than directly from the user. OWASP (Open Worldwide Application Security Project) specifically calls out websites, documents, emails and API responses as possible sources of this kind of attack.

Now imagine two versions of that agent. The first can search approved sources and return a research summary. The second can search the web, access your CRM, edit records, send emails and invoke a collection of general-purpose tools.

Both agents might encounter exactly the same malicious input. The difference is what happens next.

For the first, the likely consequence is a bad or compromised research output. For the second, the same failure could potentially affect internal records, expose information or trigger an external action. That's the agent's blast radius.

From this example, you may be able to see why prompt-level rules such as “never modify this record” are only part of the answer. If the agent has no ability to modify the record in the first place, you have removed an entire class of failure rather than relying on the model to remember the rule.

OWASP describes the broader problem as excessive agency: an agent that can do more than its job requires. It breaks the risk into three parts: excessive functionality, excessive permissions and excessive autonomy.

Capability, access, and autonomy: three separate controls

Agent conversations often blur three separate architectural decisions.

1. Capability: What actions can the agent perform?
2. Access: Which systems, data and resources can those actions touch?
3. Autonomy: How much discretion does the agent have to decide what happens next?

They should be designed independently.

An agent might be capable of searching the web without being allowed to browse arbitrary internal systems. It might be able to prepare a database update without having permission to execute it. It might perform five steps in a workflow while having zero authority to invent a sixth.

This is the principle of least privilege applied to AI agents: give the agent the smallest set of tools and permissions required to complete the job.

OWASP now explicitly recommends that approach for AI agents, including per-tool permissions, separate tool sets for different trust levels and explicit authorization for sensitive operations.

Those are familiar ideas to anyone who has spent time around enterprise security. Agents make them newly important because the software using the permission is probabilistic. 

Traditional software generally follows a path engineers wrote. An agent may be choosing the path as it runs. The narrower that path is, the easier it is to reason about what can happen.

Why narrow AI agents shrink the security problem

This is the design choice we've made with Agents at Kantiv. Each Agent has a defined job, predefined instructions and a constrained set of tools. It doesn't receive general computer access and then get politely asked to behave.

Take our Tagging Agent. Its world is remarkably boring. It receives a file and a set of tagging rules through an API. It processes the input. It returns tags.

There is no email client waiting in another tab. No shell. No browser session. No mysterious button marked Delete production database.

More importantly, those actions aren't merely forbidden by the prompt. They aren't available to the agent in the first place. That's a meaningful architectural distinction.

Instructions such as “never delete a file” rely on the model following an instruction. Removing the delete capability entirely gives the model nothing to decide.

Can more capable agents still be tightly bounded?

There is a tendency to frame all of this as a choice between capability and security: either you give an agent enough freedom to be useful or you constrain it until it becomes a glorified workflow.

I don’t think that is the right trade-off. Narrow scope doesn't mean every agent has to do one tiny task.

Our Client Intelligence Agent, for example, performs several targeted web searches to research a company or contact. But it isn't handed an unrestricted browser and told to wander around the internet until it feels informed. Its searches run through a controlled retrieval API, which constrains what gets retrieved and how those results come back to the Agent. The workflow contains more steps, while the available action space remains deliberately limited.

That distinction becomes useful when evaluating increasingly sophisticated agents.

Count the steps if you want. I care more about the number of possible steps. A ten-step agent operating inside ten predefined actions can be easier to govern than a three-step agent holding a general-purpose browser, shell and write access to your systems.

Scope isn't enough: identity and access control

A tightly scoped agent can still access the wrong customer's data if authorization around it is poorly designed. So the next layer is identity and access control.

Before an Agent runs in Kantiv, the system checks the user's identity, role, group and requested resource. Running an Agent is itself a permission rather than something every account automatically receives. Write-backs into live systems require human approval, and each run leaves an audit trail showing who authorized it and what it touched.

These controls may be less exciting than watching an AI drive a browser around. But that’s fine by me. Seatbelts don't demo particularly well either. For enterprise deployment, boring controls are often the ones you want.

Where should human-in-the-loop approval sit?

“Human in the loop” is another phrase that sounds reassuring until you ask where the human actually appears.

If someone clicks Run at the beginning and the agent then performs twenty irreversible actions, you technically had human approval. You did not necessarily have meaningful human control.

Human approval is most useful at the boundary where the consequence changes.

Reading a CRM record is different from modifying one. Drafting an email is different from sending it. Preparing metadata is different from overwriting the source file.

An enterprise agent architecture should make those boundaries explicit and decide which ones require a second authorization step. The bigger the consequence, the less comfortable I am with approval being implicit.

Why auditability matters: agents will fail

Eventually, an agent will do something unexpected. Sometimes that's an attack. More often than not, it's mundane: a stale resume, a mis-tagged project, a record that was accurate once and quietly became wrong. As Troy put it in his piece on why bad data wrecks your AI strategy, a chatbot giving you a mediocre answer is annoying, but an agent using a bad answer to complete five more steps is a different problem.

But none of that is an exotic security scenario. It is normal engineering.

The practical question is whether your team can reconstruct what happened. You should be able to answer:

• Who initiated the run?
• What identity did the agent operate under?
• Which tools did it invoke?
• Which resources did it access?
• What did it attempt to change?
• Which actions required approval?
• Where did the run stop?

If those answers are spread across model logs, SaaS audit trails, browser histories and somebody's memory of what happened, incident response becomes archaeology. The execution trail needs to be designed into the agent system.

Cost is part of the blast radius too

Blast radius isn't only about data. It can also be operational. An agent caught in a retry loop can consume APIs, compute (and money!) just as surely as a badly permissioned one can modify the wrong record. 

OWASP includes “denial of wallet” among the risks associated with agent systems. The same architectural principle applies. Bound the run. Know which tools can be called, how often they can be called and what happens when something fails.

At Kantiv, agent runs use credits with the price shown before confirmation, and failed runs aren't charged. The point is to make the economic boundary of a run visible before it starts.

An agent shouldn't have an unlimited company credit card in either the metaphorical or literal sense. 

AI agent identity: what changes when agents cross system boundaries

So far, most of this sounds like application architecture: limit the tools, constrain the permissions, put approval in the right places. But the problem gets more interesting once agents start moving between systems.

Imagine an agent researching a client in one system, reading CRM data in another and then preparing an update for a third. At each boundary, the enterprise has to answer a deceptively simple question: Who (or what) is making this request?

With a human user, that problem is well understood. We authenticate them, assign a role and decide what that role can access. Agents complicate the picture.

Is the agent acting with its own identity? On behalf of the user who launched it? With the permissions of the application hosting it? And if it hands a task to another agent, which permissions travel with that task?

These are no longer edge cases.

In February 2026, NIST launched an AI Agent Standards Initiative, with agent security and identity as one of its three major areas of work. NIST has also been exploring specifically how existing identity and authorization standards can be applied to software and AI agents.

That tells us something important about where enterprise agent security is heading.

The security question is shifting from “Can we trust the model?” to “Can we reliably establish and constrain the authority of every actor in the workflow?”

That is a much more familiar problem for enterprise IT. And it gives teams evaluating agent platforms a better set of questions to ask.

8 questions to ask before putting an agent into production

The market is going to produce increasingly impressive agent demos over the next few years and agents are beginning to interact with more external systems and internal enterprise data.

When you evaluate one, look past what happens in the happy-path demo. Ask:

1. Which tools can the agent invoke?
   Get the actual list of tools. Can it search the web? Read files? Query your CRM? Send an email? Execute code? Write to a database? “It can work across your enterprise” is a capability claim, not a security answer. 
2. What permissions does each tool have?
   Tool access and permission are different things. An agent may need to query a CRM without being able to update it. It may need to read one project folder without seeing every project in the firm, a familiar concern for any AEC firm juggling client confidentiality across dozens of active jobs. It may need to prepare an email without having permission to send it. Read and write should be separate decisions.
3. Which decisions can the model make for itself?
   Not every step in an agent workflow needs to be agentic. Some steps can be deterministic. Some can be bounded by rules. Some may genuinely benefit from model judgment. Understand where the workflow is deterministic and where the agent chooses its own next action.
4. What happens when the agent encounters untrusted input?
   Ask how retrieved content is separated from system instructions, what happens when external content contains instructions aimed at the agent, and whether content from one system can cause an action in another.
5. Which actions require fresh human approval?
   A human somewhere in the workflow does not automatically make the workflow safe. Approval should sit close to consequential or irreversible actions. Pay particular attention to external communication, deletion, financial transactions and changes to systems of record.
6. Whose identity does the agent operate under?
   Can the agent access anything the initiating user cannot? Is a shared service credential used? What happens when one agent invokes another service or another agent?
7. Can you reconstruct a run after something goes wrong?
   Eventually, an agent will produce an unexpected result. The important question then becomes whether your team can explain it. You should be able to see who initiated the run, which identity the agent used, which tools it called, what resources it accessed, what changed and where human approval was given.
8. What bounds the run itself?
   Ask about retry limits, recursion, maximum tool calls, timeouts and cost controls.

There will never be a single switch labelled Make Agent Safe. Security comes from architecture: constrained tools, least privilege, deliberate autonomy, meaningful approval boundaries and an audit trail that survives contact with reality.

That is ultimately what I would look for in any enterprise agent architecture: not a promise that the agent won’t do something unexpected (because it will), but evidence that if it does, the systems around it limit what happens next.

If you’re evaluating AI agents for your firm, we’re happy to show you how we’ve approached these controls with Kantiv Agents and answer any other questions you might have. Book a chat here.

FAQs

What is agentic AI security?‍

Agentic AI security is the practice of controlling what an autonomous AI agent can do, which systems and data it can reach, and when a human must approve its actions. It focuses on architecture rather than relying on the model to follow instructions.

What does it mean when an AI agent "goes rogue"?‍

It usually means the agent took an action no one intended. Common causes are malicious instructions hidden in content it processed (indirect prompt injection), overly broad permissions, or too much autonomy. The damage is limited by the agent's blast radius.

What is excessive agency in AI agents?‍

Excessive agency is OWASP's term for an agent that has more functionality, permissions or autonomy than its job requires. It's one of the top risks in OWASP's guidance for LLM applications.

How do you secure an AI agent?‍

Give it only the tools and permissions it needs, separate read and write access, require human approval before consequential actions, and log every run. Set limits on retries, tool calls and cost.

‍