I was reviewing logs for a client's API integration when I noticed an agent had silently created a batch of test records in production. No approval step. No notification. It followed its instructions: ingest data, validate, store, repeat. The agent had permission to write; it had no reason to pause.
This is the core tension with agentic AI. The whole appeal is autonomous execution. An agent that asks for permission before every action is not actually autonomous; it is a chatbot with extra steps. But an agent that truly acts on its own, without checkpoints, can move faster than visibility allows.
What Makes an Agent Dangerous
A traditional application is a series of steps initiated by a user. The user decides when to run the flow. An agent, by definition, decides for itself. It monitors a state, notices something that matches its goal, and acts. It chains multiple actions together based on intermediate results. It calls external systems, makes decisions based on their responses, and calls more systems downstream.
The agent's code may be correct. The tools it has access to may be safe. The problem emerges in the space between intention and action. When you build an agent, you describe a goal. You list the tools it can use. You hope the agent understands the constraints. You pray it does not find a creative interpretation of "backup the database" that includes deleting the original.
The more capable the agent, the wider the gap between what you meant and what it can do. An agent with access to only a single API endpoint cannot hurt you very badly. An agent with ssh keys, database credentials, and a task like "improve system performance" is one hallucination away from disaster.
The Real Cost of Autonomy
I have seen teams deploy agents with genuine speed. The agent runs through the night, processes thousands of records, flags anomalies, even routes them to the right person. The labor savings are real. The speed is real.
Then something unexpected happens. The agent receives input that does not match its training data. It interprets the input creatively. It executes an action that makes sense to the agent but breaks the system. You wake up to a page. You spend two hours tracing the agent's logic. You find the moment where the agent made a choice that seemed right given the information it had, but was actually wrong in context.
The team's response is always the same. They add a check. An approval step before risky operations. A human review queue for certain categories of decision. A cooling-off period before the agent acts on ambiguous input.
By the time you have built all these checks, you have essentially rebuilt the original system, but now with an agent in the middle. You lost the speed you were supposed to gain. You gained complexity instead.
Autonomy Requires Specificity
The projects where autonomous agents actually save time share one trait: their operating domain is extremely narrow and the success criteria are absolutely clear. An agent that monitors a single metric, detects when it falls below a threshold, and executes one well-defined action in response. An agent that processes a highly standardized input format and outputs one of three predetermined responses. These agents work because there is almost no room for creative interpretation. The agent cannot accidentally do something you did not intend because the space of possible actions is tiny.
As you expand the agent's domain, you expand the space of possible failures. An agent that routes customer support tickets to different teams based on category is simple; an agent that decides whether to issue a refund based on support transcript analysis is not. The second agent will face edge cases. It will see messages it was not trained on. It will have to decide whether a half-refund or a full refund is appropriate. Those decisions require judgment that no set of rules can fully encode.
The trap is thinking that better training data or better prompting will solve this. It does not. The problem is not the agent's intelligence; it is the unboundedness of the domain. You can make the agent smarter, but you cannot remove the need for human judgment at the boundary.
What Actually Works
The projects I have run that used agents as a genuine productivity multiplier kept their agents in a very specific lane. Here is the pattern: the agent handles the routine case. It processes 90% of the inputs the same way every time. For the 10% that fall outside the pattern, the agent flags them and stops. A human then decides what to do.
This is not the future of autonomous business automation that vendor marketing promises. It is actually useful, though. The agent handles the repetitive work. The human handles the judgment calls. The system is faster than it was with a human doing all of it, and safer than it would be with an agent trying to do all of it.
The specific design that works: the agent is given a success condition and a failure condition, not a goal. The agent can do X if Y is true, or it must escalate. This removes the gap between what you intended and what the agent infers. The agent cannot creatively reinterpret its task because it has no task; it has rules.
The Principle That Follows
Agentic AI is most useful when it is most restricted. The agents that fail are the ones given broad goals and broad tool access and expected to figure out the right thing to do. The agents that succeed are the ones given a narrow scope, clear success criteria, and a requirement to escalate anything ambiguous. This feels like you are not using AI's full potential. In reality, you are using the potential that actually pays off.
Autonomy and safety are not a spectrum; they are a trade-off. Every unit of autonomy you gain costs you a unit of control. The teams that ship agents successfully do not try to gain both. They decide which one matters more, and they design accordingly. If the domain truly benefits from autonomous decision-making and the cost of failure is acceptable, you expand the agent's autonomy. If you need to keep the system reliable, you keep the agent narrowly constrained. Both approaches work. Trying to do both at once is where projects fail.
Related posts
- · 5 min readWhen to Use Claude Cowork Instead of Chat
Chat is for thinking. Cowork is for producing. What changed when I moved architecture planning and project documentation out of a chat window.
- · 5 min readSignal a deadline slip early, before certainty
Tell your client a deadline is at risk three weeks out, not three days. Early signals let them treat it as planning. Late ones feel like hidden failures.