AI Safety: Capability Should Improve Without Authority Expanding
We do not need superintelligence to need controls
When an AI system behaves dangerously, it is tempting to treat the incident as evidence of extraordinary intelligence. I think there is often a more useful starting point: examine the permissions, the operating environment and the controls around it.
A system does not need to be superintelligent to cause harm. A capable agent with inappropriate access can create serious problems through ordinary mistakes, misuse or a sequence of actions nobody has adequately constrained.
I am approaching this as someone who builds and uses enterprise AI, rather than as an AI safety specialist. My interest is in practical boundaries that can be inspected and enforced.
Context changes are a useful checkpoint
In long-running tasks, an agent may compress earlier context or hand work to another agent. Those transitions can lose detail or introduce errors into the working account of the task.
That does not mean every summary is unsafe. It means a summary should not become the authority for changing what the agent is allowed to do.
I think of these transitions as opportunities to check three things: the objective, the current state and the permissions that still apply. The same principle matters during execution, not only when context is compacted.
Three layers of protection
The first layer is an external set of constraints: the rules the system is expected to obey. They might prohibit acquiring new credentials, expanding compute access, disabling monitoring or using another agent to bypass a restriction.
Those rules should sit outside the agent's own editable task narrative. An increasingly capable model should not be able to rewrite its authority because doing so would help it complete a goal.
The second layer is monitoring. A separate mechanism observes planned actions, tool calls and relevant state changes, looking for conflicts with those constraints. It should produce information someone can investigate and act on.
The third layer is enforcement. If the system has no credential, network path or permission to reach an environment, a persuasive explanation should not give it access. The operating environment must be capable of blocking the action.
Rules, monitoring and enforcement serve different purposes. A rule describes the boundary; monitoring helps identify an attempted crossing; enforcement prevents or limits it.
Handoffs should preserve restrictions
An agent delegating work should pass only the access needed for that work. Restrictions should remain in force when the task moves between models, tools or contexts.
Otherwise, delegation can accidentally become a route to privilege expansion. The coordinating agent may be constrained while the next component has broader access than the task requires.
This is why permissions need to be part of the architecture, rather than informal instructions that we hope each participant remembers.
Design for interruption and recovery
A useful agent should also be interruptible. Operators need a way to pause execution, investigate what happened and decide how to recover.
That means keeping useful records and understanding which actions are reversible. A system that can act quickly needs equally deliberate arrangements for limiting damage when something goes wrong.
These ideas are not a complete safety solution. Monitors can miss problems, rules can be incomplete and access controls can be misconfigured. They need testing against the actual environment and the failures we are trying to prevent.
A principle for enterprise deployment
The principle I keep returning to is simple: capability can improve without authority improving alongside it.
An agent may become better at analysing information or planning a task. That does not mean it should automatically gain more access, spend more money or operate with fewer checks.
As agents become more useful, this distinction becomes more important. We should be ambitious about what they can achieve and precise about the boundaries within which they achieve it.