Thinking Aloud About Agentic AI

I still remember the moment I realised something fundamental had changed.

We were testing an automated network control system. Nothing had failed. Dashboards looked normal. And yet the system quietly rerouted traffic away from a link that, to everyone in the room, appeared healthy. About twenty minutes later, that same link degraded sharply. The AI had acted on something we had not yet seen.

That was the first time I felt I was no longer working with a tool that simply analysed the world. I was working with something that formed expectations about it.

I have spent much of my career working at the intersection of AI and critical infrastructure: telecoms, energy systems, and security-sensitive platforms where mistakes scale quickly and forgiveness is limited. For years, AI in these environments was reactive. It detected anomalies, raised alerts, and waited for humans to decide.

Agentic AI breaks that pattern. These systems do not just observe and advise. They decide and act, often before a human realises there is anything to respond to.

From Reacting to Anticipating

In telecoms, agentic systems now reallocate spectrum or spin up network functions ahead of congestion. In energy systems, they forecast demand or renewable variability and adjust control strategies early. In security, they isolate assets or alter access pathways based on inferred intent rather than confirmed attacks.

The value is obvious. Humans cannot operate at this speed or scale. But something subtle happens when systems move from reacting to anticipating. They stop acting purely on facts and start acting on beliefs about the future.

And beliefs can be wrong.

When an AI throttles traffic because it predicts congestion that never arrives, who absorbs that cost? When an energy system sheds load preemptively, how do we explain that decision to those affected? When a security agent blocks access because behaviour looks suspicious rather than malicious, how easily can that decision be unwound?

These questions became real for me during the MIMER project.

What MIMER Forced Us to Confront

MIMER sits at the boundary between identity verification, security, and critical digital infrastructure. The problem we were tackling was simple to state but hard to solve. Remote identity verification systems are increasingly vulnerable to sophisticated spoofing, including deepfake-driven impersonation. These systems underpin access to financial services, telecom onboarding, and essential digital platforms.

Static rules do not survive long in adversarial environments. Attackers adapt too quickly. So we explored adaptive approaches, including agentic components that could observe evolving attack patterns, reason about intent, and adjust verification strategies dynamically.

What surprised me was how quickly uncertainty became operational. Confidence scores that work well as decision support become fragile when agents act on them autonomously. Humans can contextualise borderline cases. Agents tend to treat thresholds as truths unless carefully constrained.

We also learned that explainability degrades as agency increases. When an agent chains perception, memory, and action, reconstructing why a decision was made becomes genuinely difficult. In regulated environments, saying "the model decided" is not an answer.

MIMER made one thing clear to me: Agentic AI is not primarily a modelling challenge. It is a governance challenge that happens to involve models.

The Trust Problem

Trust behaves differently in critical infrastructure than it does in education or consumer technology.

In education, we worry people trust AI too much. In critical systems, engineers often trust it too little. That instinct is healthy, until it undermines the very benefits autonomy offers.

During MIMER trials, some operators wanted to veto every decision, reducing the agent to a sophisticated alarm. Others were tempted to step back entirely because early performance looked strong. Neither felt safe.

What we lack are good ways to calibrate trust over time. Not blind faith and not permanent scepticism, but situational trust that adapts to context, risk, and uncertainty. We talk extensively about accuracy and robustness, but far less about how humans actually experience supervising autonomous systems day after day.

The Accountability Gap

This is where the conversation gets uncomfortable.

When an AI agent makes a decision in a telecom network or energy grid, who is responsible? The engineer who deployed it? The organisation that configured it? The vendor that supplied the model? The regulator who approved its use?

Traditional systems had clearer accountability chains. Agentic systems do not fail cleanly. They drift, cascade, and compound. Without deliberate design for traceability, logging, and meaningful human override, accountability simply dissolves.

Security communities are beginning to grapple with this. Recent work on agentic AI threat modelling highlights risks such as cascading decision failures, untraceable actions, and overwhelmed human oversight in multi-agent systems. These risks are not theoretical. They emerge naturally when autonomy scales faster than governance.

What Keeps Me Awake

I worry less about AI making mistakes than about systems failing quietly through chains of reasonable-looking decisions. By the time humans notice, the system may already be far from its original intent.

I worry about boundary erosion. Each automated decision subtly shifts what we consider acceptable to delegate. Over time, no one remembers why certain decisions were once considered too important to automate.

And I worry about human disengagement. Supervising autonomy is cognitively demanding. The more capable these systems become, the harder it is for humans to stay meaningfully involved. We do not yet design agentic systems that respect human attention and fatigue in safety-critical contexts.

The Human Element

Despite these concerns, I am not pessimistic.

Agentic AI can make critical infrastructure more resilient and adaptive, reducing brittle, reactive decision-making. But only if trust, accountability, and human judgment are treated as first-class design problems.

An AI agent can decide how to reroute traffic. It should not decide whose connectivity matters more in a crisis without human involvement. An agent can detect identity risk patterns. It should not redefine fairness thresholds on its own.

The boundary between human and machine judgment is not fixed, and it should not be. But it must be intentional.

Where This Leaves Us

MIMER did not give me neat answers. What it gave me was clarity about the questions we should be asking more openly.

Agentic AI is already reshaping telecoms, energy, and security. The real question is not whether these systems will act on our behalf.

It is how much agency we are willing to delegate before we are no longer entirely sure who is still in charge.

I do not think we should rush to answer that. But I do think we need to keep asking it, honestly, and before the systems decide for us.