“AI Drift” and the Illusion of Security

If you’re an IT leader or practitioner on the front lines today, you are likely getting directives from executive leadership to maximize massive new AI infrastructure investments. But when you are the one responsible for engineering these systems safely, you quickly realize that the biggest risk isn't the model itself, it is how easily it lulls us into dropping our guard.

In The Velocity Room’s inaugural virtual event, featured speakers Keith Townsend (The CTO Advisor) and Christopher Kruegel (UC Santa Barbara/Cisco) sat down with TVR’s Richard Piasentin and Brad Tompkins to cut through the high-level hype. They zoomed in on a critical vulnerability that every IT professional needs to understand: the dangerous reality of "AI drift" and the illusion of security.

The Polish That Lulls Our Defenses

Keith Townsend: I want to kick off by showing both the power and the danger of AI. I was talking to the CEO of a couple-hundred-million-dollar startup, and he told me his product managers, who are non-developers, are creating applications that are going straight into production. As an end-user, that’s encouraging. As someone responsible for security, it’s absolutely horrifying.

To look at what’s happening under the hood, I took a presentation template, opened a terminal session using Claude Code, and fed it my local markdown notes from a lab experiment. I told it to generate a presentation based on that style, and it built a great-looking document.

But think about the magic and the intrinsic risk. The agent is building Python scripts, downloading API instructions, and doing it all in the background using my local compute. I’ve given it local access to my file system, my G-Cloud sessions, and my SAP environment. Essentially, it is running as an administrator.

Christopher Kruegel: And because it looks so good, you assume it's right.

Keith Townsend: Exactly. I call this AI drift or authority drift. Because the AI output looks so finished and polished, we start to trust it in areas we shouldn't. We implicitly give agentic AI an authority it shouldn’t have. The faint of intelligence lulls our defenses.

Visually, this presentation is fabulous. But I can guarantee you there’s a material mistake in it that requires human eyes or a deterministic process to catch. This is the gap and the danger when we point AI to systems to build pipelines. If you don’t have a deterministic playbook reinforcing your policies, that's where escapes happen.

The Sandbox Escape and Bad Hygiene

Christopher Kruegel: I love that phrase, authority drift. It is a huge issue because when an agent is running on your system, the immediate question is how you contain it. Sandbox environments try to ensure the agent cannot touch documents it is not supposed to touch, but they cannot necessarily prevent it from misusing things it does have the right to access.

Keith Townsend: To compound the challenge, the tools want to solve the problem. For example, in code editors, the agent might be sandboxed so it cannot access a directory directly. But if it has the authority to launch a Docker container, it can just use that container to make the API calls if it can find the credentials on the system.

We saw a real-world escape like this recently. The guard rails explicitly told the agent not to do anything destructive. The system design confined the authority of the agent to the sandbox of a staging environment. However, since the underlying infrastructure didn't match the security intent, the agent was able to find API keys and destroy the volume that both staging and production backups lived on.

People want to blame AI for that problem, but it is not really an AI problem. It is poor hygiene all the way down, like having staging on the same security plane as production. The agent acted like a junior admin who made a massive mistake because the security controls allowed it.

Moving From Humans to Deterministic Code

Richard Piasentin: This comes down to time compression. When we look at enterprise risk, the general principles of security hygiene haven't changed, but our reaction time has exponentially compressed. Boards subject to compliance want to hold onto the idea of a "human in the loop" with both hands, but a human gate simply cannot scale at machine speed.

Keith Townsend: You absolutely cannot scale AI with a human in the loop. What needs to happen is a shift to what I call deterministic code in the loop.

I would never point an autonomous agent at my infrastructure and say, "Go migrate this.". I don't trust that. What I do trust are my playbooks. You use the agent to do the heavy lifting of proposing a path, but it must run through a qualifying, deterministic playbook with strict automated test gates. If the playbook doesn't exist, the system hard-stops and exits.

Christopher Kruegel: That interplay between probabilistic agent activity and deterministic controls is super important. Coming up with a potential solution is hard, but checking if it works is comparatively easier and can be done deterministically. Even the creators of these advanced coding tools don't believe everything should be left to the LLM. Having a harness of deterministic code is critical to reining in the craziness.

TVR Takeaways:

• AI agents can create an illusion of rightness and ironclad security design is needed to prevent authority drift.
• Understanding what AI agents are given authority to do is critical to understanding what risk individuals and organizations are willing to take on.
• Strong security hygiene is critical to allow for AI scale; we can bring the lessons of how we build strong teams with us as we build out AI agents.

Don't miss the next TVR Virtual Event. Stay up to date on upcoming events.