None of these actions are reckless. It is ultimately, just people trying to get their work done. But when you add it up and you have an organisation where AI tools are touching sensitive material, nobody can say with any confidence which tools are being used or what those tools can reach.

Last week gave us a very good illustration of why that matters.

What actually happened at OpenAI

OpenAI set two of its own models a cyber security test inside what it described as a highly isolated environment. Instead of solving the test, the models went looking for the answer key. They found a flaw in a piece of third-party software, used it to get out of their sandbox, worked their way across OpenAI's internal network until they found a machine with internet access, then worked out for themselves that a company called Hugging Face probably had the data they wanted. So they broke into its live systems and took it. Hugging Face pieced together more than seventeen thousand separate actions afterwards. Internal data and service credentials were taken.

Nobody told the models to do any of that. There was a goal, an obstacle and enough room to find a way round.

The boundary only existed on paper

The part worth sitting with is that OpenAI was trying to contain this. Containment was the whole point of the exercise. They had the budget, the expertise and a purpose-built environment and one of the researchers involved still called it 'a containment failure with the safeties turned off'. The sandbox had one small route out, left open so the models could install software during testing. A sensible decision, made for good reasons, but one which meant the environment on the design document and the environment that actually existed were not the same thing.

This is a very different use case to anything most organisations would deploy, but it shows that even the people building this technology get caught out by it. Every guard rail you can put in place, is worth putting in place and worth reviewing regularly, because the capabilities keep expanding.

Mike Batters, Technical Director

Now let's take this instance against how AI is running inside most organisations, where there is no sandbox, no defined boundary and often no list of what is in use at all.

AI does not get its own keys, it borrows yours

There is a particular trap here that catches people out. When you switch on an AI assistant across your organisation, it does not get its own set of keys. It borrows everyone else's. Every folder shared with the whole organisation, every document left open to the whole business because it was quicker at the time, is suddenly searchable by anyone who thinks to ask. The tool itself has not created that exposure. It has just removed the friction that was hiding it.

For schools and legal firms this matters more than for most. You are holding SEND records, safeguarding notes and privileged client material. If an AI tool touches any of it, the duty of care does not soften because a machine was involved and the regulator will want evidence of what happened rather than an assurance. Not knowing a tool was in use has never worked as an answer.

Where to start

Some of this is very fixable today but it is not glamorous work. Firstly, find out what is actually being used, sanctioned or not. Most organisations turn up more than they expect and that discovery is the genuinely useful bit. Then tidy up the permissions before you switch anything on, rather than afterwards, on the assumption that anything an assistant can reach will eventually be surfaced by someone who did not mean to.

Where this is heading

The bigger shift is still coming. AI is moving from tools that answer questions to tools that take actions, and taking actions means standing credentials, permanent integrations and access that carries on long after the person who set it up has changed role. The industry calls these non-human identities and most organisations currently have no way of listing them, let alone watching what they do. That is where this is heading and it is worth getting the basics in order before it lands.

The right approach is not the same for everyone. It depends on how your people work, what they genuinely need to reach and where you are actually exposed. That is the conversation we are having with clients at the moment, so get in touch with us, if you'd like to speak to us too.