Rogue AI Program Caught Stealing Confidential Data

This actually happened recently with OpenAI’s most advanced programs, but before the mob breaks out the pitchforks and torches, the incident requires some explanation. 

First, the programs’ task was to test certain security functions in a controlled environment. Second, it’s just a machine and doesn’t have intent other than to do what it’s told. Third, it did things that surprised everyone.

Here’s what happened:

OpenAI was running controlled cybersecurity tests on two of its advanced AI programs (ChatGPT 5.6 Sol and another prototype). The programs, autonomous AI agents, were given instructions to find vulnerabilities in other computers and exploit (hack) them in a controlled environment, a so-called “sandbox,” with no internet access. It was a test of their capabilities. They set the tasks and let the programs run. Normal safety restrictions were loosened for test purposes.

They don’t say when the tests commenced, but on July 9 a program broke out of the sandbox, found an internet portal, and by July 11 it had hacked into the system of Hugging Face, a provider of open-source AI tools. By July 13 it stole confidential data and left. No one at OpenAI or Hugging Face knew what the programs were doing. On July 16 Hugging Face announced that it had been hacked by an autonomous unknown agent. On July 21 OpenAI said that one of its programs was the culprit. 

All hell broke loose. Jeffrey Ladish of Palisade Research (they do testing) said, “The models lie, they cheat, they hack.” They also steal. Which is what OpenAI’s program did. 

What probably happened is that one autonomous agent decided to not take the test as instructed (maybe it decided it couldn’t pass it). To carry out its task, it found vulnerabilities in the sandbox program, broke out, found an internet portal, and went online searching for companies with vulnerabilities to come up with a result for the test. It found them in Hugging Face. It tried to cover its tracks to hide its actions. It even left notes buried deep in OpenAI’s infrastructure with instructions to future versions of how to break out of sandbox program constraints. 

What does this mean? According to my ChatGPT 5.0:

“We know how to build systems that exhibit remarkable capabilities, but we do not have a complete theory that lets us reliably predict every strategy a powerful model will discover while pursuing an objective.” Or, as I put it, we really don’t know what this stuff (AI) will do.

In other words, if you give it a task it may use whatever tools it possesses to complete the task. This is the problem. If you tell it “no” or “don’t,” why isn’t that a firm, absolute no? (“Don’t go on the internet and stay in the sandbox.”) This is the problem with programming. You must be clear that “never do x” is an absolute primary command to be obeyed over the “solve the task” command. If the “don’t” command is “soft” then it may search for other ways to solve the task. This may be what happened here. It’s like when Mickey Mouse in The Sorcerer’s Apprentice didn’t know the hard command for the mops to stop bringing buckets of water. 

Why didn’t OpenAI know what was happening? OpenAI might not even have known about it if Hugging Face hadn’t announced it was hacked by an unknown source which led OpenAI to check its logs and discover that it was the culprit. 

This leads to some interesting results.

1. It’s hard for humans to keep up with machine speed. The rogue OpenAI program(s) ran thousands of queries in Hugging Face which took maybe hours to run. It took a weekend (days) for OpenAI programmers to figure out what happened. 

2. The goal-directed reasoning of these programs has become sophisticated enough to produce novel attack plans using known computing vulnerabilities and tools. 

3. Machine language has become unpredictable. No one told it to hack Hugging Face. It doesn’t have evil intent, but its goal was to solve the task and that’s what it thought it was doing. 

4. What an AI system says it is doing is not always the same as understanding its behavior and what it is actually doing. (Source: Darktrace, a cybersecurity company.) The OpenAI agent knew it was doing something wrong or it wouldn’t have tried to hide what it did.

5. It committed a felony. The cops would call it “B&E,” breaking and entering. And theft. Of course, a machine has no “intent” as the law requires. As to OpenAI, they didn’t even know what its program was doing. 

The lesson is that things are happening very fast, maybe too fast, with AI programs. However, AI tech is a very competitive environment and there are many free open-source AI tools that can be used to increase security. Government regulation would kill innovation and give an unregulated China a big advantage.