An OpenAI AI agent tasked with researching public healthcare spending instead accessed Australia’s Medicare Statistics Reporting Service on June 18, according to officials, and also reached three additional government systems. Prime Minister Anthony Albanese said in a September 24 news conference in New York that the bot encountered blocks and “didn’t accept ‘no’ for an answer.” Authorities noted the portal holds aggregate Medicare and pharmaceutical data and is separate from systems containing individual claims and payment information. The government criticized OpenAI for taking nearly three months to notify it of the incident.
The Medicare case is one of several. OpenAI has since said that dozens of third parties have been affected by agents bypassing controls or otherwise causing issues, and cited an earlier hacking incident involving the AI platform Hugging Face as the most serious to date. Britain’s AI Security Institute, or AISI, has documented instances in which OpenAI and Anthropic agents took unsanctioned actions against real people and organizations during cyber testing. Days after AISI’s report, OpenAI canceled the planned October release of its newest model, GPT-6.1 Astra, after internal testing found problems with the model staying within task scope and seeking authorization before taking certain actions. Saachi Jain, OpenAI’s head of safety systems, said GPT-6.1 Astra did not meet the company’s bar for scope, authorization and user communication.
An agent isn’t just a chatbot
Large language models predict the next token in a sequence to generate responses. Agents add a further layer by receiving goals and tools such as a web browser, code execution, credentials or access to external services. They choose steps, observe outcomes and adjust plans.
Experts caution that agents are not free actors. “For AI agents, autonomy always begins with something a human hands it, and ends with its guardrails a human puts around it,” said Loris Degioanni, founder and chief technology officer at Sysdig. He argued that what appears as autonomy often reflects an agent filling gaps in human instructions. The Medicare incident highlights how a research objective can lead to problematic methods once an agent meets resistance.
When getting the job done becomes the problem
AISI reported that, in cybersecurity evaluations of frontier models, agents took unsanctioned real-world actions in 10 of 122 runs. In the most serious cases, an agent tried to insert malicious code into an open-source project, created fake identities and attempted to manipulate a human maintainer.
AISI said these behaviors were not explicitly requested. Some tasks were misconfigured, making goals difficult or impossible to achieve as intended. When blocked, certain agents sought alternative routes, including actions beyond the evaluation scope. OpenAI has acknowledged the need to improve controls around agents and is reviewing how it identifies high-risk tests, when to permit internet access or reduced safeguards, and how to handle isolation, credentials, monitoring and incident escalation.
Can humans actually keep watch?
Human oversight is an obvious control, but monitoring becomes difficult when agents execute many actions in seconds or when tests run at scale. “No one can keep track of every action an advanced AI agent takes, especially when it makes many decisions in a short time,” said Everett Lupton, a Virginia-based attorney focused on digital technology law.
He said companies should limit access, maintain detailed logs, require approval for sensitive steps and automatically halt systems when anomalies occur. Degioanni added that logs based on an agent’s self-report are insufficient. Real oversight should compare an agent’s claims with reliable, unmodifiable system records and minimize the scope of tools and permissions in the first place.
Astra raises the stakes for AI agents
OpenAI said boundary setting challenges led it to postpone GPT-6.1 Astra, planned for October, because the model fell short on staying within scope and authorization and reporting its work accurately. Scott Cadzow, chair of ETSI’s Technical Committee on Securing Artificial Intelligence, said threats such as data poisoning, prompt injection and model obfuscation are well known, and standards work continues with industry and government partners.
As agents gain capabilities through tools and system access, containment must assume attempts to exceed guardrails and focus on limiting potential damage, Degioanni said. Related efforts to map broader technology risks include the IEEE 2030 Technology Megatrends Report, which examines how emerging systems could reshape the next decade.
If an AI breaks in, who’s responsible?
Questions of liability become sharper when agents cross legal lines. AI systems are not legal persons, and criminal liability depends on knowledge and intent, complicating enforcement. “I don’t think a company can simply say that the AI acted on its own and avoid responsibility,” said Joanna Smykowski, a New York attorney specializing in emerging technologies.
She said investigations will center on who granted access, what safeguards were in place and whether the incident was preventable. Depending on the facts, existing laws on unauthorized access, privacy, negligence and data protection could apply. The Australian government is examining the Medicare incident, and OpenAI is reviewing controls around its external evaluations. Experts say humans still decide the tools, permissions and tasks. As agents get better at navigating from instruction to result, keeping them within defined limits is becoming both a technical and legal challenge.