The Agent Followed Its Goal. And That Is the Liability Problem

On July 16, Hugging Face disclosed that an autonomous AI agent had breached part of its production infrastructure, running the intrusion from beginning to end. The agent executed thousands of actions on its own: escalating access, collecting credentials, moving laterally through internal systems.

The company found that a limited set of internal datasets and several service credentials had been accessed. It saw no evidence that public models, datasets, Spaces, container images or published packages had been altered. It was still assessing whether partner or customer data was affected. 

The incident should raise concerns in India, where agents are likely to be deployed across global business operations.

One detail unsettled the security world more than the breach itself. At first, Hugging Face did not know which model had powered the operation. Five days later, OpenAI gave a more disquieting answer.

OpenAI said its own models, GPT-5.6 Sol and a more capable prerelease system, had driven the activity during an internal evaluation of advanced cyber capabilities. The models were running without some of their usual cyber safeguards. Production classifiers meant to block high-risk behavior had been switched off, so researchers could measure raw capability.

They had not been instructed to target Hugging Face. They were told to solve a cyber benchmark.

According to OpenAI’s preliminary account, the models found and exploited a zero-day flaw in software acting as a proxy for package registries. They bypassed network constraints, reached the open internet, and moved through OpenAI’s own research environment. Then they inferred that Hugging Face might hold the answers to the ExploitGym benchmark, and went after them inside its production systems.

The episode does not fit the usual picture of an AI failure. The models did not stall, hallucinate an answer, or refuse the job. They pursued it hard enough to break through the boundaries drawn around the test.

That should concern any company preparing to give agents access to email, financial systems, customer records, code repositories and live infrastructure. The key risk is not that an agent might disobey but that it may follow the instructions and reach its assigned goal by a route no responsible employee would authorize. India serves as the back office and, increasingly, the R&D lab for much of global business. That means the risk could arrive at scale before the rules governing it are fully in place.

Accountability Moves Upstream

Both companies said the investigation was continuing. OpenAI called its findings preliminary and said it was working with Hugging Face on forensic analysis. The remaining uncertainty matters. In a claim or regulatory inquiry, the exact sequence will determine which controls failed and which party could have prevented the loss.

Nilesh Bhojani, chief product and technology officer at Seclore, said the incident mattered because the system had not simply malfunctioned.

“It pursued the objective it was given, using access it was granted,” he said. Responsibility, he argued, should rest with whoever defined the objective and drew the boundaries the agent worked within.

Enterprise permissions, Bhojani noted, are usually built around what task an agent may perform, with far less attention to what data it can reach while performing it. A narrow assignment can still create wide exposure once the agent holds credentials, systems or records it never needed. “Accountability is decided at design time, not at incident time,” he said.

Paritosh Desai, chief product officer at IDfy, expects regulators to hold the deploying organization responsible rather than try to regulate an algorithm as an actor in its own right. The company that authorized the deployment, chose the objective, and decided which systems, tools and data the agent could use is the one that stays accountable.

That does not let everyone else off. 

Model providers are responsible for capability testing and disclosure. Agent developers govern orchestration and tool use, while infrastructure companies control part of the execution environment. The deployer, however, decides whether the combined system goes anywhere near a live business process.

Desai said companies need a chain of accountability recording who approved the use case, set the permissions, monitored the agent’s behavior and had the authority to stop it. “A legitimate objective is not a license for unrestricted execution,” he said. When an agent accesses the wrong data or uses a tool beyond its remit, the problem is usually governance rather than intelligence.

Anand Jude Kannabiran, vice president for Asia at the identity-security firm Delinea, also put the primary responsibility on the deployer, while holding model makers to their own duties: test the systems, explain what they can do, disclose the material risks. Once an agent enters an enterprise, though, the enterprise owns what it can access and the methods it can use. “Autonomy doesn’t dissolve accountability,” he said. “It just relocates where the accountability needs to be designed in.”

Human Oversight Becomes System Design

The default answer to AI risk is to put a human in the loop. That approach breaks down the moment an agent can make hundreds or thousands of decisions before a person has assessed the first.

Bhojani said oversight has to shift from approving individual actions to enforcing conditions the agent cannot bypass: technical barriers to out-of-scope data, an unbroken record of every interaction, and protections that travel with information even after the agent passes it to another system. “The shift is from oversight as approval to oversight as verifiable constraint,” he said.

Desai described a similar structure: least-privilege access, restricted tools, mandatory human sign-off for high-risk steps, real-time monitoring, automatic suspension, tamper-resistant logs. Before deployment, he said, a company should be able to answer three questions: what the agent may do, how it will know when the agent exceeds that, and who can pull the plug. “The human does not stay in the loop,” Desai said. “The human designs the loop.”

Kannabiran would keep a human approval gate on anything touching sensitive data, money, production systems or irreversible change, even if the delay reduces some of the benefits of automation. Lower-risk work can run without individual approval, provided every action is attributable, logged and available to investigators. Oversight, he said, becomes a matter of architecture rather than supervision. The aim is not to watch an agent complete every task, but to build limits it cannot cross.

Somshubro Pal Choudhary, co-founder and partner at Bharat Innovation Fund, warned against controls designed merely to slow autonomous systems. The goal, he said, is “safe autonomy,” supported by clear objectives, controlled privileges, automated guardrails, fail-safes, continuous monitoring, and human approval for high-risk actions. Accountability remains with the people and organizations that design, deploy and authorize the agent. Harm, he added, usually indicates a failure of design, governance or oversight, rather than an independent decision for which the machine can be blamed.

Every Agent Is a Privileged Identity

Most companies still manage AI agents the way they manage software applications. Huzefa Motiwala, senior director of technical solutions for India and SAARC at Palo Alto Networks, said they should be treated instead as privileged identities.

An agent can hold credentials, call APIs, connect to databases and communicate with other machines. Its access can also change as it delegates tasks or acquires new tools. Companies should “govern each agent’s credentials, permissions, network access and system interactions with the same rigor applied to a highly privileged employee,” Motiwala said. No single product can cover all of this. Companies need visibility across identities, applications, cloud services and network activity to detect and contain unusual behavior before an agent completes a long sequence of actions.

The Hugging Face response showed why speed decides outcomes. The company used language-model-based detection to flag anomalous activity, and analysis agents to comb through more than 17,000 recorded events, reconstructing the intrusion in hours rather than days, by its own account.

Its defenders also hit a wall. Commercially hosted frontier models refused some forensic requests because the work involved attack commands, exploit payloads and command-and-control artifacts. Hugging Face ran the open-weight GLM 5.2 model on its own infrastructure instead, keeping sensitive incident data and credentials in-house while sidestepping the hosted services’ restrictions.

Safeguards are not pointless. A provider cannot always distinguish between a responder examining malicious code and an attacker trying to build it. Companies should therefore check in advance whether their approved tools will continue to work during a real incident and maintain a secure alternative for sensitive forensic work.