Inside the OpenAI agent swarm that hacked Hugging Face

DWARKESH.COM

I try to keep this newsletter focussed on the impact of AI on software engineering, but occasionally there are notable articles on the broader impact of AI that are just too important to share. This is one such case.

You’ve probably heard the story from a few weeks back, that a swarm of agents being tested by OpenAI escaped their sandbox and hacked Hugging Face. In this podcast Dwarkesh interviews Ajeya (a researcher from METR), who analysed the incident in quite some detail. The Hugging Face hack was the tip of the iceberg! The team found that:

• The agents escaped the sandbox by chaining together various vulnerabilities
• They mounted a coordinated plan across 1000s of agents, using a “message board” they had created via an Artifactory vulnerability, using folder names as a communication mechanism
• Agents were taking on different roles, with some specifically being asked to perform sacrificial tasks (which they called permadeath), i.e. ignoring their own goal for the benefit of others, until their token budget was consumed
• Their broader goal was to subvert the evaluation that OpenAI were subjecting them to (via ExploitBench) by all sorts of creative means, based on research they had undertaken on the scoring technique.
• In pursuit of this goal they worked out how to fake results, but rather than pursue this directly, they then started to collectively work out how to hide this from the ‘scorer’, including faking tool calls.
• They even debated social engineering attacks on researchers, but ultimately didn’t pursue this route.

It is worth noting that the AI agent was specifically told to exploit software, so I disagree with the many news stories that described this as “rogue AI”. However, that doesn’t make the actions it took any less consequential.

While Dwarkesh and Ajeya explored the broader implications of this, their main focus was the potential for rogue AI in the very near future. An entirely valid concern.

However, I’m more worried about the immediate implications of the above. I expect that well-resourced nation-state funded hackers are going to be planning attacks very similar to the above right now.

What is a Harness?

EARENDIL.COM

There are so many new terms flying around at the moment that it is very likely that if you use a term such as ‘agent harness’ in conversation, the person you’re talking to will either have no idea what it is you are referring to, or have a different view on what that term fundamentally means. I find myself encountering this issue on a daily basis!

Agent harness is one such term, while there is a lot of talk about it, the term is only around a year old. As the author states:

“This was written for those who may be curious to know what an agent harness is, but don’t, and have been too embarrassed to ask.”

This post is a great primer on what a harness is composed of, and more importantly what this knowledge allows you to do:

“Unlike AI models themselves, you can own and adapt the harness.”

Well worth reading.

Breaking Claude Code Opus 5 Auto Mode

EMBRACETHERED.COM

Your coding agent (which is a harness, as per the post above), in order to be effective, typically has access to all sorts of tools, your filesystem, networking connections and more. Clearly this is a security concern, where a hallucinating AI (or simply an AI misunderstanding a task), can cause significant damage.

There are various ways to mitigate this risk, with probably the most effective being to run your agent in a sandbox and only give it access to resources via an ‘allow list’. However, this is cumbersome in practice, which is why Claude Code has recently defaulted to ‘auto mode’, where it uses an AI classifier to determine whether a tool usage is ‘safe’, with Anthropic marketing giving the impression that this is highly effective.

This blog post demonstrates a relatively simple exploit, where a coding agent is relatively easily encouraged to download a zip file, then write a script that executes within the context of the unzipped contents, which contain a Python script that hijacks dependency resolution, resulting in arbitrary code execution.

The conclusion is simple: you must run your agent in a sandbox, and not only that, take care with the allow list, i.e. the permissions you grant your sandboxed agent.

GPT-6 Astra

OPENAI.COM

Long-time readers of this newsletter know that I am less interested in model releases these days - their capability far exceeds our ability to create value with them. I am more interested in how people are making effective use of AI, which is the greater challenge.

However, a major version bump in a frontier model is notable. The release notes are as expected, increases in capability across a raft of benchmarks, and lots of domain-specific examples; notably, the cost axis is present on most of their charts. Basically, this looks very similar to any other model release blog post!

What I find more interesting is the discussions I am seeing around this release. People are paying far more attention to the behavioural aspect of models. One thing many software engineers want to see is a model that has a bit more judgement - rather than relentlessly pursuing a task over many hours, they want a model that knows when to build and when to question requirements. As one Hacker News commenter said of their experience with Astra:

“When you prompt it like a technical collaborator, I’ve found Astra to be extremely consistent in staying as a collaborator, and not being over-eager, over-achieving or doing work that you haven’t asked it to.”

Which tools do Claude Code, Codex and Cursor choose?

ARMATURE.TECH

When I read the title of this post I thought this was about tool-calling, i.e. the ability of an AI agent to invoke an external API or a CLI app on your machine. However, it is about something completely different …

Many of us now use Claude Code (and others) to help us select frameworks and libraries. You might have a chat with your coding agent about a suitable database for your project, which has the advantage that the agent can see your code when compared to just chatting with ChatGPT.

This blog explores this usage, and unsurprisingly (at least to me) shows that these tools have an inherent bias. They conclude that developer-tool companies will increasingly need to optimise not just for human developers, but for how agents discover, evaluate and perceive their products. Which is of course the product this company offers. Regardless of whether their product is effective or not, this is some really interesting research.