How well do agents use test/verification techniques?

DANLUU.COM

GitHub, LinkedIn, YouTube, everywhere you look there are new prompting and skills frameworks emerging. But just how good are they? Is one provably better than another? This is something that has been bugging me for a while - especially since I discovered that the trending Ponytail skill performed worse than my simple seven-word alternative. So I was delighted to see that Dan has tested no less than 26 different techniques for guiding coding agents towards effective testing practices.

Dan gave GPT-5.6 Sol agents a reasonably difficult implementation task: implement parts of Zstd in Rust from the specification. He then ran many variants of the prompt, adding instructions such as “use TDD”, “use fuzzing” or directing it towards various published prompting skill libraries.

The headline result is notable: simply naming a sophisticated technique usually didn’t improve correctness. In fact, the default agent, given no special testing instruction, performed above average.

Dan continued to explore the various failure modes in quite some detail. Generally speaking, agents performed the described ‘ritual’; they adopted the requested style, and generated copious quantities of tests. However, the tests themselves tended to be built on false assumptions (of what should be tested, which parts of the system are most risky, etc).

Some important lessons from this post:

1. Agents can produce tests much faster than humans, but test volume was never the goal (well, in many organisations it is, but you get my point!)
2. Verification needs to be adversarial, not confirmatory. One of the best results came from a skill that Dan wrote which developed tests at the point of requirements elaboration, not implementation.
3. “Use technique X” is a weak form of agent instruction.

Portal by Spotify cut my Claude Code token usage by 90%

ATSPOTIFY.COM

A few years ago Spotify launched and open-sourced Backstage, their developer portal framework, that provides a central catalogue of an org’s software, services, APIs and documentation. It has become really quite popular and is arguably the de-facto open source standard. Portal is basically a managed version of Backstage, with Spotify’s opinionated defaults baked in.

As you’re no doubt aware, the cost of our ever-increasing AI consumption is a growing concern. This blog post makes the argument that many of us use the most expensive models / modes for everything that we do. Many of these tasks, for example reading codebases, understanding, and extracting information often don’t require reasoning, so pointing a frontier model at ‘max’ setting is simply a waste of money.

Portal provides the infrastructure to allow the creation of lightweight agents, with the author creating a couple of examples for simpler tasks “read code and summarise” and “write boilerplate”, giving a claimed 90% reduction in cost and tokens. The really clever part here is that they use Claude Code hooks to deterministically use these lightweight agents, rather than rely on Claude to make that decision.

There is a lot of interest in smarter routing at the moment and is something worth looking into. I still have a few doubts, firstly, you can demonstrate a cost saving, but what about quality? Has it been compromised? Secondly, frontier labs already have routing baked-in to some of their products, will these approaches become redundant in the near future?

Native is now the future of mobile at Shopify

SHOPIFY.ENGINEERING

I’ve dabbled with mobile development on and off over the past ~15 years, having covered a wide range of platforms (Windows Phone, iOS, jQuery Mobile, Xamarin, React Native …). One recurring theme has been the pendulum swinging back and forth between native development, where you write the same app for each platform, and cross-platform, which offers a write-once alternative. With each new cross-platform technology, app developers jump on board, only to grow frustrated at the compromises, and revert to maintaining separate codebases once again.

Shopify has been through that cycle before, having adopted React Native back in 2020. It has helped them deliver a quality application on both iOS and Android, but has required an ongoing investment in the React Native ecosystem. With the ever-increasing power of LLMs, the team have questioned the old logic that maintaining two native codebases is double the effort, ultimately deciding to swing back to native.

This post describes their migration approach, which once again challenges previous wisdom. Many organisations migrated to React Native gradually, and conceivably their migration back to native could have taken a similar path. However, they opted for an AI-accelerated greenfield redevelopment.