Advancing coding in low‑resource programming languages
OPENAI.COM
LLMs are created by training models on vast datasets, gaining their programming expertise from the millions of software projects that exist on GitHub and elsewhere on the internet. Programming languages are not represented equally in this dataset, while there are 37,500 TypeScript repos with more than 100 stars, there are just 19 COBOL repos! This gives the models precious little data to train upon. The problem is, while languages like COBOL are not terribly popular, they are important. OpenAI notes that “COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions”.
This blog post shares the results of a couple of new benchmarks (yay, more benchmarks!), one for recreating a SQLite database, and the other, potentially more interesting, evaluating COBOL migration capability. Given that this is an OpenAI blog post, you can guess the results: they show significant jumps in capability for more recent models.
I am very impressed at how capable these models are on languages such as COBOL, where the training data is quite minimal, although I’d expect OpenAI (and others) will have struck deals with organisations that have large COBOL applications to allow them to use that for training. To me, these results demonstrate that models have an inherent understanding of programming, and have generalised, which allows them to perform very well in environments where they lack significant training data.
Developing Taste
EMILKOWAL.SKI
As we transition to a world where AI writes most of the code, each of us needs to consider where we now add value into the software development process. One thing I find reassuring is that while it is now easier than ever to create a one-million-line application, building the ‘right’ thing hasn’t got any easier. The skills required to shape a product and make the right decisions and compromises are now more valuable than ever.
A simple term I see people using more often these days is ‘taste’, a phrase that describes the ability to make product design decisions in an instinctive fashion. But what is taste? And how do you develop it? In this brief post Emil gives a few pointers. I’d also recommend the essay by Paul Graham (Y Combinator founder) from more than 20 years ago, Taste for Makers.
An honest review of Matt Pocock’s Skills framework
SCOTTLOGIC.COM
Most of the frameworks and skills I’ve tried for AI engineering have disappointed. SpecKit mostly emitted excessive markdown, and the ‘viral’ Ponytail Skill did no better than asking an agent to “follow the YAGNI principles” (its creator has since improved their benchmarking). I now favour my own Skills, or those of my colleagues.
Last month I listened to Gergely Orosz (Pragmatic Engineer) interview Matt Pocock on his podcast and found the conversation fascinating. Matt’s approach to Skill design is really interesting, and the interview is well worth listening to in full. One specific point that jumped out at me was his consideration of an agent’s “priors”, the inherent knowledge gained through training, and how prompting can unlock latent behaviour. Given that Matt is the author of ‘skills’, possibly the most popular agentic engineering skills framework (as of today, 294k GitHub stars), so I thought it was worth trying out his framework to see if it could change my perspective.
In this blog post I put his framework through its paces, with decidedly mixed results. I was impressed by the simplicity, and it sparked a few new thoughts and ideas, but the app it helped me create disappointed in a few ways. More importantly, it caused me to think about judgement and how and when I yield control to an agent.