On learning programming in an age of LLMs
PLOEH.DK
As we move to a world where AI writes much, if not all, of our code, this requires a significant shift in the human skills needed to engineer good quality software. At the moment, these skills tend to be those that we acquire when we’ve been working on software projects for a number of years, often learning from our past mistakes!
I think one of the most urgent questions we need to answer within our industry is how we teach the next generation of software engineers the skills they need for the job. How do they acquire the skills that took us five years to develop in a much shorter timescale?
Mark Seemann has been writing about software engineering for a number of years, and a recent letter from one of his readers prompted this post. He starts with an important caveat:
“None of my answers I consider particularly rigorous; the situation is so uncertain that I can only answer to the best of my abilities, but I don’t claim them to hold any kind of immutable truth.”
Some of the most wise and experienced people in our field are struggling to answer these questions.
This post is a good read: Mark’s evolving thoughts on the subject, rather than a direct and confident answer. My thoughts are in a similar state of flux.
F**k it, make it anyway
JOELOTTER.COM
Joel is an indie game developer who clearly loves his craft, as I am sure many indie devs do. As well as creating games, he writes scripts, utilities and ‘stuff’, just for the fun of it (and a little bit of praise from his colleagues). However, that has all changed for him. He is quick to acknowledge that LLMs are generally very capable now, however, they have drained the fun out of his work.
“even disregarding all the environmental and societal issues, I simply do not enjoy programming with a code assistant. It isn’t fun for me, the output doesn’t feel like mine, and I take no pride in what it produces.”
As a result, his productivity has dropped as the motivation has drained away.
And that is what brings us to the simple and punchy title of this post. Joel realised that a more positive attitude is simply “f**k it, make it”, and more importantly make it the hard way, make it the fun way, write it by hand.
While I personally enjoy building with AI … mostly … there is no reason why you all have to build with AI too.
HarnessTax: How Much Does the Harness Matter for Coding Agents?
GITHUB.IO
Firstly, a bit of terminology … the term “model” refers to the AI itself, typically a hosted Large Language Model (LLM), which receives text-based messages, reasons for a bit, then emits text-based responses. The harness is a local runtime that allows a model to invoke “tools”, for example, web searches, editing files and executing scripts. The harness is what turns a model into an agent, a tool that is able to pursue complex goals, iteratively, through the use of tools and reasoning.
Given the important role of the harness in turning a model into an agent, it is fair to ask: how much does the harness matter when reviewing the overall performance of a coding agent? Most discussions on agent performance tend to focus on the underlying model.
This team from UC Berkeley have tested 21 model–harness pairs on a couple of standard benchmarks (SWE-bench Lite and Terminal-Bench 2.0), finding that harnesses affect cost more than correctness, that simple harnesses can be highly competitive, and that models can perform competitively outside of their own harness.
How To Write With An LLM
SOCKPUPPET.ORG
While this newsletter is primarily about using AI for writing software, many of us in the industry also write words for human consumption too! This blog post shares a few thoughts one author has on how to use LLMs for (quality) writing.
Their opening stance is pretty clear: “Readers can detect LLM words in the parts per trillion”, and that’s quite true. It is incredibly easy to spot AI writing, even if it has been post-processed with a suitable skill to suppress some of its more noticeable traits. Therefore, you cannot use an LLM to write for you!
The post describes how you can use an LLM to support the review process, from helping re-structure or fine tune the flow of your writing, to more detailed aspects like word choice.
I certainly agree with these observations, and for this newsletter every word is written by me. There is another important point that can be succinctly described as “writing is thinking”. The act of articulating your thoughts on a subject really solidifies your own understanding.
However, while I agree with the points raised in this blog post, there are a number of instances where I will allow AI to ‘hold the pen’. If I simply want to communicate something (I fully understand) in a reasonably factual way, I will happily let AI write the first draft, then review and tweak the text myself.
Writing is much like software, you have to use your judgement to determine when to write with care, thought and consideration, and when vibe-writing is suitable!