We should be more ambitious with the software we build
Since early this year, Michael Malis (of pgrust) has been showing me interesting things he’s done and saying
[W]e should be more ambitious1 with the software we build
There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn’t know Malis and didn’t look into it, I would just assume that pgrust was another vibe slopped project that’s never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you’ll see that in testing pgrust, he’s testing thoroughly enough that he’s even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader2, this money overflow bug, etc.
This isn’t to say that pgrust is bug free3, but it does indicate that there’s a substantial amount of testing happening and, even if pgrust itself is never successful, just making the attempt will have improved Postgres in a durable way. The main pitch for pgrust is actually performance, and that’s where the really interesting work is happening; if pgrust fails, it’s possible that some of the optimizations they’ve demonstrated will make it into Postgres, Clickhouse, etc., but that’s obviously less direct than filing Postgres bugs which get fixed.
That being said, I find it understandable that, every time I see pgrust discussed on a news aggregator, most people dismiss it as vibeslop garbage because almost all of these rewrite projects are slop. If I didn’t know Malis, I would make the same assumption, because who has the time to look into all of these projects? Because I know Malis, I didn’t bother to look at it; I just trusted him (I only found the above because I wanted to check on pgrust before using it as an example for this post).
As a consumer of software, everywhere you look, there’s slow software that the creator is claiming is fast because the creator is touting some slop benchmark, e.g., here’s one of multiple examples I saw yesterday, and of course low-quality software that doesn’t work is more common than ever.
But, as a builder of software, it doesn’t matter that almost all of these projects are vibeslop, that other people are out there shoveling out huge piles of crap and calling it gold. You can build things like pgrust or other projects that would’ve been too large to really be feasible in the past and, if you do a reasonable job with testing and benchmarking, that stuff can actually be good-enough quality to use.
If I were writing it, I would say “we can be more ambitious with the software we build” and not “we should”, but Malis has been casually chatting with me when he’s said this, so maybe he would say the same in an actual post; don’t read too much into should vs. can here. What’s interesting to me is how capabilities have advanced such that we can do a lot more.
Just for example, after hearing that OpenAI was going to use Cerebras for their SOTA models and not just fast-but-low-quality models like 5.3-Codex-Spark, I looked at where time on my computer would go if I was using a SOTA model, but 10x to 100x faster; some things that were already sometimes a bottleneck that would only get worse with a faster model were:
At the level of a “works for me” project, these are all things that can seriously be improved. I don’t have much spare time at the moment but, even so, I built this regex engine that can be tweaked to be faster than other general-purpose regex libraries on any particular workload I care about, this ripgrep fork that uses the native code compiler from the regex engine and is faster for my codex ripgrep workloads (of course a text index is the better solution, but I’ve already worked on text indexing and this was a “just for fun” project where I wasn’t trying to do the optimal thing), and a Rust that sped up builds for my projects (huge improvement for cold builds, moderate for warm, limited by wanting to avoid forking the Rust compiler front-end). For syscalls and filesystem overhead, just switching development from Mac to Linux improved this a lot, and of course this is another thing that can be sped up a lot as well, especially when agents are hammering lots of small filesystem operations (it’s not uncommon to see my machine waiting on filesystem ops because the standard OS filesystem interface wasn’t designed with modern disk performance in mind). With modern disks, there’s no reason to use the *nix filesystem API. In the past, writing a developer toolchain that uses direct I/O instead of normal filessytem operations or building a custom OS or a funny layer that fakes out syscalls would’ve been an absurd endevaor, but that’s totally doable now.
For someone like me who doesn’t know anything about regexes, string matching algorithms, compilers, interpreters, etc., this set of projects would’ve taken a year or more to get to “works for me” quality. Instead, this was a low single-digit number of hours of my time. If I wanted to make any of these things meet a “useful product or open source project” quality bar, that would be a lot more work0, but it’s still pretty incredible how much this process is sped up.
My read of Malis’s meaning when he says “we should be more ambitious with the software we build”, is that it doesn’t even occur to people to do these things that are now easy. For weeks, every time I logged into Twitter, I saw someone complain about how Rust compilation is slow and is limiting development velocity (I think this wasn’t actually true when I saw the complaints, in that this was before 5.6 Sol and 6 Astra were available on Cerebras, and if you measured where the time actually went, Rust compilation time wasn’t actually that much of a bottleneck at the time). Sure, Rust compliation is slow. You can just write a Rust interpreter as well as speed up compilation directly. There are problems like this which I see people complaining about daily that are solvable without a lot of time or effort now. As far as I can tell, these problems are problems because people are implicitly carrying around a pre-agentic cost model. Ten years ago, the kind of person who implemented a new compiler because their compiler was slow was the kind of person who started programming a new 3d game engine when they wanted to make an indie game, i.e., the kind of person who never ships anything. Most productive programmers I know still have an aversion to this kind of thing, even though it is now actually reasonable to implement this kind of thing now4.
Another project I previously mentioned was this Azul AI which is superhuman at matched time controls. When I did that (with GPT-5.1/5.2), with zero AI knowledge, it took about 5 hours of my time to get something superhuman, and about 20 hours of my time to become what appeared to be the strongest AI at the time, including being stronger against humans than this AI that was used to cheat on Board Game Arena. Gabriel Cangussu recently built an AI that demolishes my AI, and AFAICT, it took him substantially less time than my AI took me. From talking to him about it, one key difference is that Fable 5.1 and Opus 5.5 have a much better understanding of board game AIs. When I used GPT-5.1/5.2, pretty much every suggestion they had was wrong and I had to do a lot of steering to get things to be remotely reasonable. If I had GPT read a codebase or a paper to implement relevant techniques from the paper, it would fail. If I read the paper myself and named a technique, it would say it did the technique, but it would generally do something that superficially looked like the technique but wasn’t the actual technique until I issued some corrections. When the model is that far from being able to do the right thing, if you have no background in the area, it’s fairly time consuming to check if the model did the right thing since you have to learn enough about what it’s supposed to do to be able to check that it did what was claimed.
When I did that project, I said to friends that I didn’t think the state models were in could last. It seemed impossible that they wouldn’t get good enough to be able to read a paper and implement the techniques soon, and it seemed like they should also be able to pick out codebases and papers to crib from as well. From my experience messing around a bit with Fable 5 and Cangussu’s experience with Fable 5.1 and Opus 5.5, that seems to have happened, which is a huge time saver for someone like me who doesn’t know anything about board game AIs (I know a little bit now, but still less than any real AI researcher)5.
For any particular domain, like board game AIs, you can say, “who cares? Board game AIs don’t matter”, but I’ve found this increase in capability to be true in quite a few different domains. Just for example, I did this toy eval of verification and test techniques with GPT-5.6 Sol implementing zstd on medium and xhigh. I haven’t had time to write it up, but when I published that post, I also re-did the evals with GPT-6 Astra. 6 Astra was so good that I had to find different evals because models had advanced too much. For zstd, 6 Astra succeeds at passing all of the registered tests more than 99% of the time across all conditions, even the ones that significantly impair performance, like using TDD (and, BTW, it does it more cheaply as well, taking about $1 to implement zstd). I ended up using the IMAP RFC, which 5.6 Sol couldn’t do effectively, and Pandoc, which is a pretty major thing to implement as it needs to be able to read and write many different formats6.
BTW, when people have said they used the new model and everything just works, software is solved, etc., I haven’t found this to be true at all. It’s true that there are a lot more things the newer model can one-shot, where things basically just work, but this just means that I end up working on more ambitious things where models still struggle7. I’m more productive and can work faster, but I find that, in terms of what I’m doing, I get farther from “it just works” with each more capable model because the gap between what the model can verify effectively without significant handholding and what the model can implement has been increasing8. All of that is a long-winded and less eloquent way of saying: we should be more ambitious with the software we build.
Thanks to Laurence Tratt, Yossi Kreinin, Alex Kolovsky, and Michael Malis for comments/corrections/discussion.
I’ve had this note on my recent posts, which also applies to this one. In order to get more posts out, I’m trying to crank posts out in half an hour and not review them much. This definitely means less well-supported posts. If I were writing this in my standard style, I would’ve had comprehensive benchmarks for all of my projects that I mentioned. Instead, I did a quick comparison that satisfied my curiosity and am writing this up without rigorous benchmarking, and there are maybe twenty or thirty things in this post that I’d like to do but haven’t done.
I haven’t even written up this Rust interpreter. If, five years ago, I’d cobbled together a Rust interpreter that got a 50x speedup on cold compiles and 2x on warm/cached builds (with much more being possible if edits to the Rust compiler were in bounds), that woudl be a super interestin thing that I’d want to write up, but even operating in this mode where I barely edit posts and just blast them out, this isn’t interesting enough to write up because it’s so easy to do things that would’ve been considered very interesting not all that long ago.
Here’s a comment from Yossi Kreinin, quoted (with permission, of course):
If you publish this I will publish a response explaining that for most people they should be less ambitious with the software they build with AI or they’ll experience abject failure. I think a lot of the problems don’t lend themselves that well to “agentic loops” since you don’t have clear pass/fail criteria, and even when you do most people have trouble reasoning about giant black boxes which they can impose constraints on but can’t understand the innards of [I include myself in this set.] This is a little bit like microservices I guess, where some people both need it and are very good at using it but likely for most people, they need it less and can’t use it well at all. I sort of envy the ones who should and actually can be this ambitious about the software they build of course; not saying you shouldn’t be more ambitious at all but moderating said ambition is a big part of handling the current situation for most I think
I have a similar thought about “there’s no reason for software to be slow” BTW; in many cases, if you use AI to speed up some flow, you will add a bunch of messy code that will be overfit for the use case