What it took to trust a package manager Claude wrote

Posted on Oct 16 • Originally published at getjpm.sh

In the podcast the hosts were answering questions from their audience about AI, questions everyone is asking right now. Can it build real software, or just demos? Should engineers be worried, or excited? Most people believe AI can one-shot anything. Spoiler it can. The results are just not good without directing it and understanding software development.

So that afternoon after I was wrapping up my day job work I opened up a tab in my terminal and asked Claude what it would take to rewrite upm as a Rust CLI. I read over the summary and gave the ok and it started on it. I think it maybe took 45 minutes and it had a working version using a bunch of third-party Rust crates which made it much bigger and was only tested on a small package.json file. To make it smaller I had it break down what was making it so big and by the end of that evening I had a custom HTTP, JSON, crypto, and TLS module and a much smaller binary. I did direct it to only write the parts we need but also to heavily test them with other projects tests if we could or our own with some fuzz testing. I wanted to make sure they were solid.

So this is the part everyone that is "vibe coding" projects leaves out. Sure I had it working on a small package.json file running in Linux on Windows WSL2 but I had not tested Mac or Windows or any really big projects. So I had it write a script to harden it that took 100, eventually 166, large public projects and tried to install and build them. Then install and build them with the package manager they use and compare. Including running their tests where it could, as some tests needed secrets or other setup. Also do this on Windows command line and PowerShell as well as I started an agent on my Mac to do the same on the Mac. And five days later, making many calls on what to support and how, we got 261 merged PRs and a hardened and well tested package manager.

So I didn't write code but I did direct it and make lots of small calls. Like Claude saying a faster lock file format would only remove a few ms but I asked for the change anyways. Claude also wanted to make the global store an opt-in feature but I called out if 99% of packages can use it why, and it became the default. I also didn't want it running scripts including dependencies install scripts unless you listed them out as ok. My thinking here was that AI does most the work now, so before approving a package's scripts I can just ask it to review them for issues first.

There were also two times Claude actually pushed back on me but both times I insisted.

The first was TLS. I wanted to get the binary smaller, and the TLS and crypto crates were a big part of it. Claude said that's the code you never write yourself. I told it that was old engineering thinking. We are in Rust, so the memory bugs behind things like Heartbleed are off the table, and we can test our version hard just the same if not better. It agreed it had overstated the risk, laid out which test suites would prove it (Wycheproof, NIST vectors, x509-limbo, BoGo). I still question if this is a good idea or not and Claude likes to glaze you so it saying it overstated the risk could still be it just being helpful. It passes a bunch of tests including tests made from real bugs from other projects. I know I would trust it way more now regardless of who wrote it.

The second was HTTP/2. I wanted more speed and asked about it. Claude tested it with curl first and said it wasn't worth building, because jpm's whole install over HTTP/1.1 was already faster than curl's HTTP/2 download alone. I pushed anyway. curl has to handle every case, and we only need many GETs to one registry. We have to think outside the norm to beat bun. It agreed and had a working client about two hours later.

Then we measured it. On my home connection HTTP/2 was about 17% faster. On GitHub's runners it tied or lost, was up to 2x slower on a cold nuxt install, and used 8 to 15% more CPU on big tarballs. CI is where installs cost real time and money, and it added 60 KB to the binary. So I had it delete all 3,671 lines. I was wrong on that one, and the numbers told me so.

It went the other way too. jpm will install Node for you if you want and I wanted Node's signature check behind a flag because I thought it would slow installs. Claude measured it at 15 to 19 ms and said a flag just means nobody turns it on. It's on by default and I am glad as it adds security.

The short answer is Wycheproof, node-semver, pnpm, yarn and bun's own tests, and the 166 repos we tested it on. Some of the bugs they found along the way:

I also know after this many years building software nothing is bug free regardless of tests and testing. And trust is earned so I plan on using it in all my projects going forward as well as supporting fixes, even AI-submitted ones. It doesn't mean I will take them all. They need to prove they are fixing something with tests. The point is I don't plan on turning off PRs but I will be using AI to help with it.

We got the executable to 2MB and it isn't always the fastest but it is up at the top. It also is one of the most secure as well with it not running scripts without approval. Not updating packages to the latest version if they are not at least a day old. And the security around.npmrc.

Here is one of the charts showing how fast we can be on a GitHub Actions runner compared to other package managers:

Every machine, phase and project is on getjpm.sh.

I am going to be honest here the benchmark is small, only three repos. We are also not always the fastest. Like on Windows nuxt the Defender scans are killing it. On macOS bun and aube win for warm installs of nuxt. So there is room for improvement if speed is the end goal. We did run 166 project installs on Linux, macOS, and Windows, plus even more random ones on Linux to find bugs but we didn't performance test each one in CI and on machines.

Is jpm perfect? No. Neither is anything I've written by hand in 27 years, or anything any human has written. Perfect doesn't exist. There's good enough, there's great, and there's tested enough to know which one you have.

Try it with curl -fsSL https://getjpm.sh | sh on macOS and Linux, or irm https://getjpm.sh/install.ps1 | iex on Windows, then run jpm install in any project. If you find a bug, open an issue on GitHub.