I Joined an AI Competition With 7 Days Left. 940 Teams Had the Same Score.

Here is what happened, and what a week spent teaching a computer to read knee scans taught me about work that has nothing to do with medicine.

A global medical imaging society ran an open competition. The task: build software that looks at an MRI scan of a knee and flags twelve common problems. Torn ligaments, torn cartilage, arthritis, fluid on the knee, fractures, that sort of thing.

About 25,700 people entered, in roughly 6,000 teams, over two months. I joined with seven days left, working alone, on the free cloud computers Kaggle hands out each week. Total spend: nothing. Total compute: about six hours.

Then I looked at the leaderboard and found something strange. Around 940 teams were sitting at exactly the same score. Not similar. Identical. They had all downloaded the same publicly shared solution and run it unchanged.

Catching the actual leaders in seven days, alone, was not going to happen. But the competition had a second prize, for models that are both accurate and fast, and almost none of those 940 copied solutions were fast.

My model scores 0.919 on the public leaderboard and reads the whole test set of roughly 1,300 knee scans in about nine minutes. Under half a second per scan. The popular public solution took a few hours for the same job.

What does 0.919 mean in plain English? Show the model two knees, one with a given problem and one without. About 92 times out of 100 it rates the problem knee as the more suspicious of the pair. It is a ranking measure, not a diagnosis.

The leading models score around 0.96. So I am a few points behind them, running in minutes instead of hours.

Whether that wins anything is not settled yet. The week was worth it regardless, because almost everything I learned transfers.

Here is the part that surprised me most.

The training set had about 4,400 knee scans. Only 58 of them had been properly labelled by expert radiologists. Fifty-eight. That is nowhere near enough to teach a computer anything.

But every single scan came with something else: the radiologist's written report. The actual notes a doctor typed after looking at it. In several different languages, because the data came from more than twenty hospitals across Europe, the Americas, Asia, Africa and Australia.

People in the competition community wrote code to turn those free-text reports into usable yes/no labels for each condition, and published the results. I combined fifteen of those label sets into one.

A dataset of 58 became a dataset of 4,400, and it came out of writing that already existed.

The lesson: your organisation is sitting on years of emails, case notes, support tickets, incident reports and meeting minutes. That messy pile of text is very often the dataset you have been telling yourself you do not have. You do not need to re-interview every customer if you already have a decade of their complaints.

Nine hundred and forty teams at an identical score is not a competition. It is a queue.

Fighting for a fractional improvement at the front of that queue, with seven days and one person, was hopeless. But the organisers had quietly set a second scoring rule, one almost nobody in the queue was optimising for.

The lesson: in a crowded market, the opening is rarely a better version of what everyone else is doing. It is usually a different definition of winning. Look for the scoring rule nobody is playing by.

A model that is slightly less accurate but reads a scan in half a second can sort an entire morning's queue before the radiologist has finished their coffee. The likely fracture goes to the top of the pile. The routine follow-up goes to the bottom. A near-perfect model that takes hours cannot do that job at all.

Think of a triage nurse. Their job is not to diagnose every patient correctly. It is to make sure the most urgent person is seen first, quickly. Different job, different tool.

The competition actually priced this trade-off into its formula: one hundredth of extra accuracy was worth about eleven and a half minutes of runtime. I think being forced to state that exchange rate out loud is a useful exercise for any organisation.

The lesson: before optimising anything, ask what the output is for. Sorting a queue and making a final call are not the same requirement.

I assumed the AI would be the bottleneck. It is the complicated bit, so it must be the slow bit.

The rest of the nine minutes went on opening files. Over 800,000 individual image files, each with a small block of information attached describing what it is and where in the knee it sits. Just reading those well enough to put the slices in the right order was the real cost.

Reading only the four pieces of information I actually needed, instead of everything, and opening many files at once instead of one after another, is where the speed came from. None of that is clever. All of it mattered.

The lesson: measure before you optimise, because your intuition about where the time goes is probably wrong. In most organisations the delay is not the expert's decision. It is the paperwork wrapped around the decision.

Near the end I blended my model with the strong public solution, expecting the combination to beat either one alone. That is usually how this works.

The score went down. From 0.956 to 0.952.

A noticeably weaker voice added to a strong panel did not add balance. It added noise.

The lesson: adding people to a decision helps when they bring something the group lacks, at a comparable standard. A weaker duplicate of an opinion already in the room makes the room worse, not more democratic.

With only 58 expertly labelled scans to check against, any score I measured on them could be off by around three points in either direction. Most of the improvements I was testing were smaller than that.

So the test could not tell me whether my changes worked. I relied on a rougher but much larger check across all 4,400 scans instead.

The lesson: do not judge a new hire on one week, a campaign on one day, or a supplier on one order. A small sample gives you a clear, confident, meaningless answer.

Several popular solutions used outside medical datasets whose licence terms were unclear for this competition. Using them would almost certainly have raised my score.

I built only from the competition's own data and from starting points with clear open licences, and kept a record of where everything came from. That cost me points.

The lesson: in healthcare, finance, law or anything regulated, where your data came from is part of the product. A lower number you can defend beats a higher one you cannot.

The knee is the most commonly injured and imaged joint in the body. Osteoarthritis alone affects an estimated 654 million people, and acute knee injuries are somewhere between 15 and 40 percent of all sports injuries.

Specialist musculoskeletal radiologists are scarce, much more so outside major cities and wealthy countries. Scans wait. Readings vary between doctors.

A tool that reads a scan in under a second on modest hardware is the kind that can actually be deployed in a hospital with one radiologist and a long queue. The heaviest, most accurate model in the world is no use if nobody there can run it.

An important caveat. This is a research competition model. It is not a medical device, it has not been clinically validated, and it does not diagnose anything. At best it helps an expert decide what to look at first.

A week, one person, free computers, open-source tools, and a community that published its working. Ten years ago this needed a funded lab.

The barrier is no longer access to the tools. It is clear thinking about which problem to solve, and what "good" means for the person who has to use the result.

I could not have done this in seven days without the labels and tooling other competitors published for free. I have published my code, models and notebook in return.

Credits: to RSNA and Kaggle for running the competition and releasing the data, and to the competitors who published the report-derived label sets that turned a 58-scan problem into a 4,400-scan one.

For the technical reader: 2.5D EfficientNet-B0 and B2 ensemble, labels derived from the free-text radiology reports, series selected by MRI plane and fat suppression, threaded DICOM header reads pulling only the four tags needed for ordering, and horizontal flip test-time augmentation.