I measured my AI search. The numbers were humbling 📏

In part 1 I built Because I liked: you describe something you loved and why, and it finds films, shows, books and games that match the why, then explains each pick.

Part 1 ended with "the misses are the most useful thing you can send me". People sent them, across several platforms. This post is about what I did with them: building a test set, a fix that worked, a fix that should have worked and didn't, and what real visitors typed that I never planned for.

For weeks my process was: change something, type three searches I liked, squint at the results, ship. Every change "looked better". Some of them weren't.

So I wrote a test set. Each case is a search (some from real visitors, some written to mirror them) plus 3–5 titles a well-read friend would name:

{ "id": "hard-but-fair", "query": "a hard but fair game where dying teaches you something", "expect": [ "Dark Souls", "Hollow Knight", "Celeste", "Sekiro: Shadows Die Twice", "Hades" ] } Enter fullscreen mode Exit fullscreen mode The score is recall@12: what fraction of the expected titles appear in the first 12 results. It's a blunt instrument. Five titles can't capture "good taste", and a great pick that isn't on the list scores zero. But it's the same blunt instrument every time, and that's the point.

Here's what "a hard but fair game where dying teaches you something" returned at baseline: White Knuckle, Causality, I Wanna Maker, Levelhead, NetHack, Celeste… Every one of them is hard, and plenty are fair. Celeste at #6 was the only title a person would actually name, and Dark Souls wasn't in the top 12 at all.

To an embedding, an obscure precision platformer and Dark Souls are close neighbours. They're described the same way. Only one of them is the one people mean.

The fix is to let audience opinion break ties. For each candidate:

score = similarity + ACCLAIM_WEIGHT * acclaim ( medium, votes, rating ) // acclaim is 0–1 Enter fullscreen mode Exit fullscreen mode acclaim combines the audience rating with how many people rated it, so a 9/10 with 40 votes doesn't beat a 9/10 with 40,000. It only re-sorts the top 50 by fit, so a famous title that doesn't fit can't sneak in.

The interesting part is the weight. I swept it:

0.15 scored higher, and I didn't take it. Her is a fine film, but it isn't a quiet film about ordinary rooms. It's a famous film that's near one. That's the failure the whole app exists to avoid: fame over fit. The test set can't see that distinction, so I made the call by reading the results.

Not every case improved across these changes, either. "Time loop" dropped from 40% to 20%, and "90s romcom" went to zero. A mean that goes up can hide cases that go down, so the eval prints a delta per case, and I read all of them.

"An analytical drama where the main character is a sarcastic stoic who struggles with his own emotions."

That's House. Or Sherlock. Neither appeared. In a sample of 155 titles, House ranked 153rd.

The reason is a variant of part 1's lesson. Each title has one description, and a description of House talks about medicine, mysteries, a hospital, a team. The lead's personality is one sentence among many, and one sentence barely moves a vector.

So I tried giving every film and show a second fingerprint that is only about the lead:

"The lead is a brilliant, sharp-tongued diagnostician who treats patients as puzzles and colleagues with open contempt, while privately fighting chronic pain and an addiction to painkillers."

Appending the line to the description did nothing, for the same reason as before: it gets averaged away. As its own vector it worked. Writing 7,971 lead lines through Anthropic's Batch API cost $0.22.

In the real pipeline, the model that reads the query writes a matching "The lead is…" sentence, embeds it, searches the lead vectors, and fuses that ranking with the normal one using reciprocal rank fusion:

// each list votes 1/(k + rank); titles in both lists add up const add = 1 / ( RRF_K + rank + 1 ) Enter fullscreen mode Exit fullscreen mode Then I ran the test set. The case still scored 0%.

The results were better: Scorpion, Death Note, The Imitation Game, Good Will Hunting, all reasonable answers to that request. But not the five titles on the list. Digging in, the lead vector turned out to be very sensitive to wording. The sentence the model wrote put Sherlock at #35; a sentence I wrote by hand put it at #2. Tuning the prompt until one test case passes is how you overfit a test set of 28.

So it's parked. The 7,971 lines and their vectors are in the database, ready for a better idea, maybe several phrasings per query with the best match kept. I'm not editing the test case to say "Scorpion counts". Moving the goalposts after missing is exactly what a test set is supposed to stop you doing.

My mental model of a search was a sentence with a why in it. Here's an actual search from this week:

"Dark, Severance, Black Mirror, Love, Death & Robots, MR. Robot, Lost, 1899, From, Alice in Borderlands, Squid game,"

No why at all, just a list. The results were decent (Silo first, which they clicked), but #3 was 1899 and #7 was FROM. I'd handed back two of their own shows.

The filter that drops titles you named skipped one-word titles on purpose, so that "dark thrillers" doesn't remove the show Dark. A comma-separated list of capitalised entries is a different thing, so now that counts too:

// "Dark, Severance, 1899, From" is a list; "dark, moody, slow thrillers" isn't const titled = entries. filter (( e ) => /^ [\p {Lu} \p {N} ] /u. test ( e )). length return entries. length >= 3 && titled >= entries. length * 0.66? entries: [] Enter fullscreen mode Exit fullscreen mode The bigger question was product, not code. A list of titles can only be matched on the titles themselves, which is exactly the genre-shelf matching this app exists to avoid. Refusing the search would be worse, though. So the results stay, and above them the page now asks:

These match the shows themselves. Tell us what you love about them, and we'll match that instead.

The input comes with sentence starters ( The mood, A character, A moment, How it felt ) that drop "because the mood was…" into the box for them to finish. Whatever they write becomes the why the next search matches on.

Every title page lists its nearest neighbours. For games, the catalog has every edition: Special, Enhanced, Story Mode, DLC. They're all described almost identically, so they're each other's nearest neighbours. All six of GTA V's top picks were GTA V: the base game twice, then the Special, Enhanced, Premium Online and Story Mode editions.

The fix is a deliberately blunt franchise key: the first two meaningful words of the title, once the subtitle and any trailing number are stripped:

franchiseKey ( ' Grand Theft Auto V: Special Edition ' ) // "grand theft" franchiseKey ( ' Grand Theft Auto: San Andreas ' ) // "grand theft" franchiseKey ( ' The Lord of the Rings ' ) // "lord rings" franchiseKey ( ' Lord of War ' ) // "lord war": stays separate Enter fullscreen mode Exit fullscreen mode At most one title per franchise per list. I recomputed all 12,016 lists. That cost nothing, since it's database work only. Skyrim's page now suggests Enderal, The Witcher 3 and Dragon Age instead of a shelf of its own editions.

I finally set up Google Search Console. Its first report: six of my title pages were "Crawled – currently not indexed". Google had read them and decided they weren't worth keeping.

It's hard to argue. A title page's synopsis comes from TMDB (or IGDB, for games), so it's the same blurb as on IMDb and a thousand other sites. The part only I have is the why, and the pages didn't really say it.

So the most popular title pages now open their "More like…" section with what people love about the title, and why each pick fits. The first draft was bad in an instructive way. My catalog descriptions are heavy on look and sound (part 1's "golden hour" work), so the model described Breaking Bad as "sun-bleached desert light set against fluorescent-lit rooms". "Sun-bleached" appeared 13 times across 20 drafts. Nobody searching "shows like Breaking Bad" means the lighting.

The second draft fed in the synopsis, themes and the lead lines from Lesson 3, and asked for story and character first:

People love Breaking Bad for its slow corruption of a decent, proud man, the tension between his family life and his criminal double life, and the dread that he will not stop before he destroys everyone around him.

The Americans: like Walter, a controlled person hides a second life from his family, and the strain between loyalty and deception drives the story.

It cost $0.009 for 20 pages. There was one factual slip in 16 pages: it gave Fear the Walking Dead a "protective father", when the lead is a mother. That slip, plus wanting proof the copy helps, is why it's on 25 pages so far, not 12,000. Those six rejected pages got the new copy, and I asked Google to look again. Whether it changes Google's mind is part 3.

While I was in there, I fixed the URLs. Title pages were /browse/3bd40852-97cc-4767-8584-388236de35cc. They're now /browse/forza-horizon-5-3bd40852. The trailing 8 hex characters are the start of the item's UUID, and the lookup is a range query, so it still uses the primary key index:

// ids are uuid-typed, so "starts with" is a range. gte ( ' id ', ` ${ prefix } -0000-0000-0000-000000000000` ). lte ( ' id ', ` ${ prefix } -ffff-ffff-ffff-ffffffffffff` ) Enter fullscreen mode Exit fullscreen mode Old links 308 to the new ones. The best time to change your URLs is before Google has indexed them, and for me that was this week.

Part 1 said re-describing the catalog cost about $2.30. True. What it didn't say: when you're iterating, you run it again. My bill showed that $2.30 more than once in two days. It's not a lot of money, but it's the wrong habit, and it scales with the catalog. The rules now:

It's free, no sign-up needed to search, no ads: becauseiliked.com

Try a search with a why, then one without. And if you have a search it gets wrong, especially a "the main character is…" one, put it in the comments. It goes straight into the test set.