My AI directory has 92 categories. Ten of them hold 37% of the listings.

I ran one query against my AI tools directory last night, and the result explained why some of its pages get crawled while others sit ignored.

Those ten categories hold 2,576 of 7,006 category assignments — 36.8% of the whole taxonomy. The directory has 4,323 tools across 92 categories.

That last number is not the problem. The tail is.

Three of those are empty, and they are also the most tempting pages to publish, because the names sound valuable: "New AI Tools" and "Programming" are exactly what a searcher types. So I checked what the site actually returns for them:

$ BASE = https://aiverse.mindutil.com $ curl -o /dev/null -w "%{http_code}" " $BASE /category/last-ai-en" 404 $ curl -s " $BASE /category/last-ai-en" | grep robots <meta name = "robots" content = "noindex, nofollow" /> Enter fullscreen mode Exit fullscreen mode And the sitemap filter is one line:

where: { tools: { some: {} } } Enter fullscreen mode Exit fullscreen mode So 89 of 92 categories ship in the sitemap. The three empty ones do not.

An empty category that returns 200 with "0 tools found" is a page a crawler can index. Do that twenty times and you have told a search engine your site has twenty useless URLs before it has read a single tool page. The 404 does the opposite: it says this hub does not exist yet.

None of this is free. A user who bookmarked the page gets a dead end, and the internal link from the categories index has to be removed or the link graph starts leaking 404s. The trade is simple: a little maintenance for the absence of thin pages. On a directory where the median tool page is still fighting for its first indexing, that trade is worth taking.

4,323 tools produce 7,006 category assignments, an average of 1.62 per tool. Exactly 2,100 tools (48.6%) sit in two or more categories. That overlap is real — a video editor can be "Video Editing", "Productivity", and "Business" without stretching any of those labels.

Overlap is also what keeps category pages from being template clones: each one is a different basket of results. Directory sites get flagged as doorway pages when the only unique word on 92 pages is the category name. The fix is not fewer categories. It is making each category page answer something the tool pages cannot: which five of these should I try first, and why.

RIP AI is the largest bucket in the whole directory: 405 tools, 9.4% of everything, 236 of them marked free. These are discontinued or dead products. Publishing that number is useful — a directory that never deletes anything is lying by omission — but RIP AI is not a category. It is a status.

Mixing the two is how you end up with a tool that is both "RIP AI" and "Video Generators", an empty "Programming" bucket, and all three looking equally important in a sitemap.

Only the first rule is already live. The other three are the work for the next few weeks.

The top ten are not the categories I would have guessed. I expected "AI Chatbots" to lead; it has 8 tools. Video Generators has 294. The taxonomy drifted toward whatever the crawler found in bulk rather than toward what people actually search for. That is a data problem wearing a design problem's clothes, and it is the next thing on the list.

If you want to see the current state of the catalog, including the free and paid filters, it is here: https://aiverse.mindutil.com/free-ai-tools?utm_source=devto&utm_medium=social&utm_campaign=topic