Which Malaysian websites block AI crawlers? I checked 100

Posted on Oct 30 • Originally published at yappatyih.com

An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites block the most, 5 of the 15 I checked, and GPTBot is the crawler blocked most often, by 9 sites. 74 of the 100 let every AI crawler in, and 13 would not show my script their robots.txt at all. Below are the sites, what their files say, how I checked, and what a Malaysian business should do with its own file.

Of the 100 major Malaysian websites I checked on 20 October 2026, 13 tell at least one AI crawler to stay out, and most of the rest say nothing about AI at all.

"Block" here means the file tells that crawler not to fetch the home page, "/", which keeps it out of the whole site unless other paths are opened for it. I counted 17 AI crawlers, from the list of 20 my free checker uses. The other three, Googlebot, Bingbot and Applebot, are search engine crawlers, and no site in the sample blocked Googlebot.

Most big Malaysian sites have not written down a choice about AI. Only 18 of the 78 files I could read name an AI crawler at all. The other 60 do not mention AI at all.

A larger survey points the same way. Zyte, a web data company, reports in its State of Web Access 2026 that 13.4% of the 1,106 Malaysian websites it looked at block AI crawlers, and 78.2% have a robots.txt. My sample is smaller, but every site in it is named.

News is the only sector where blocking AI crawlers is common: 5 of the 15 news sites I checked block at least one.

By language, 4 of the 8 English news sites block an AI crawler: the New Straits Times, Free Malaysia Today, The Edge Malaysia and Bernama. 1 of the 4 Malay sites does, Harian Metro; Berita Harian refused my request, so I cannot say. None of the 3 Chinese papers, Sin Chew Daily, China Press and Oriental Daily, blocks any. The Star blocks one bot outside my list of 17: Diffbot, which extracts data from web pages ("user-agent: Diffbot", then "Disallow: /").

Outside news, blocking is rare and scattered: KWSP and Bank Negara Malaysia in government, Zalora and Lotus's in shopping, Maukerja and Ricebowl in jobs, EdgeProp in property, and foodpanda. Banks are the least readable sector: only 5 of 10 served a robots.txt my script could read.

These are the 13 sites whose robots.txt, read on 20 October 2026, blocks at least one of the 17 AI crawlers I checked, most blocks first. A block here means the file tells that crawler not to fetch the home page.

The other 87 let every AI crawler in, have no robots.txt, or did not let my script read the file. All 100 are in the study's CSV file.

Training crawlers are blocked far more often than the bots that fetch pages for AI answers. GPTBot, which collects pages to train OpenAI's models, is blocked by 9 of the 100 sites.

10 of the 13 blocking sites keep out at least one training crawler, GPTBot, ClaudeBot or CCBot, while leaving OAI-SearchBot and PerplexityBot free to come in. Only 4 sites block any bot that fetches a page for AI search or for a person's question.

Google's AI is blocked less often than OpenAI's. Google-Extended, the robots.txt token that says whether Google may use pages for its AI models, is blocked by 4 sites, under half the 9 that block GPTBot. The New Straits Times and Harian Metro block GPTBot and ClaudeBot but write "Allow: /" for Google-Extended.

Each line below is a fact from the site's own robots.txt on 20 October 2026, with rules quoted as written.

The New Straits Times and Harian Metro are the strictest, blocking 12 of the 17 AI crawlers. Their two files are the same except for one folder name. The search and assistant bots of OpenAI, Anthropic and Perplexity are blocked from the home page but allowed into one section: "Allow: /plus/" on the NST, "Allow: /hmextra/" on Harian Metro. Both files carry the line "Content-Signal: search=yes,ai-input=no,ai-train=no", and the NST's opens with a heading dated 16 October 2026.

Bernama blocks five: GPTBot, ChatGPT-User, ClaudeBot, Bytespider and CCBot. Its file gives the reason in Malay, in a comment: "Halang Bot AI / Scraper Agresif (Ringankan Beban Server)", block AI bots and aggressive scrapers to lighten the load on the server.

The Edge Malaysia has the shortest blocking file in the sample: four lines, 54 bytes, which shut out GPTBot alone and allow everything else.

Free Malaysia Today names 15 AI crawlers and lets most of them in by name, GPTBot, ClaudeBot, PerplexityBot and Google-Extended included. It blocks three: CCBot, Bytespider and Amazonbot.

KWSP and EdgeProp serve the same file, byte for byte. It is a block of rules marked "# BEGIN Cloudflare Managed content" that keeps out eight crawlers and tokens tied to AI training, GPTBot, ClaudeBot and Google-Extended among them, and says "Content-Signal: search=yes,ai-train=no,use=reference". Two of the 13 blocking sites, then, block AI with a file their CDN provides.

Bank Negara Malaysia blocks one AI crawler, Common Crawl's CCBot ("User-agent: ccbot"), at the end of a long list of SEO tools and other search engines' bots it keeps out.

Lotus's blocks GPTBot, ChatGPT-User, Google-Extended and CCBot. It also names "anthropic-ai" and "Claude-Web", older names for Anthropic's bots, but not ClaudeBot, the name Anthropic's crawler uses now. So ClaudeBot falls under the file's general rules, which let it in.

foodpanda blocks GPTBot and gives OAI-SearchBot and ChatGPT-User "Allow: /". That is the split OpenAI's own documentation describes: out of training, still in ChatGPT search.

Zalora blocks only Meta's training crawler: "User-agent: meta-externalagent", then "Disallow: *".

Maukerja and Ricebowl, two job sites with near-identical general rules, block different sets. Maukerja blocks GPTBot, ClaudeBot, Google-Extended, Amazonbot and Bytespider; Ricebowl blocks ClaudeBot, Meta-ExternalAgent and Bytespider.

One more is worth a line, though I count it as allowing. JobStreet Malaysia gives GPTBot, CCBot, Google-Extended, Bytespider and three other bots "Disallow: /companies" and "Disallow: */job/". AI training crawlers are kept off company pages and job ads, and the home page stays open.

Some Malaysian sites name AI crawlers only to let them in, and these files are as deliberate as the blocks.

13 of the 100 sites did not let my script read their robots.txt, which is a finding in itself.

8 answered "403 Forbidden": TM, Berita Harian, the Ministry of Health, PG Mall, Watsons, PropertyGuru, Batik Air and McDonald's Malaysia. Most of these answers came from a service in front of the site: the response named Cloudflare on four, and Akamai, Microsoft Azure's gateway and Envoy on one each. A 403 tells me the site screens unknown bots before they reach any file. It does not tell me what its robots.txt says, or how it treats GPTBot itself.

4 did not answer. Maybank2u closed the connection, Public Bank did not reply within 10 seconds, MyKasih reset the connection, and for JPA my script could not verify the site's certificate chain, a gap a browser can often fill on its own. Indeed Malaysia sent my request through three redirects to a sign-in page.

9 sites have no robots.txt at all. AmBank, Bank Islam, FPX, MetMalaysia, Universiti Malaya, UKM, USM and Air Selangor answered 404, and Firefly's robots.txt address leads to its home page. Under the robots.txt standard, no file means every crawler may fetch everything, so I count these 9 as letting every AI crawler in.

On 20 October 2026, from 16:31 to 16:34 Malaysia time, a small script made one request to each site: GET /robots.txt over HTTPS, following at most three redirects, waiting at most 10 seconds, one site a second. Its user agent named this study and linked to my checker. It fetched nothing else.

The result for each of the 100 sites is in this CSV file, free to reuse with a link to this page.

Most big Malaysian sites have not written down a choice about AI crawlers, and yours may not have either. Saying nothing lets every crawler in, which is a choice too. Five steps.

A few do. On 20 October 2026, 13 of 100 major Malaysian websites I checked blocked at least one AI crawler in robots.txt, and none blocked all of them. 74 let every AI crawler in, and 13 did not let my script read their file.

On 20 October 2026, the New Straits Times, Harian Metro, Bernama and The Edge Malaysia blocked GPTBot in robots.txt. Free Malaysia Today named GPTBot and allowed it, while blocking CCBot, Bytespider and Amazonbot. The Star, Malaysiakini, Malay Mail, Astro Awani, Sinar Harian, Utusan, Sin Chew Daily, China Press and Oriental Daily blocked none of the 17 AI crawlers in this study; The Star blocks Diffbot, a data extraction bot outside that list.

Not entirely. GPTBot collects pages to train OpenAI's models, while ChatGPT's search uses OAI-SearchBot and a page someone asks about is fetched by ChatGPT-User. A site that blocks only GPTBot can still appear in ChatGPT answers, which is how foodpanda Malaysia's robots.txt is set up.

Open yourdomain.com/robots.txt and look for user-agent lines that name AI bots, such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended, followed by "Disallow: /". A free AI crawler checker reads the file for 20 crawlers at once. Then ask whoever runs your CDN or firewall whether it blocks AI bots, because that never shows in robots.txt.

No. robots.txt is a request that well-behaved crawlers follow, not a lock, and a crawler that ignores it can only be stopped at your firewall or CDN. Use robots.txt to state your choice, and the firewall for bots that do not listen.

I write about running a company and building with AI at yappatyih.com.