The best Arabic LLM for your enterprise is the one that passes your eval, not the leaderboard

Posted on Oct 28 • Originally published at azrty.com

Four serious Arabic models now come out of the Gulf: Falcon-H1-Arabic from Abu Dhabi, ALLaM from Saudi Arabia, Fanar 2 from Qatar and Jais 2 from the UAE. None of them can be picked from a leaderboard, because the Open Arabic LLM Leaderboard measures academic Arabic, not your tickets, your policy documents or your customers' dialect. The two things that do transfer to your organisation are both measurable in about a week: a 200 to 500 item eval set built from your own material, and the tokenization economics that decide what each Arabic conversation really costs in money, latency and GPU memory.

If you are building Arabic-language customer service, a government services assistant or an internal knowledge bot in the GCC, you face a decision that looks like a model choice but is actually two measurements. First: does a regional open-weight model score within your tolerance on your content, with your product names, place names, dialects and policy language? Second: what does an Arabic token actually cost you, in API spend, time to first token, KV-cache footprint and concurrent sessions per GPU? Answer both and the decision makes itself. Skip them and you will either overpay a frontier API for years or ship a model that fails quietly on Gulf dialect.

Four model families are credible candidates for enterprise Arabic work today. All of them are open weight unless noted, which means you can run them on your own GPUs and keep data in-country.

Primary sources: the Falcon-H1-Arabic announcement and release blog, the Fanar-2-27B-Instruct model card, the ALLaM-7B-Instruct-preview model card, the HUMAIN Chat launch and the Jais 2 documentation.

The fourth candidate in every real procurement is a frontier API model prompted in Arabic. It belongs in the comparison, because on raw Arabic quality it often wins. The question is by how much, and what that gap costs.

The Open Arabic LLM Leaderboard is genuinely good work. It runs reproducible evaluations through the EleutherAI lm-evaluation-harness Arabic task set, so a score is a score, not a vendor claim. TII even documents that its own vLLM-based runs land within about one point of the leaderboard's numbers.

But read what it measures: knowledge, reading comprehension and reasoning over web-derived Arabic text, mostly Modern Standard Arabic. Your workload is not that. It is a customer writing "ما أقدر أرسّع الفاتورة من التطبيق" (I can't pay the bill faster from the app, in Gulf dialect), a code-switched complaint ("الـ bundle خلص، كيف أعمل top-up؟"), and questions about a product name that appears nowhere in any pretraining corpus. Leaderboard rank measures none of the things that will make your assistant succeed or embarrass you.

This is not an argument against the leaderboard. It is the screening stage: it tells you which models are worth a week of your team's time. The final stage happens on your data.

The single highest-leverage artefact in an Arabic LLM project is a small, versioned eval set built from your own material. Two hundred items is a floor, five hundred is plenty; beyond that the marginal information per item collapses.

A concrete mix for a 300-item set, drawn from a telecom support case:

That is one analyst-day to sample, one subject-matter-expert day to write gold answers, one more to review. Gold answers should be short and checkable, not essays.

Score each response on five criteria, 0 to 2 each, so the maximum is 10 per item and a 300-item set yields a 3,000-point scale:

Two ways to score: a human reviewer (expensive, about 1 to 2 minutes per item, so a full pass on 300 items is a day) or a stronger model as judge. If you use a model as judge, check agreement first: have the human score a 50-item overlap sample and compute percent agreement on the Correctness criterion. Below roughly 85% agreement, your judge is not measuring your rubric, and you must fix the prompt before trusting it at scale.

Here is a copy-paste judge prompt that has the properties that matter: it forces evidence, it forbids generosity, and it outputs a parseable score.

You are evaluating a customer-service assistant's reply in Arabic. QUESTION (verbatim from a real user, may contain Gulf dialect or Arabic/English code-switching): {question} GOLD ANSWER (written by a domain expert): {gold_answer} ASSISTANT'S REPLY: {reply} Score each criterion 0, 1 or 2. Be strict: a score of 2 must be clearly earned, not merely plausible.

1. correctness: 2 = matches the gold answer's facts, entities and amounts; 1 = partially right; 0 = wrong. 2. grounding: 2 = every claim is traceable to the gold answer or the supplied policy text; 1 = vague; 0 = invents facts. 3. dialect: 2 = correctly understands the question's register and replies naturally in it; 1 = understands but replies in stiff MSA; 0 = misunderstands the dialect or code-switching.

4. language: 2 = natural, correct Arabic; 1 = awkward but usable; 0 = broken Arabic or wrong language. 5. completeness: 2 = covers steps and caveats; 1 = usable with gaps; 0 = misses something essential. Reply ONLY with JSON: {"correctness": n, "grounding": n, "dialect": n, "language": n, "completeness": n, "total": n, "fatal_error": "<one line, or 'none'>"} Enter fullscreen mode Exit fullscreen mode Run every candidate model, including the frontier API baseline and its Arabic-prompted variant, on the identical set with the identical retrieval context if you use RAG.

Report the mean total, the per-criterion breakdown, and the fatal-error rate. The per-criterion breakdown matters more than the total: a model that loses two points per item entirely on dialect handling might be fixable with prompt work, while one that loses them on grounding needs retrieval surgery, and one that loses them on correctness needs to be dropped.

For dialect specifically, published per-dialect evaluations such as DialectalArabicMMLU show how much scores move between MSA and regional varieties, and TII reports AraDice dialect sub-scores for the Falcon-H1-Arabic family. Your own dialect slice settles the question for your dialect, which is the only one that matters.

Arabic text costs more tokens than the equivalent English text on English-centric tokenizers, and that premium appears in four places at once: your API bill, time to first token, KV-cache memory and the number of concurrent Arabic sessions a GPU can hold.

The peer-reviewed evidence is blunt. Petrov and colleagues showed that the GPT-4 tokenizer encodes the same content in Standard Arabic with roughly 3.04 times as many tokens as English, and the ChatGPT tokenizer with about 4.40 times as many, with direct consequences for cost, latency and effective context length ( arXiv:2305.15425 ). The original Jais team measured the same pattern at the word level: GPT-2's vocabulary fragments Arabic into about 4.17 tokens per Arabic word against roughly 1.1 per English word, while the bilingual Jais vocabulary achieves about 1.05 ( fertility figures summarised here ). Newer multilingual vocabularies have narrowed the gap, but a 1.5 to 2 times premium on an Arabic-inefficient tokenizer is still normal, and nothing on your invoice labels it.

Measure it on your own traffic before believing any ratio. Ten lines of Python gives you fertility, tokens per word, on a sample of your real conversations:

from transformers import AutoTokenizer text = open ( " sample_arabic_conversations.txt ", encoding = " utf-8 " ). read () words = len ( text. split ()) # Verified repo IDs; add the other candidates from their model cards for repo in [ " QCRI/Fanar-2-27B-Instruct ", " humain-ai/ALLaM-7B-Instruct-preview " ]: tok = AutoTokenizer. from_pretrained ( repo ) fertility = len ( tok ( text ). input_ids ) / words print ( f " { repo }: { fertility:. 2 f } tokens per Arabic word " ) Enter fullscreen mode Exit fullscreen mode One caution: studies count words and normalise text differently, so compare ratios within your own sample rather than against published absolutes. Your sample is the number that will hit your bill.

Take a 400-word Arabic support answer and compare a model whose tokenizer gives 2.4 tokens per Arabic word against an Arabic-optimised model at 1.5. Same content, same job.

Spend. The answer is 960 tokens on the first tokenizer, 600 on the second, a 1.6 times premium on every request. At an illustrative $5 per million output tokens, one million such answers costs $4,800 versus $3,000, a $1,800 difference before you have written a word of prompt engineering. Multiply by your volume. And note the trap when comparing providers: a model priced 30% lower per token that fragments Arabic 1.6 times more is not 30% cheaper for you, it is roughly 12% more expensive.

Latency. Prefill is compute-bound on prompt length, so time to first token scales with the token count of your system prompt plus retrieved context. Decode is per token, so an answer fragmented into 1.6 times the tokens streams 1.6 times longer at the same hardware speed. Prompt tuning cannot remove this; it is in the vocabulary.

KV cache and concurrency. Per-token KV-cache size for a 7B-class model with 32 layers, 8 grouped-query KV heads and 128-dimension heads at fp16 is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 128 KiB per token. A 4,000-token Arabic conversation at the efficient tokenizer holds about 500 MB of KV per session; the same content at 1.6 times fertility holds about 800 MB. On a GPU with 60 GB free after model weights, that is roughly 120 concurrent sessions versus 75. Tokenization, not parameter count, decided your concurrency.

Context. A 256K context window (Falcon-H1-Arabic 7B and 34B) holds far more Arabic at a low fertility than a longer window holds at a high one. This is why the Jais 2 team built a custom 150,272-entry Arabic-centric vocabulary and report it makes inference efficient: the efficiency is not marketing, it is arithmetic you can reproduce with the snippet above.

Now the choice reduces to one comparison on one spreadsheet.

A practical note on keeping this sustainable: whatever prompts and gold answers survive the eval, version them. The Arabic system prompts that work, the dialect-handling instructions, the few-shot examples, these are assets, and they are exactly the kind of asset that walks out the door in a Markdown file. We keep them in a shared, versioned prompt library ( PromptForge ), and when we build and run these assistants for clients, from eval set through deployment, that is our AI engineering work, done by our own team here in Dubai.

The best Arabic LLM for enterprise is not a fixed answer, because "enterprise" is not a fixed workload. It is whichever of Falcon-H1-Arabic, ALLaM, Fanar 2, Jais 2 or a frontier API passes an eval set built from your own tickets, at a token cost you have measured rather than assumed. Both measurements take a week. The alternative is guessing, and guessing is the most expensive part of any AI project.