yabasha/arabic-stt-eval

Dialect-stratified evaluation for Arabic speech-to-text.

Arabic breaks word error rate. Not slightly — enough to reverse a vendor ranking.

The same spoken sentence has several correct spellings. إحنا and احنا. مدرسة and مدرسه. على and علي. ٣ and 3. A transcriber picks one, a model picks another, and naive WER scores every one of those as a substitution. The model heard correctly. The metric lied.

This harness measures Arabic ASR the way it has to be measured: normalization made explicit and tiered, scores stratified by dialect, code-switching separated out, and the normalization gap reported as a first-class number — how many points of apparent error rate were orthography rather than recognition.

On a ten-utterance smoke corpus, that gap is 32.8 points. Same transcripts, same model, one methodology decision.

(Figures above come from demo/, which is synthetic — hand-written reference/hypothesis pairs built to exercise the normalizer, not a measurement of any provider. Real numbers need real audio. See Getting real numbers.)

---

Findings

Measured results produced with this harness, each with its method and its limits stated.

• 01 — Arabic regional subtags resolve to a single Nova-3 model All six Arabic codes tested (ar, ar-JO, ar-PS, ar-SA, ar-EG, ar-MA) returned one model UUID and byte-identical transcripts on ~7 minutes of MSA-leaning audio. Invalid subtags are rejected and other languages route to distinct models, so the parameter is honoured — it just did not change this decode. Single sample; dialect-heavy audio untested.

Why this exists

Deepgram's Nova-3 Arabic benchmark carries a footnote: before comparing against other providers, they normalized "diacritics, alef forms, ta marbuta, and number formats" so models were compared on what they heard rather than on stylistic writing differences. That is the correct methodology.

It also means a prospect who evaluates providers by piping raw output into jiwer is running a different experiment than the vendor did, and will get a different answer. Sometimes a different winner. And the bake-off spreadsheet is usually assembled in an afternoon by someone with other priorities — after which whatever it says becomes the decision.

So: publish the harness, make the methodology arguable, and let the number survive scrutiny.

What it measures

Install

Python 3.10+. No dependencies — standard library only, deliberately, so a customer's security team can read the entire network path in one file before granting an exception.

git clone https://github.com/yabasha/arabic-stt-eval
cd arabic-stt-eval
python3 -m unittest discover -s tests -t .      # 52 tests, ~3ms

Use

export DEEPGRAM_API_KEY=...

# See what the normalizer does to a string, under every policy.
python3 -m arabic_stt_eval.cli normalize "إحْنا رايحين ٣ مرّات، إن شاء الله"

# Scaffold a manifest from a folder of audio. References come out empty —
# fill them in by ear, before you transcribe anything.
python3 -m arabic_stt_eval.cli init --audio-dir audio/ --dialect ar-JO --out manifests/levantine-jo.jsonl

# Check the corpus before spending API calls on it.
python3 -m arabic_stt_eval.cli validate --manifest manifests/EXAMPLE.jsonl

# Transcribe and score.
python3 -m arabic_stt_eval.cli run --manifest manifests/EXAMPLE.jsonl --out reports/baseline

# Same corpus with Keyterm Prompting, to measure the lift.
python3 -m arabic_stt_eval.cli run --manifest manifests/EXAMPLE.jsonl --keyterms --out reports/keyterms

# Score transcripts produced somewhere else — a self-hosted deployment,
# a competitor's SDK, or a run inside a customer's VPC.
python3 -m arabic_stt_eval.cli score \
  --manifest manifests/EXAMPLE.jsonl \
  --hypotheses out/hyps.jsonl \
  --provider-label whisper-large-v3 --out reports/whisper

run and score are split on purpose. In a regulated engagement the audio frequently cannot leave the customer's environment — SAMA and CBUAE rules keep core financial data in-country, and that constraint arrives before the accuracy conversation, not after. Transcription runs where the audio is allowed to be; scoring runs on your laptop.

Manifest format

JSONL, one utterance per line. Audio paths resolve relative to the manifest.

{"id": "amm-001", "audio": "audio/amm-001.wav", "reference": "شو أخبارك اليوم",
 "dialect": "ar-JO", "domain": "call-center", "code_switched": false,
 "keyterms": ["كليك", "إي فواتيركم"]}

Only audio and reference are required. dialect should be one of Deepgram's Arabic variants (ar, ar-AE, ar-SA, ar-QA, ar-KW, ar-SY, ar-LB, ar-PS, ar-JO, ar-EG, ar-SD, ar-MA, ar-DZ, ar-TN, ar-IQ, ar-TD, ar-IR); validate will tell you if it isn't.

The normalization tiers

Invisible characters (ZWSP, ZWNJ, bidi marks, BOM) are stripped at every tier including raw. They survive copy-paste out of RTL documents, they are invisible in every editor, and they silently break token equality. There is no reading of "measure the raw output" under which they should count.

aggressive is included because it is instructive, not because it should be quoted. It turns سماء into سما — which is exactly the kind of over-normalization that makes a bad model look acceptable. Know where the line is; do not stand on the wrong side of it.

Design notes

Pooled, not averaged. Corpus WER pools errors over pooled reference length. Averaging per-utterance rates weights a three-word greeting the same as a two-minute call, and is the most common way an ASR benchmark quietly overstates itself.

Alignment is kept. A WER scalar tells you a system is worse. The substitution list tells you whether the failure is a dialect lexicon miss, a code-switching drop, or a number-format disagreement — which is what a customer conversation is actually about.

Failures are per-item, not per-batch. One corrupt file should not cost you the other 399 API calls you already paid for. A partial run is reported as partial, in the report header.

Batch RTF is not streaming latency. The report says so, in the report, because that distinction gets lost between an evaluation and a capacity commitment.

Getting real numbers

Nothing is bundled. Bring audio you have the right to use:

• Common Voice Arabic (CC0) — read speech, MSA-leaning. Good for a smoke test, weak proxy for conversational dialect.
• MGB-2 / MGB-3 — broadcast and Egyptian dialect, licence required.
• QASR — 2,000 hours of Aljazeera broadcast, code-switching annotated.
• Casablanca — multi-dialect, released for dialect ASR research.
• Your own recordings. For a genuinely representative Levantine or Gulf contact-centre distribution, 40 hand-corrected utterances you recorded beats 4,000 of read MSA.

Customer audio does not go in this repo. Ever. Run score against transcripts instead, and keep the manifest on their side of the boundary.

Roadmap

• Streaming evaluation — time-to-first-token and interim-result stability, which is the number that actually matters for voice agents and which batch RTF cannot stand in for.
• Diarization scoring (DER) for multi-speaker contact-centre audio.
• A second provider adapter, to make the bake-off path concrete rather than theoretical.
• Bootstrap confidence intervals on corpus WER. A 40-utterance corpus has a wide interval and reports that print three decimal places are lying about their precision.

Licence

MIT.

---

Built by Bashar Ayyash — Amman, Jordan. Native Levantine Arabic, twenty years shipping software into regulated environments. I build evaluation gates for AI features professionally; this is that work pointed at audio.