Hugging Face and Voice Arena announced an open evaluation for Hindi and Indian English on August 28, 2026. The change adds the Monsoon en-IN and Monsoon hi-IN evaluation sets to the Open ASR Leaderboard, giving model builders and readers a broader way to examine speech-recognition performance across Indian speakers and recording conditions.
Begin with the language and split
Check the language label before comparing any score. Monsoon en-IN evaluates Indian English, while Monsoon hi-IN evaluates Hindi. Hugging Face describes Hindi as the first Indic language in the leaderboard’s multilingual tab, which had previously been limited to European languages. A model’s position in either evaluation is evidence about that language setting, not a general ranking for every language or accent.
Each evaluation has a public split for self-evaluation and a private split kept secret to reduce benchmark-specific optimization. The four splits are speaker-disjoint, cover 4,888 speakers, and record 12 attributes for each speaker. When reading a result, note whether it comes from the public or private split and whether the comparison uses the same language. Without those labels, the number supports very little.
Understand what Monsoon is testing
Monsoon was collected from spontaneous, unscripted conversations rather than prepared reading. The recordings used two channels, segmented separately so that each clip contains the voice of one speaker. The evaluation therefore focuses on conversational speech and single-speaker clips, rather than one controlled microphone setup or a rehearsed script.
The collection varies across nine dimensions: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and whether multiple valid transcriptions exist. The public Indian English split contains 5.62 hours from 1,444 speakers across 428 districts and 556 distinct devices. The public Hindi split contains 1.33 hours from 468 speakers across 202 districts and 315 distinct devices.
Those figures show substantial coverage, but they do not prove that every Indian speaking context is represented equally. They do explain why an overall average can conceal important differences. If a model is intended for speech captured on varied devices or in changing acoustic environments, Monsoon is more relevant than a narrowly controlled evaluation. If the deployment involves another language, different speech conditions, or a population outside the reported coverage, treat the score as context rather than a direct forecast.
Read Hindi results with the right metric
Hindi references use a lattice that lists accepted spelling variants. They are evaluated with Orthographically-Informed Word Error Rate, or OIWER, instead of conventional WER. That measurement choice affects whether a transcription is counted as an error, so a Hindi OIWER result should not be compared casually with a WER number from another evaluation.
Indian English results also need more than an overall average. In Hugging Face’s illustrative analysis of the public split, WER variation across regions was 1.68 points for mistralai/Voxtral-Mini-3B-2507 and 0.46 points for openai/whisper-large-v3-turbo. These examples do not establish a universal winner, nor do they replace the leaderboard’s full results. They illustrate why regional variation belongs in the comparison.
Use the public split as the first checkpoint
Start with the public data when evaluating a model. Run the model against the relevant split, keep the language and metric aligned, and record the score alongside the split and its coverage. A result without that context is difficult to assess or reproduce.
The public portion is particularly useful for identifying where a model may need closer inspection. Compare the language first, then look at the conditions covered by the split: speakers, regions, devices, environments, speech type, and rate. A model with a strong average may still be weaker in specific parts of that coverage.
Submit to the private evaluation when the comparison is ready
A model must be added to the leaderboard through a pull request to use the private splits. Hugging Face verifies the public results, calculates the private metrics, and then asks for confirmation of the results. The private score is part of that submission workflow; it cannot be inferred reliably from a public result.
This two-part design creates a useful separation. The public split lets model builders inspect and reproduce the measurement, while the private split provides a less visible test against overfitting to the published material. The trade-off is that private results are harder to inspect directly, so their meaning still depends on the language, metric, and coverage labels attached to them.
What the benchmark changes—and what it does not
The immediate change is measurement coverage. Hindi and Indian English now have dedicated Monsoon evaluations within the Open ASR Leaderboard, with public data for self-checking and private data for a less transparent test. Readers have a clearer basis for asking how a speech-recognition model performs across the Indian contexts represented in the collection.
A leaderboard result is not a guarantee for a particular deployment. The documented splits cover many attributes, but they remain evaluation sets with defined participants, districts, devices, and recording conditions. Use the results to narrow a model choice and identify what needs further validation, then check whether the intended application matches the relevant language, speech style, hardware, and environment.
The practical approach is to use both sides of the benchmark. Reproduce the public measurement first, inspect the relevant variation, and submit through the private path once the model and comparison are ready for leaderboard verification. That sequence preserves the value of open checking without treating benchmark performance as universal accuracy.