Hindi In, Urdu Out: I Tested Voice AI on Indian Languages

Developers have reported since at least 2023 that OpenAI's open-source Whisper sometimes writes Hindi in Urdu script, even with the language set to Hindi. I wanted to know whether OpenAI's newer gpt-4o-transcribe still does it, how often, when, and what fixes it, and how it compares with Sarvam and Azure. So I sent the same recorded sentences in Hindi, Bengali, Marathi, Tamil and English to all three and scored every transcript. With the language set to Hindi, it returned 50 of 390 Hindi clips in another script, mostly Urdu. For clips under 8 seconds, it was 1 in 3. The lesson that will outlast these numbers: setting the language isn't enough, so check the script.

You don't need to know how speech models work. If you've ever called an API, you can follow this. Every section ends with a one-line Remember, and the whole article fits in 10 lines at the end.

• Part 1 · The test: what I sent, to whom, and how a transcript is scored, sections 1–2.
• Part 2 · What came back: wrong scripts, why short clips slip, accuracy, a one-line fix, speed and cost, sections 3–8.
• Part 3 · Using this: what to check before you ship, how to test without surprises on the bill, and the limits of this test, sections 9–11.

Four things people often assume, and what I measured:

1. The test

Start by listening. This is one of the clips, and what each service wrote down.

The audio. FLEURS is a public dataset from Google: the same sentences read aloud by native speakers in over 100 languages, released under CC BY 4.0. I took 100 sentences that exist in all five languages, one recording each. For Hindi, the language with the main finding, I then added every other Hindi sentence in FLEURS, for 391 Hindi clips. FLEURS has no Indian English set, so English here is US English.

The services and models.

The rules. Every service got the same files, converted to 16 kHz mono 16-bit WAV, and a fixed language code. No service was asked to detect the language itself. For English, Sarvam got en-IN, its only English code, and Azure got en-US. Every raw answer was saved before scoring, so the scoring rules can change without calling any API again. Three clips were longer than 30 seconds, which Sarvam's real-time API doesn't take, so I left them out for every service. That leaves 788 scored clips per service: 390 in Hindi, 100 in Bengali and English, and 99 in Marathi and Tamil.

2. How a transcript is scored

A transcript is compared with what was actually said, after cleaning both the same way. Then you count the edits needed to turn one into the other.

• Word error rate (WER): wrong, missing and extra words, divided by the number of words said.
• Character error rate (CER): the same, counted in letters.

Hindi, Marathi and Tamil often have more than one accepted way to join or split a word. So a transcript that a person would call correct can still score badly on words. That's why this article leads with character error rate. Numbers written as digits ("40") against words ("चालीस") also count as errors, for every service alike. I checked that the ranking doesn't change if you also ignore the nukta (ज़ vs ज) and the chandrabindu (हूँ vs हूं).

There's a third check, which turned out to matter most: is the transcript in the right script? I count an answer as wrong-script when less than half of its letters are in the script the language uses.

3. Right words, wrong script

Here are all 390 Hindi clips for each service, in the same order on every row.

With the language set to Hindi (language=hi), OpenAI's gpt-4o-transcribe returned 50 of 390 Hindi clips in another script: 46 in Urdu's Perso-Arabic letters, 2 in Gurmukhi (the script used for Punjabi), 1 in Japanese and 1 as Latin-letter text that isn't a real language. Sarvam and Azure returned all 390 in Devanagari. With this many clips, the true rate for this kind of audio is most likely between 10% and 16.5%, so this isn't a fluke.

This isn't a new kind of problem. In a GitHub discussion on OpenAI's open-source Whisper, opened in September 2023, developers report Hindi audio transcribed in Urdu or Persian script, including with the language set to Hindi (--language hi, language='hi'). The thread is closed without an answer. One possible reason: the Whisper paper lists 104 hours of Urdu speech-recognition training data and only 12 hours of Hindi. That's for the original Whisper models; OpenAI hasn't published what gpt-4o-transcribe was trained on. What this test adds is that a newer OpenAI model still does it, how often, on which clips, and what fixes it.

It isn't random. I ran the first 100 Hindi clips through the same model a second time: the 15 clips that slipped the first time all slipped again, plus 2 more. Something about those recordings consistently pushes the model to the wrong script.

The words were usually right. Spoken Hindi and spoken Urdu are close enough that the audio alone often doesn't decide the script. Score only the 340 Hindi answers that came back in Devanagari, and OpenAI's character error rate is 4.5%. Score all 390 and it's 12.0%. Most of the gap is the script.

For an app, the script is the whole answer. A Hindi search box, a database, a text-to-speech voice or another model downstream can't use Urdu letters for Hindi text.

Marathi showed the same problem, less often: 3 of 99 answers came back in another script. One of them was in Sinhala, the script of Sri Lanka, with words that don't exist. Bengali, Tamil and English had no wrong-script answers.

4. Short clips slip most

The clips that slipped had something in common: they were short. Here's the share that came back in another script, by clip length.

For gpt-4o-transcribe, 31 of 91 clips under 8 seconds (34%) came back in another script, against none of the 61 clips over 16 seconds. With fewer words, the model has less to go on when it picks a script. Voice commands, questions and quick replies are mostly short, which is exactly where this matters.

The slipped sentences also had slightly more words of Urdu or Persian origin, the ones written with a nukta like ज़ or फ़: 14.6% of sentences with such a letter slipped, against 11.3% of sentences without. That difference is small and mixed up with length, so I can't say vocabulary is a cause. Clip length is the clear pattern.

5. Accuracy by language

Character error rate on the same clips, per language. Switch to word error rate to see why it's the harsher measure for Indian languages.

I checked which gaps are bigger than chance by resampling the clips. Sarvam's lead is clear in Hindi and Marathi, and narrow but clear in Bengali. In Tamil, Sarvam and OpenAI are too close to call, and so are Azure and OpenAI. OpenAI's lead in English is clear. Tamil was the hardest language for Sarvam and Azure; for OpenAI, the script problem made Hindi and Marathi score worse than Tamil.

A few transcripts in every service were in the right script but mostly wrong, more than half the letters off. That happened to 2 Hindi clips for Azure and 3 for OpenAI, and to 5 Tamil clips each for Sarvam and Azure. The script check doesn't catch those; the error rate does.

These are best-case numbers from clean read speech; section 11 explains why.

6. One line of instruction

OpenAI's transcription API accepts a short text prompt alongside the audio. I re-ran the same Hindi and Marathi clips with one line asking for Devanagari, written in that language.

In Hindi, wrong-script answers fell from 50 to 3 out of 390, and the character error rate from 12.0% to 5.0%. That's level with Azure (5.3%; the difference is within chance) and still behind Sarvam (3.2%). The reply time didn't change. In Marathi, wrong-script answers fell from 3 to 0, but the error rate only moved from 10.2% to 8.3%, still about double Sarvam (4.4%) and Azure (5.7%). So in Marathi the model also mishears more.

data = {"model": "gpt-4o-transcribe", "language": "hi", "response_format": "json",
        "prompt": "यह हिंदी भाषा की रिकॉर्डिंग है। देवनागरी लिपि में लिखें।"}

The instruction isn't free of side effects. On one short Hindi clip about a man firing an M16 rifle, the instructed model returned a fluent Devanagari sentence about someone's birthday preparations. It was the right script and completely invented. That's 1 clip in 390, but it's a reminder that a prompt steers the output and doesn't make it true.

Two other OpenAI models, for comparison:

• gpt-4o-mini-transcribe, half the price, slipped less often without any instruction: 11 of 390. But its character error rate was 6.9%, higher than Azure's and the instructed gpt-4o-transcribe.
• whisper-1, the older model, tried on the first 100 Hindi clips: 1 in the wrong script, but an 11.3% error rate and 2.62 seconds per reply. That fits the GitHub report, where OpenAI's API returned Devanagari for a file that the local open-source model wrote in another script.

I sent language=hi with every request. OpenAI's speech-to-text guide notes that the newer transcription models have limited parameter support, and I couldn't confirm from the API reference how they use the language code. What I can say is what came back.

7. How long a reply takes

For a voice feature, the time between someone finishing a sentence and your app having the text matters. This is the time from sending each Hindi clip to getting its transcript back, measured from Gurugram.

Sarvam replied in a median of 0.60 s, OpenAI in 1.29 s and Azure in 2.28 s, for Hindi clips averaging 11.6 seconds. Part of this is distance and network routing, which I didn't measure separately. I called Azure's Central India region; I don't know where the other two served each request from. From another city or with another region, these numbers would change. All three were sent one clip at a time, without streaming, which is not how a live voice app would usually call them.

8. What an hour of audio costs

List prices from each service's official pricing page, checked on 25 October 2026. Move the slider to your monthly volume.

Converted at ₹94.94 per US dollar, the RBI reference rate on 23 October 2026. Three things are outside these numbers:

• Free tiers and credits. Azure has a free tier with a monthly allowance of audio hours, and other services offer starting credits. Good for testing; not the price at scale.
• Commitment discounts. Azure's commitment tiers cost less per hour if you pre-pay for volume. Check them if you expect hundreds of hours a month.
• Taxes, and anything else your app adds around the API, such as storage of the audio.

Sarvam's pricing page lists one Speech to Text rate and doesn't name a model version. I used that rate for saaras:v4.

This whole experiment, including the reruns and extra models, cost about $5 at these prices.

9. Before you ship a voice feature

Check the script of every transcript in code. It's a few lines, and it catches the failure that error rates hide:

def script_share(text, lo=0x0900, hi=0x097F):  # Devanagari range; change it per language
    letters = [ch for ch in text if ch.isalpha()]
    return sum(lo <= ord(ch) <= hi for ch in letters) / len(letters) if letters else 0.0

if script_share(transcript) < 0.5:
    ...  # retry with a script instruction, try another service, or ask the user to repeat

Say which script you want, not only which language. If the API accepts a prompt, write it in the target script.

Test with short clips. A test set of long, clean sentences will hide the problem. Include the 2-to-8-second clips your users actually send.

Test on your own audio. FLEURS is clean, read speech. Your users will speak over traffic, mix in English words, and use their own accents. Record 50 to 100 real clips with consent, write down what was said, and run them through the same steps.

Measure from where your users are, with the API style you'll use in production (streaming or not).

Plan for a bad answer. Empty transcripts, wrong scripts, invented sentences and long clips that the API won't take all happened in this test. Decide what your app does in each case.

10. Testing without surprises on the bill

Speech APIs are billed per minute of audio, and a loop over a folder of clips is easy to start and easy to forget. This is how I ran it:

• Keys live in a .env file, read from the environment and never written into code, committed to git, or pasted into a chat with an AI assistant. When an AI assistant helped run this test, it only checked whether each key was set, never its value.
• Budget alerts on every account before the first call. An alert tells you when spending passes a number. It doesn't stop the spending. Check whether each service offers a hard limit, and set it where it exists.
• A small run first. Five clips per language per service, to catch wrong language codes and audio-format errors before paying for the rest.
• Ask before paid calls. The test script prints how many minutes of audio it's about to send and waits for a yes.
• Save every raw answer before scoring. Changing a scoring rule then never means paying for the calls again.
• Expect refused requests and timeouts. Azure refused a few requests ("unauthorised" and "too many requests") that went through on retry, and one OpenAI call timed out. The script records failures and retries only those.
• Revoke the keys when the test is done.

11. What this test doesn't show

• Real-world speech. FLEURS is people reading clear sentences in quiet rooms. Noise, fast speech, accents and code-mixed Hinglish will all raise error rates, maybe unevenly across services.
• Whether a service has seen these sentences. FLEURS is a public benchmark. A service could have trained on it, which would make its scores look better than on new audio. One Marathi clip came back from Sarvam word for word, punctuation included. That proves nothing on its own, but it's a reason to test on your own recordings.
• Statistical certainty everywhere. Hindi has 390 clips; the other languages have 99 or 100, from one run. Small differences, such as Tamil between Sarvam and OpenAI, could change with more clips.
• Why short clips slip. The length pattern is clear; the reason isn't proven. Vocabulary may play a part, but this data can't separate the two.
• Google. I didn't test Google's Chirp models.
• Indian English. FLEURS English is US English.
• Live voice. I sent whole clips one at a time. Streaming APIs, which live voice features use, can behave differently.

Sources

• openai/whisper Discussion #1662: Hindi audio transcribed in Arabic or Persian script, including with the language set to Hindi.
• Radford et al., 2022, Robust Speech Recognition via Large-Scale Weak Supervision, Appendix E: Whisper's training hours per language.
• FLEURS: the recorded sentences (CC BY 4.0).
• Official pricing: Sarvam, OpenAI, Azure Retail Prices API; exchange rate from the RBI.
• OpenAI speech-to-text guide: note on limited parameter support for the newer transcription models.

The whole article in 10 lines

1. I sent the same recorded sentences in 5 languages to Sarvam, Azure and OpenAI: 788 scored clips each, 390 of them in Hindi.
2. Character error rate is the fairer measure for Indian languages, because words can be joined or split in more than one accepted way.
3. With the language set to Hindi, OpenAI's gpt-4o-transcribe returned 50 of 390 Hindi clips in another script, mostly Urdu. The same clips slipped again on a second run.
4. For clips under 8 seconds, it was 1 in 3. For clips over 16 seconds, none.
5. On its 340 Devanagari answers, OpenAI's Hindi error rate was 4.5%. The script was most of the problem.
6. One line asking for Devanagari cut wrong-script Hindi answers from 50 to 3, and errors from 12.0% to 5.0%, level with Azure.
7. In Marathi, the same line fixed the script but not the accuracy: 8.3%, against 4.4% for Sarvam.
8. Sarvam was most accurate in Hindi, Bengali and Marathi; Tamil was too close to call; OpenAI was most accurate in English.
9. List prices ran from ₹17.09 to ₹94.94 per hour of audio, before free tiers and commitment discounts.
10. Set the language, ask for the script, check the script in code, and test on short clips of your own users' audio.

Run on 25 October 2026 from Gurugram, India: Sarvam saaras:v4, OpenAI gpt-4o-transcribe, gpt-4o-mini-transcribe and whisper-1, Azure AI Speech standard real-time in Central India, through each service's REST API. Audio: FLEURS dev and test splits (CC BY 4.0), converted to 16 kHz mono 16-bit WAV. Error rates computed with jiwer 4.0.0 after Unicode NFC normalisation, lower-casing, dropping invisible format characters and removing punctuation, and checked against a separate implementation. Prices from each service's official pricing page and the Azure Retail Prices API on 25 October 2026, converted at the RBI reference rate of 23 October 2026. All numbers are real output; the code, raw answers and scoring rules are kept with the lab.