Building a Hindi Speech Model That Understands Context

Hindi is the fourth most-spoken language in the world, yet production-grade Hindi speech recognition that goes beyond raw transcription remains rare. Most systems can convert audio to text, but very few can tell you who is speaking, how they feel, or what they want — all from the same audio signal, in real time.

We built Whissle META-ASR v18 to solve exactly this: a single model that transcribes Hindi speech while simultaneously extracting speaker age, gender, emotion, intent, and named entities. In this post, we describe how we trained it, how it performs against two leading commercial APIs, and what makes a single-pass approach different from traditional pipelines.

The META Approach

META-ASR (Meta-aware Transcription with Entities) extends the standard CTC-based ASR architecture with two capabilities that run in parallel during inference:

• Inline entity tokens: The model's vocabulary includes special tokens like ENTITY_PERSON_NAME, ENTITY_CITY, and END that bracket named entities directly in the transcription output.
• Tag classifier head: A lightweight classification head attached to the encoder produces utterance-level predictions for AGE, GENDER, EMOTION, and INTENT without consuming any CTC decoder capacity.

The core encoder is a 17-layer FastConformer pretrained on 60K hours of multilingual audio, then fine-tuned on curated Hindi data. The tag classifier uses masked mean pooling over encoder outputs followed by per-category linear heads. Both outputs are produced in a single forward pass — no pipeline, no post-processing.

Training Data

We curated 223K Hindi utterances from a pool of 760K candidates. The curation pipeline uses diversity-aware selection (k-center greedy sampling on MFCC embeddings) to maximize coverage of speakers, topics, and acoustic conditions. Each utterance is annotated with:

Training Details

The model was fine-tuned on an NVIDIA A100 80GB GPU with the following configuration:

• Base model: FastConformer-CTC-BPE with language adapter (17 Conformer layers, 256-dim)
• Tokenizer: Aggregate SentencePiece (660 tokens covering Indo-Aryan scripts)
• Batch size: 24 utterances, max 20s duration
• Optimizer: AdamW with cosine annealing (peak LR 1e-4)
• Augmentation: SpecAugment + ambient noise perturbation (environmental sounds injected at random SNR)
• Tag classifier loss weight: 0.3 (30% of total loss allocated to metadata classification)

Early stopping at epoch 0 might seem surprising, but it reflects a common dynamic in fine-tuning from strong pretrained models: the initial adaptation captures the target domain well, and further training causes the model to overfit to training-set-specific patterns (we observed spurious END tokens appearing in predictions by epoch 2).

Benchmark Setup

We evaluated four systems head-to-head on two Hindi test sets:

Test sets:

• In-house validation (5,000 samples): Randomly sampled from our 36K-sample validation set. These samples include annotated entities, age, gender, emotion, and intent labels — allowing us to measure both transcription and metadata accuracy.
• FLEURS Hindi (418 samples): Google's standardized evaluation set for Hindi (google/fleurs hi_in test). Pure transcription evaluation — no metadata labels available.

Results

Tag Classification Performance

Beyond transcription, Whissle v18 extracts utterance-level metadata that no pure ASR system provides. Here's how the tag classifier performs on the in-house test set:

The tag classifier runs on pooled encoder representations — it adds no overhead since the encoder computation is already performed for CTC decoding. This is fundamentally different from pipeline approaches where a separate model would need to re-process the audio for each metadata dimension.

Sample Predictions

Here are real examples from the benchmark showing the model's output compared to ground truth:

What Makes This Different

The key differentiator isn't just accuracy — it's what you get out of a single inference pass:

On our in-house test set, Whissle v18 achieves the lowest WER (17.87%) while also extracting metadata — Gemini comes close at 18.75%, followed by Sarvam Saaras v3 at 19.02%, but neither provides metadata. On the standardized FLEURS Hindi benchmark, Sarvam leads (16.67%) followed closely by Gemini (17.42%), with Whissle at 21.25%. This is expected: FLEURS consists of clean, read-aloud speech that plays to the strengths of cloud APIs with massive pretraining data, while our in-house data reflects real-world conditions with ambient noise, spontaneous speech, and diverse speakers where Whissle's task-specific training excels. Deepgram trails on both benchmarks (26.68% and 24.13%), suggesting its Hindi coverage may not be as mature. Critically, Whissle runs at 6ms per sample — 400x faster than cloud APIs — making it viable for real-time edge deployment.

For applications like real-time voice assistants, call center analytics, or live captioning with context, this single-pass approach eliminates the need for orchestrating multiple models.

Try It Yourself

The Hindi META-ASR model is available through multiple channels:

• Whissle API: api.whissle.ai — production-ready REST API with streaming WebSocket support
• HuggingFace: WhissleAI/STT-meta-HI — download the model weights and run locally
• Self-hosted Docker: Our unified gateway image bundles ASR + agent + backend into a single container (deployment guide)

We're actively expanding to more languages — Mandarin, Bengali, Marathi, Punjabi, and European languages are in the pipeline. If you're building voice-first applications that need more than transcription, reach out — we'd love to hear what you're working on.