Meta-aware Voice Action Model

For AI companions to feel truly interactive, speech recognition needs to do more than just transcribe words. It must understand context, adapt on the fly, and act in real time. While models like Whisper are great at transcription, they often miss crucial metadata like speaker changes, named entities, or user intents which are vital for natural conversation.

Existing approaches often require complex pipelines of multiple models, and while recent multi-modal LLMs can understand context, they suffer from high latency. This makes real-time, context-aware voice interaction a challenge.

The Challenge with Traditional ASR

Modern end-to-end ASR systems are excellent at transcription but fall short in real-world conversations where contextual metadata is key. The common challenges include:

• Cascading Pipelines: Most voice applications use a multi-step process where transcription is just the first step, followed by separate models for understanding intent, entities, and sentiment. This is often slow and inefficient.
• Loss of Information: Important acoustic and prosodic cues (like tone and emotion) that are crucial for understanding human behavior are often lost during simple transcription.
• Latency in LLMs: While powerful, large multi-modal language models that can understand context often have high latency, making them unsuitable for real-time applications.

How META-1 Works

META-1 is a multimodal, context-aware speech recognition model that simultaneously transcribes speech and annotates it with contextual tags like intents, named entities, and speaker changes. It builds upon pre-trained speech encoders but extends their vocabulary with special <user-defined-symbols>. These symbols are used to predict various tasks alongside transcription.

The model is fine-tuned on pairs of speech recordings and annotated text. The text contains both the transcription and the user-defined symbols in-line. Here’s a look at the core components:

• Feature Extractor: An MFCC-based feature extractor processes raw audio into latent representations.
• Voice Encoder: A Transformer or Conformer-based encoder further processes these features.
• Multi-Task Decoder: A single autoregressive decoder generates both the transcription and the contextual tags. The special symbols are part of the decoder's vocabulary, allowing it to mark events like entities, intents, or speaker changes directly in the output.

This single-stage approach means the model outputs a final, enriched transcript without needing any post-processing, making it highly efficient.

Architecture for Real-Time Performance

To ensure real-time operation, META-1 uses a two-part design:

• Backbone Transformer: A large encoder (based on Llama or Conformer) processes speech frames and previously generated tokens to capture long-range contextual dependencies.
• Lightweight Decoder: A smaller, specialized decoder head generates both transcription and metadata tokens at each step. This design keeps latency low, even with a large backbone, by amortizing the computational cost.

Efficient Training with Compute Amortization

Training the model with long audio contexts and metadata can be resource-intensive. We use a compute amortization scheme to manage this:

• The full backbone is trained on every input frame to maintain high transcription quality.
• The decoder is selectively trained for tagging tasks on a random subset of frames (e.g., 1 in 16).

This approach significantly improves training efficiency without a noticeable drop in performance.

Datasets

We compiled a diverse, multilingual dataset from several public sources to train and evaluate our models, enabling them to understand and process speech in multiple languages for global applications.

Results and Conclusion

Our model achieves competitive Word Error Rates (WER) on benchmarks like SLUE and SLURP, while also excelling at inserting user-defined symbols for entity and intent tagging. The integrated approach leads to higher F1 scores for entity and intent detection compared to multi-stage systems.

META-1 simplifies the deployment of context-aware voice applications by eliminating the need for separate NLP or diarization modules. Our ongoing work focuses on integrating external language models for even better performance and extending the model to more languages and complex conversational scenarios.