- About the Project
- Results
- Tech Stack
- File Structure
- Dataset
- Model Architecture
- Installation and Setup
- Future Scope
- Acknowledgements
- Contributors
This project focuses on developing an advanced voice-to-English translation system that converts spoken input from any language into English text. The pipeline begins with Automatic Speech Recognition (ASR) models, which transcribe the spoken language into its original text form. Next, a Neural Machine Translation (NMT) model translates this transcript into fluent and contextually accurate English. The dataset is divided into training and testing sets to ensure fair and consistent evaluation. The system leverages modern deep learning architectures optimized through extensive hyperparameter tuning to maximize transcription and translation accuracy. Finally, the project includes a real-time speech translation feature, enabling seamless live audio input and instant English output.
https://github.com/harsh-sankhe/Voice2English/blob/main/Media/Testing_Video.mp4
| Model | WER (↓) | BLEU Score (↑) |
|---|---|---|
| ASR (CTC) + NMT | 18.4% | 31.2 |
| ASR (Transformer) + NMT | 14.7% | 37.8 |
| Category | Technologies |
|---|---|
| Programming Languages |  |
| Frameworks |  |
| Libraries |   |
| Deep Learning Models |     |
| Dataset |  |
| Tools |    |
| Visualization & Analysis |  |
├── Dataset_Preprocessing
├── Preprocessing_ASR(Transformer)
├── Preprocessing_ASR(Transformer)
├── Mini_Projects
├── fruit_classifier.ipynb
├── alpa_classi.ipynb
├── miniproject_lstm.ipynb
├── EmotionalRecognizer.ipynb
├── Model_Architecture
├── VOICE2ENGLISH_TRANSFORMER.ipynb
├── VOICE2ENGLISH_CTC.ipynb
├── VOICE2ENGLISH_NMT.ipynb
├── Model_Evaluation
├── Model_Testing_Transformer
├── Model_Testing_CTC
├── Notes
├── Purvasha_Notes
├── Vaishnavi_Notes
├── Harsh_Notes
├── Media
├── README.md
git clone https://github.com/shnavii11/Voice2English.git
cd Voice2English
pip install -r requirements.txtOpen the notebooks in Google Colab or Kaggle for GPU access.
The Bhaashaanuvad dataset is designed to support research in multilingual speech-to-text and translation systems. It focuses on converting spoken audio from various Indian and global languages into accurate transcriptions and their corresponding English translations. Below is a breakdown of its structure and contents:
Audio Samples: Contains recordings of spoken sentences in multiple languages (such as Hindi, Marathi, Bengali, Tamil, and others).
Transcriptions: Each audio file is paired with a text transcript of the original language, facilitating ASR (Automatic Speech Recognition) training.
Translations: Provides parallel English translations of the transcripts, enabling effective NMT (Neural Machine Translation) training.
Purpose: Built to train and evaluate end-to-end speech translation pipelines, particularly systems that integrate ASR and NMT models for real-time multilingual voice translation.
Format: Data is organized in JSON and CSV formats, with fields for audio_path, source_text, and translated_text.
├──DatasetDict (Root)
├── 📁 hindi (88,566 rows)
├── chunked_audio_filepath → path to each audio chunk (.wav)
├── text → original transcript (in Hindi)
├── pred_text → model-predicted transcript
├── audio_filepath → full original audio file path
├── start_time → starting time (of chunk in seconds)
├── duration → chunk duration (in seconds)
├── alignment_score → matching score between audio & text
├── en_text → English translation of transcript
Download the Bhaashaanuvad dataset on HuggingFace
Format 1: ASR (BiLSTM + CTC + CNN) + NMT (Transformer)
-
Convolutional Neural Network (CNN) Initial convolutional layers extract low-level spectral and temporal features from the input spectrograms or MFCCs, capturing important acoustic patterns.
-
Bidirectional Long Short-Term Memory (BiLSTM) The BiLSTM layers process the extracted features in both forward and backward directions, enabling the model to understand context from the entire sequence of speech frames.
-
Connectionist Temporal Classification (CTC) The CTC layer aligns predicted sequences with variable-length input audio. It enables training without requiring precise frame-level alignment, producing a sequence of transcribed text characters or phonemes.
-
Reshape & Output Layer (ASR Output) The final tensor output from the CTC decoder represents the transcribed text in the original spoken language.
-
Transformer-based Neural Machine Translation (NMT) The ASR transcript is fed into a Transformer-based NMT model, which uses multi-head attention and positional encoding to translate the source-language text into fluent English.
-
Decoder Layer (English Output) The final decoder produces the translated English text, leveraging both encoder context and learned attention mechanisms for accuracy.
Format 2: ASR (Transformer) + NMT (Transformer)
-
Feature Extraction (Spectrogram/MFCC) The input audio is converted into spectrograms or MFCC representations, forming the foundation for transformer-based speech modeling.
-
Transformer Encoder (ASR) The ASR model uses a pure transformer encoder with self-attention mechanisms to capture both local and global speech dependencies without recurrence.
-
CTC or Attention-based Decoder (ASR Output) Depending on configuration, the decoder generates either character-level transcriptions (CTC-based) or word-level outputs using attention mechanisms.
-
Intermediate Text Representation The recognized speech text (original language) becomes the input to the translation model, ensuring modularity and flexibility.
-
Transformer Encoder-Decoder (NMT) A second Transformer model takes the recognized text as input, encoding the source sentence and decoding it into English. It leverages multi-head attention, positional encoding, and feed-forward layers for context-aware translation.
-
Final Output Layer The NMT decoder outputs a sequence of English words or tokens, forming the final translated sentence.
-
Multilingual Expansion: Extend the current system to support more regional and global languages, enabling seamless translation between multiple language pairs and making it highly adaptable for diverse linguistic users.
-
Real-Time Speech Translation Interface: Develop a user-friendly web or mobile interface that allows users to speak in any language and receive instant English translations — making the system practical for live communication, travel, and accessibility tools.
-
Emotion & Context Awareness: Incorporate paralinguistic features like tone, pitch, and emotion recognition to produce translations that better capture the speaker’s intent and sentiment, not just the literal meaning.
-
Pre-trained Model Fine-tuning: Fine-tune state-of-the-art pre-trained models like Whisper (OpenAI) for ASR and mBART/mT5 for NMT on the Bhaashaanuvad dataset to potentially achieve significantly lower WER and higher BLEU scores with less training time.
-
Edge Deployment: Optimize and quantize the models for deployment on edge devices (mobile phones, embedded systems) using techniques like knowledge distillation, pruning, and INT8 quantization to enable offline translation without internet access.
We would like to express our gratitude to all the tools and courses which helped in successful completion of this project.
Research Papers
- https://huggingface.co/learn/audio-course/en/chapter3/introduction
- https://keras.io/examples/audio/transformer_asr/
- https://keras.io/examples/audio/ctc_asr/
Videos
Courses
A special thanks to our project mentors Sourish Phate, Niharika and to the entire Project X community for unwavering support and guidance throughout this journey.