Back to Projects

Speech Emotion Recognition on Low-Resource Chittagonian Dialect

A Hybrid SSL-Acoustic Approach with a Newly Created Voice Dataset

RESEARCH / SPEECH AIBuilt during: Speech AI Researcher & Machine Learning Engineer
Tags -
AI/MLSpeech AIPyTorchPythonIndicWav2VecAudio ProcessingLibrosaSelf-Supervised LearningDeep LearningResearch
Source

View the source code, research documentation, and benchmarks:

View Source Code

Project Overview

Existing speech emotion recognition (SER) research focuses predominantly on standard languages and high-resource corpora. Prior to this research, no computational study addressed emotion recognition in low-resource Bengali dialects such as Chittagonian, which exhibits distinct phonetics, intonations, and prosodic characteristics.

This work addresses the research gap by creating a novel Chittagonian emotional speech corpus covering five emotional states: Neutral, Happy, Sad, Angry, and Fear.

The proposed hybrid feature engine fuses 1024-D IndicWav2Vec-Bengali self-supervised learning (SSL) embeddings with spectral flux and mean fundamental frequency (F0) into a 1026-D vector, combined with gender-aware pitch shifting, additive noise, and emotion-specific time stretching to overcome class imbalance.

Key Features

  • First Chittagonian SER Corpus — Created and annotated the first Chittagonian emotional speech corpus across five core emotions: Neutral, Happy, Sad, Angry, and Fear.
  • Hybrid Feature Engine (1026-D) — Fuses 1024-D IndicWav2Vec-Bengali SSL embeddings with spectral flux and mean F0 into a comprehensive 1026-D feature vector.
  • Cross-Speaker Generalization — Engineered deep and prosodic representations that generalize across diverse native speakers on dialectal speech.
  • Targeted Audio Augmentation — Gender-aware pitch shifting, additive acoustic noise, and emotion-specific time stretching (Sad slower; Happy and Fear faster).
  • Neutral Bias Mitigation — Overcame severe Neutral class bias by generating approximately 300 synthetic samples for minority emotion classes.

Technical Highlights

  • IndicWav2Vec SSL Integration — Utilizes pre-trained self-supervised acoustic representations trained on Indic language corpora for low-resource feature extraction.
  • Prosodic & Spectral Fusion — Combines latent representations with fundamental frequency (mean F0) and spectral flux to capture regional dialect intonation contours.
  • Arousal-Specific Temporal Augmentation — Applies targeted temporal modifications based on emotional arousal dynamics, accelerating high-arousal and decelerating low-arousal samples.

Challenges & Solutions

Challenge: Research Gap Addressed (Lack of Dialect SER Datasets): Existing speech emotion recognition research focuses on standard languages; no prior work computationally addressed Bengali dialects like Chittagonian.
Solution: Created the first Chittagonian emotional speech corpus covering five emotions (Neutral, Happy, Sad, Angry, and Fear) with balanced speaker demographics.
Result: Established the inaugural benchmark dataset for speech emotion recognition in regional Bengali dialects.
Challenge: Dialect Phonetic Variance & Small Sample Size: Acoustic models trained on standard languages fail to generalize across non-standard dialect speech.
Solution: Engineered a hybrid feature engine fusing 1024-D IndicWav2Vec-Bengali SSL embeddings with spectral flux and mean F0 into a 1026-D representation.
Result: Created a robust model that reliably captures emotional prosody across diverse speakers on dialect speech.
Challenge: Severe Neutral Bias in Natural Speech: Real-world recordings heavily favored Neutral speech, leaving minority classes like Happy and Fear severely underrepresented.
Solution: Implemented targeted gender-aware pitch shifting, additive noise, and emotion-specific time stretching (Sad slower; Happy/Fear faster) on non-Neutral classes.
Result: Synthesized ~300 high-quality emotional samples, breaking the Neutral bias and significantly elevating classification recall.

Tech Stack Breakdown

frontend

Matplotlib, Librosa Display (Waveforms, Spectrograms, Pitch Contours)

backend

Python, PyTorch, IndicWav2Vec, Librosa, SoundFile

database

NumPy, SciPy, Pandas, Scikit-learn

tools

Audacity, Git, Jupyter Notebook, Google Colab Pro