Enhancing Automatic Speech Recognition for Child Speech in Low-Resource Languages

Abstract

Automatic Speech Recognition (ASR) for children faces severe domain mismatch challenges due to anatomical differences (shorter vocal tract length, higher fundamental frequency $F_0$) and cognitive developmental speech characteristics [3, 4, 5]. In low-resource languages such as Persian, this bottleneck is compounded by an acute scarcity of standardized child speech corpora [2].

This research implements and evaluates a multi-stage hybrid text post-processing framework alongside direct multimodal audio-to-LLM inference and Parameter-Efficient Fine-Tuning (PEFT/LoRA) [16] on Whisper acoustic models [1, 13]. In baseline evaluations on the Persian Child Speech Dataset (PSRB) [2], raw offline models (Vosk and Whisper Base) suffered from extreme acoustic hallucinations, phonetic confusion, and Word Error Rates (WER) exceeding $100\%$. To overcome these challenges, we engineered a multi-tier text correction pipeline combining exact lexicon rules, Levenshtein fuzzy matching ($d \le 2$) [11], and constrained LLM context smoothing (Gemini 3.5 Flash) [14], which dramatically improved baseline accuracy.

Furthermore, passing raw audio directly to multimodal LLMs (Gemini 3.5 Flash Direct) registered the single best overall system performance, achieving a $27.52\%$ WER and $12.90\%$ CER. Experimental evaluation of LoRA fine-tuning [16] revealed a striking structural contrast: while LoRA adaptation yielded a $4.92\%$ absolute WER improvement on the compact Whisper Base model (74M parameters), fine-tuning the 1.5B parameter Whisper Large-v3 model induced severe overfitting and catastrophic forgetting of prior linguistic representation due to data scarcity.

Our findings demonstrate that under low-resource constraints, deploying structured linguistic post-processing pipelines or direct multimodal models offers a vastly superior operational pathway compared to acoustic fine-tuning of oversized models.

Keywords: Automatic Speech Recognition (ASR), Child Speech Processing, Persian (Farsi) NLP, Contextual LLM Post-Processing, Direct Multimodal Audio-to-LLM, Parameter-Efficient Fine-Tuning (PEFT), LoRA Adaptation, Acoustic Domain Shift.

Interactive Speech Correction Playground

Chapter 1: Introduction & Research Background

1.1 Context & Background

Natural Language Processing (NLP) and Automatic Speech Recognition (ASR) have undergone foundational shifts in recent years. The transition from classical statistical frameworks—such as Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs)—to deep learning architectures based on Recurrent Neural Networks (RNNs) and self-attention Transformers has elevated machine comprehension of human speech to unprecedented accuracy levels [1].

Despite massive gains in high-resource languages like English, engineering high-precision ASR systems for low-resource languages remains a major hurdle. Persian (Farsi), despite being spoken by tens of millions worldwide, suffers from significant morphological variations, complex orthographic rules (such as Zero-Width Non-Joiners / ZWNJ), and a pronounced shortage of standardized child speech datasets [2].

1.2 Problem Statement & Anatomical Mismatch

Processing child speech is recognized as one of the most intricate challenges in speech processing [3]. Pre-trained baseline ASR models available today (e.g., OpenAI Whisper [1], Wav2Vec 2.0, Vosk) are almost exclusively trained on hundreds of thousands of hours of adult speech [1]. Child speech deviates dramatically from this adult training distribution due to both physical [4] and cognitive factors:

1. Anatomical Distortions: Children possess significantly shorter vocal tract lengths, which elevates their fundamental pitch frequency ($F_0$) and shifts acoustic formant frequencies ($F_1, F_2, F_3$) higher [5].
2. Developmental Phonological Deviations: Young children regularly introduce phoneme substitutions (e.g., replacing the liquid phoneme $/r/$ with $/l/$, pronouncing "لفتـم"→"رفتـم"), vowel deletions, and colloquial contractions.

When standard ASR models encounter child audio, the acoustic domain shift triggers catastrophic misrecognition. Empirical baseline tests conducted during this thesis on child audio from the Persian Child Speech Dataset (PSRB) [2] revealed raw Word Error Rates (WER) ranging between $78\%$ and $102.77\%$, rendering off-the-shelf ASR practically unusable.

Furthermore, Persian orthography introduces secondary textual complications [2]:

• Separated vs. Joined Words: Inconsistent use of ZWNJ (e.g., "می‌نویسم" vs. "مینویسم") creates artificial errors in standard string metrics.
• Colloquial vs. Formal Registers: Children speak exclusively in informal conversational registers ("می‌خوام" instead of "می‌خواهم").
• The "Hah-e-Kasreh" Phenomenon: Misspelling short vowel kasreh as the silent letter "ه" (e.g., "کتابِ من" as "کتابه من").

1.3 Literature Review & Scientific Background

1.3.1 Frequency Perturbation in Child Speech (Abaskohi et al., 2022) [3]

Abaskohi et al. demonstrated that standard ASR fails on children due to acoustic pitch shift. They introduced Random Frequency Pitch (RFP) augmentation during Wav2Vec 2.0 training. This inspired our acoustic preprocessing layer to standardize sample rates and normalize amplitude.

1.3.2 Generative Error Correction via Retrieval-Augmented Generation / GEC-RAG (Robatian et al., 2025) [6]

Robatian et al. formulated ASR output correction as a black-box LLM post-processing task, querying past error patterns from a knowledge base to prompt an LLM using in-context learning. This highlighted the importance of our proposed LLM post-processing layer.

1.3.3 Error-Level Noise Vector Modeling (Rahmani et al., 2025) [7]

Rahmani et al. extracted the 5-best hypothesis sequences from Whisper and calculated entropy across variants to construct Error-Level Noise (ELN) embeddings for downstream LLM cleanup. In our pipeline, extracting candidate words via fuzzy matching directly draws from this insight.

1.3.4 Phonetically-Aware Persian Correction / PERCORE (Dashti et al., 2024) [8]

The PERCORE framework combined phonological vector representations with deep learning to correct real-word homophone spelling mistakes in Persian (e.g., distinguishing "خوار" vs. "خار").

1.4 Research Objectives

The overarching objective of this thesis is to design, implement, and quantitatively benchmark a high-precision, scalable software framework to dramatically improve Persian child speech recognition without requiring costly full-model audio retraining from scratch.

The research methodology spans four core pillars:

1. Benchmark baseline zero-shot acoustic models (Vosk, Whisper Base, Whisper Large-v3).
2. Develop a multi-stage post-processing pipeline combining exact lexicon mapping, Levenshtein fuzzy string matching [11], and LLM prompt engineering [12].
3. Benchmark direct multimodal audio-to-LLM inference pipelines (Qwen2-Audio, Gemini 2.5 Flash, Gemini 3.5 Flash) [14].
4. Conduct Parameter-Efficient Fine-Tuning (PEFT/LoRA) [16] on Whisper architectures using real and synthesized child speech data [15].

Chapter 2: Methodology & Proposed Architecture

2.1 System Architecture Overview

To rectify acoustic hallucinations and phonological errors without retraining heavy models [10], we engineered a multi-tier asynchronous post-processing pipeline [2, 7]. The 5-stage architecture operates as follows:

1. Speech-to-Text Inference: Transcribe raw audio to text using Whisper with example-based prompts.
2. Exact Lexicon Correction: Apply rule-based phonetic dictionary mappings for common child mispronunciations.
3. Fuzzy String Matching: Catch unseen phoneme mispronunciations using Levenshtein distance ($d \le 2$).
4. LLM Contextual Cleanup: Pass candidate text to Gemini 3.5 Flash for grammar, spacing, and context smoothing.

2.2 Acoustic Standardization & Model Selection

Before audio signals reach the inference engine, incoming files are normalized to a uniform single-channel (mono) format at a $16\text{ kHz}$ sampling rate [13]. Volume peak normalization is applied to mitigate intense amplitude fluctuations caused by children varying their distance to the microphone.

For acoustic transcription, we selected models spanning different parameter scales [1]:

• Whisper Base (74M parameters): Highly lightweight and ultra-fast, serving as our primary baseline for resource-constrained edge execution.
• Whisper Large-v3 (1.5B parameters): A zero-shot state-of-the-art multilingual model providing high acoustic robustness.
• Vosk (Offline Hybrid): Lightweight offline GMM-HMM/TDNN engine with fixed lexicon limits.

Decoding temperature was fixed to $0.0$ ($temperature=0.0$) for deterministic output [13].

Example-Based Acoustic Prompting

Injecting structured example prompts directly into the ASR decoder significantly improves context awareness prior to text post-processing. Replacing raw key-word lists with full structured child dialogue sentences reduced initial acoustic hallucinations:

Vocabulary-based Prompt: "مدرسه، مامان، بابا، بازی، خوراکی، خاله، اسباب‌بازی..."
Example-based Prompt: "سلام خوبی؟ من امروز رفتم مدرسه با دوستام بازی کردم. مامانم برام خوراکی خریده بود. خاله زهرا هم اومده بود خانه‌مان."

2.3 Classic Post-Processing: Lexicon & Fuzzy Matching

Exact Lexicon Matching

A specialized dictionary maps recurring child phoneme errors directly to standard Persian orthography:

• "لفتم"→"رفتـم" (Phoneme substitution $/r/ \rightarrow /l/$)
• "مدلسه" / "ملاسه"→"مدرسه" (Consonant reduction)
• "پدل"→"پدر", "مادل"→"مادر"

Fuzzy String Matching (Levenshtein Distance)

Fuzzy matching uses the Levenshtein edit distance metric $d(s_1, s_2)$ [11] with a strict maximum edit distance threshold:

$d(s_1, s_2) \le 2$

Where the edit distance $d(a, b)$ between strings $a$ and $b$ is computed as:

$m_{i,j} = \begin{cases} \max(i, j) & \text{if } \min(i,j) = 0, \\ \min \begin{cases} m_{i-1,j} + 1 \\ m_{i,j-1} + 1 \\ m_{i-1,j-1} + \mathbb{I}(a_i \neq b_j) \end{cases} & \text{otherwise.} \end{cases}$

This allowed the pipeline to catch unseen phonological variants, such as repairing "بزولگ"→"بزرگ" and "خاهش"→"خواهش".

2.4 LLM Contextual Cleanup & Direct Audio-to-LLM

Even with exact and fuzzy matching, fixing complex grammatical errors (such as Hah-e-Kasreh, colloquial verb stems, and split compound words) requires semantic reasoning [12]. The output of the classical phase is passed to a Large Language Model (Gemini 3.5 Flash) [14].

Direct Multimodal Audio-to-LLM Architecture

In addition to the cascaded ASR + LLM post-processing pipeline, we evaluated a single-stage end-to-end multimodal architecture [13, 14]. By feeding the raw audio signal directly to Gemini 3.5 Flash via API, the model bypassed the intermediate textual ASR layer entirely, processing acoustic pitch, stress, and semantics in a single forward pass.

2.5 Multi-Stage Pipeline Algorithm

1.S_clean = normalize_audio(S, 16000)
2.T_raw = whisper_decode(S_clean, prompt="example-based")
3.T_exact = exact_lexicon_replace(T_raw, dict)
4.T_fuzzy = fuzzy_match(T_exact, max_dist=2)
5.T_final = llm_cleanup(T_fuzzy, system="Persian Corrector")
6.return T_final

Chapter 3: Experimental Evaluation & Findings

3.1 Experimental Setup & Datasets

To model real-world low-resource constraints, experiments were conducted using two primary datasets:

1. PSRB Child Subset (Natural Speech Benchmark) [2]: 14 natural speech utterances extracted from the Persian Child Speech Dataset (PSRB). Partitioned into Training (11 samples), Validation (1 sample), and Independent Test (2 samples).
2. DSP-UT (University of Tehran Child Corpus) [15]: Single-word recordings of preschool children ("آهو", "قطار", "قیچی", "هواپیما"). To generate synthetic training sentences for LoRA adaptation, an audio stitching algorithm concatenated 6 randomly selected words per sentence with $0.2\text{s}$ silence gaps, synthesizing 100 augmented training samples.

Evaluation metrics include Word Error Rate (WER), Character Error Rate (CER), and Mean Edit Distance (MED):

$\text{WER} = \frac{S + D + I}{N} \times 100\%$

$\text{CER} = \frac{S_c + D_c + I_c}{N_c} \times 100\%$

Where $S$ is Substitutions, $D$ is Deletions, $I$ is Insertions, and $N$ is total reference count.

3.2 Baseline Acoustic Model Benchmarks

Analysis of Baseline Failures

• Over-generation in Whisper Base (WER > 100%): High pitch and unseen formants caused Whisper Base to hallucinate rhyming words and loop endlessly, causing insertion errors ($I$) to exceed the total reference length ($N$).
• Fixed Lexicon Traps in Vosk: Vosk produced a lower WER than Whisper Base (94.69%), but a much worse CER (66.28%) because its static dictionary substituted unrecognised child sounds with phonetically adjacent adult words.
• Scale Resilience in Whisper Large-v3: The 1.5B model achieved 54.89% WER zero-shot, proving that scale improves acoustic pitch invariant representations.

3.3 Direct Audio-to-LLM Evaluation

Why Qwen2-Audio Failed vs. Gemini Succeeded

Local Qwen2-Audio suffered from linguistic hallucination (generating long English/Arabic disclaimers) because its audio encoder was trained on minimal Persian child audio. Conversely, Gemini 3.5 Flash Direct achieved the single lowest error rate of the entire thesis (27.52% WER / 12.90% CER), demonstrating the immense power of unified end-to-end multimodal perception.

3.4 Parameter-Efficient Fine-Tuning (LoRA) Analysis

We evaluated Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) adapters [16] on Whisper architectures. Hyperparameters were tuned via random search ($LR = 5 \times 10^{-4}, r=8, \alpha=16$).

Evaluated on an independent, unseen test dataset:

The Capacity Paradox & Catastrophic Forgetting

This striking contrast provides a critical scientific finding:

1. Whisper Base (74M): Limited parameter capacity prevented drastic weight divergence. LoRA adapters successfully learned basic acoustic adaptation for high-pitched child formants, yielding a $4.92\%$ absolute WER gain.
2. Whisper Large-v3 (1.5B): Possesses immense parameter capacity and rich prior Persian linguistic knowledge. When exposed to a tiny training dataset (11 real + 100 stitched samples), the large model rapidly overfitted on speaker-specific pitch traits and suffered catastrophic forgetting of general Persian language syntax, causing WER to surge from $26.23\%$ to $45.90\%$.

3.5 Comprehensive Comparative Discussion

Combining all benchmarked approaches highlights essential engineering trade-offs:

• Resource-Constrained Edge Deployments: Whisper Base + Classic Lexicon Matching + Gemini LLM Post-Processing reduces WER from $102.77\%$ to $68.88\%$ with minimal local compute overhead.
• Maximum Accuracy (Cloud API): Gemini 3.5 Flash Direct delivers peak performance ($27.52\%$ WER), though it relies on closed-source cloud endpoints [14].
• Open-Source Strategy: For privacy-sensitive deployments, fine-tuning lightweight models (Whisper Base) or combining them with linguistic post-processing is far safer than fine-tuning massive models on small datasets.

Chapter 4: Conclusion & Future Horizons

4.1 Key Findings Summary

This thesis addressed the critical bottleneck of automatic speech recognition for Persian child speech under low-resource constraints:

1. Raw offline acoustic baselines (Vosk, Whisper Base) fail severely on child speech (WER $> 100\%$) due to acoustic domain shift and phoneme substitution [3, 4].
2. Lexicon-grounded post-processing combining exact mapping, Levenshtein fuzzy matching [11], and Gemini LLM cleanup [14] successfully recovers speech accuracy without costly acoustic retraining.
3. Direct audio-to-LLM multimodal models (Gemini 3.5 Flash) achieve state-of-the-art results ($27.52\%$ WER), proving the superiority of unified acoustic-linguistic perception.
4. PEFT/LoRA fine-tuning [16] is effective on lightweight models (Whisper Base WER down by $4.92\%$), but triggers catastrophic overfitting on oversized models (Whisper Large) under low-data regimes.

4.2 Future Research Directions

Based on our findings, we propose five key avenues for future investigation:

1. Human-in-the-Loop (HITL) & Reinforcement Learning (RLHF/DPO): Build an entropy-based confidence scoring system that flags low-confidence child transcriptions for human verification, automatically expanding the lexicon dictionary.
2. Comprehensive Persian Child Speech Corpus: Collect, annotate, and standardize a multi-speaker natural Persian child speech dataset across diverse age groups and dialects.
3. Phonological Embedding Integration (PERCORE-style) [8]: Embed phonetic distance vectors directly into the LLM prompt layer to prevent semantic drift during misspelling correction.
4. Streaming ASR for Educational Robotics: Adapt the multi-stage post-processing pipeline for real-time, low-latency continuous speech streams in child-robot interaction.
5. Open-Source Multimodal Audio Model Fine-Tuning: Fine-tune open-source multimodal audio-LLMs (such as Qwen2-Audio or LLaMA-Omni) on dedicated Persian child audio to achieve cloud-level accuracy in private, offline settings.

References & Bibliography

1. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv preprint arXiv:2212.04356. https://arxiv.org/abs/2212.04356
2. Nima Sedghiyeh, Sara Sadeghi, Reza Khodadadi, Farzin Kashani, Omid Aghdaei, Somayeh Rahimi, Mohammad Sadegh Safari. (2025). PSRB: A Comprehensive Benchmark for Evaluating Persian Automatic Speech Recognition Systems. arXiv preprint arXiv:2505.21230. https://arxiv.org/abs/2505.21230
3. Amirhossein Abaskohi, Fatemeh Mortazavi, Hadi Moradi. (2022). Automatic Speech Recognition for Speech Assessment of Persian Preschool Children. arXiv preprint arXiv:2203.12886. https://arxiv.org/abs/2203.12886
4. Sungbok Lee, Alexandros Potamianos, Shrikanth S. Narayanan. (1997). Analysis of children's speech: duration, pitch and formants. Fifth European Conference on Speech Communication and Technology (EUROSPEECH 1997). https://doi.org/10.21437/EUROSPEECH.1997-161
5. Sungbok Lee, Alexandros Potamianos, Shrikanth S. Narayanan. (1999). Acoustics of children's speech: Developmental changes of temporal and spectral parameters. The Journal of the Acoustical Society of America (JASA), 105(3), 1455-1468. https://doi.org/10.1121/1.426686
6. Amin Robatian, Mohammad Hajipour, Mohammad Reza Peyghan, Fatemeh Rajabi, Sajjad Amini, Shahrokh Ghaemmaghami, Iman Gholampour. (2025). GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems. arXiv preprint arXiv:2501.10734. https://arxiv.org/abs/2501.10734
7. Zahra Rahmani, Hossein Sameti. (2025). Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognition. arXiv preprint arXiv:2512.17247. https://arxiv.org/abs/2512.17247
8. Seyed Mohammad Sadegh Dashti, Amid Khatibi Bardsiri, Mehdi Jafari Shahbazzadeh. (2024). PERCORE: A Deep Learning-Based Framework for Persian Spelling Correction with Phonetic Analysis. International Journal of Computational Intelligence Systems, 17, 114. https://arxiv.org/abs/2407.14789
9. Hassan Haji Mohammadi, Alireza Talebpour, Ahamd Mahmoudi Aznaveh, Samaneh Yazdani. (2023). Mehr: A Persian Coreference Resolution Corpus. Journal of AI and Data Mining, 11(3), 413-424.
10. Rao Ma, Mengjie Qian, Mark Gales, Kate Knill. (2024). ASR error correction using large language models. arXiv preprint arXiv:2409.09554. https://arxiv.org/abs/2409.09554
11. Vladimir I. Levenshtein. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8), 707-710.
12. Majid Adibian, Saeedeh Momtazi. (2022). Converting Persian Informal Text to Formal Using Transformer-Based Neural Networks. Journal of Language and Linguistics, 18(35), 1-23.
13. OpenAI. (2024). Speech to text - OpenAI API Documentation. OpenAI Platform Guides. https://platform.openai.com/docs/guides/speech-to-text
14. Google AI. (2024). Gemini API Documentation - Models and Multi-modal Capabilities. Google AI for Developers. https://ai.google.dev/models/gemini
15. M. Khanzadi, H. Veisi, R. Alinaghizade, Z. Soleymani. (2022). Persian Phoneme and Syllable Recognition using Recurrent Neural Networks for Phonological Awareness Assessment. Journal of AI and Data Mining (JAD), 10(1), 117-126. https://jad.shahroodut.ac.ir/article_2338.html
16. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=nZeStateee