Karelian speech recognition system with support for Karelian-Russian code-switching

Kipyatkova I.S., Dolgushin M.D., Kiseleva K.O., Kagirov I.A. Karelian speech recognition system with support for Karelian-Russian code-switching. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2026, vol. 26, no. 4, pp. 826–834 (in Russian). doi: 10.17586/2226-1494-2026-26-4-826-834

This paper focuses on the development of an automatic speech recognition system for the Livvi-Karelian variety of the Karelian language, as it is spoken under conditions of code-switching between Karelian and Russian. The study of bilingual speech recognition methods is carried out. In order to improve the quality of speech recognition, a methodology for training text data augmentation via partial translation and intra-word code-switched wordforms artificial synthesis was developed. Acoustic modeling was performed by fine-tuning a pre-trained multilingual Wav2Vec2-BERT 2.0 model with the use of the data from two previously collected corpora containing 7.5 hours of speech. Fine-tuning was performed using the Transformers framework. When developing the language model, in order to address the problem of limited code-switching data, an augmentation method was applied based on partial automatic translation of Karelian texts into Russian, followed by the generation of word-forms with intra-word code-switching based on special linguistic rules. On the base of formulated rules, a list of words with intra-word code-switching was generated for a language model. The experiments showed that using a full vocabulary that includes generated hybrid word forms yields a consistent improvement in results. A further reduction in word error rate to 25.82 % on the development set and 29 % on the test portion of the corpus was achieved through linear interpolation of the Karelian language model with the Russian language model (interpolation weight 0.7). The conducted experiments confirm the effectiveness of the developed methodology for developing a bilingual speech recognition system. In particular, it is recommended to combine fine-tuning of multilingual acoustic models, text augmentation with morphological rules, and language model interpolation. The proposed approach can be applied to developing speech recognition systems for other low-resource languages of Russia spoken in an unbalanced bilingual environment.

1. Mabokela K.R., Manamela M.J., Manaileng M. Modeling code-switching speech on under-resourced languages for language identification. Proc. of the 4^th Workshop on Spoken Language Technologies for Under Resourced Languages (SLTU), 2014, pp. 225-230.

2. Bhuvanagirir K., Kopparapu S.K. Mixed language speech recognition without explicit identification of language. American Journal of Signal Processing, 2012, vol. 2, no. 5, pp. 92-97. doi: 10.5923/j.ajsp.20120205.02

3. Agro M.T., Kulkarni A.A., Kadaoui K., Talat Z., Aldarmaki H. Code-switching in End-to-End automatic speech recognition: A systematic literature review. arXiv, 2025. arXiv:2507.07741. doi: 10.48550/arXiv.2507.07741

4. Hsieh I.-T., Wu C.-H., Wang C.-H. Acoustic and textual data augmentation for code-switching speech recognition in under-resourced language. Proc. of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2020, pp. 302–307.

5. Hamed I., Habash N., Abdennadher S., Vu N.T. Investigating lexical replacements for Arabic-English code-switched data augmentation. Proc. of the 6^th Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT), 2023, pp. 86–100. doi: 10.18653/v1/2023.loresmt-1.7

6. Hussein A., Chowdhury S.A., Abdelali A., Dehak N., Ali A., Khudanpur S. Textual data augmentation for Arabic-English code-switching speech recognition. Proc. of the IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 777–784. doi: 10.1109/SLT54892.2023.10023313

7. Pratapa A., Bhat G., Choudhury M., Sitaram S., Dandapat S., Bali K. Language modeling for code-mixing: The role of linguistic theory based synthetic data. Proc. of the 56^th Annual Meeting of the Association for Computational Linguistics, 2018, vol. 1, pp. 1543–1553. doi: 10.18653/v1/P18-1143

8. Chuang S.-P., Sung T.-W., Lee H.-Y. Training code-switching language model with monolingual data. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7949–7953. doi: 10.1109/ICASSP40776.2020.9053775

9. Yilmaz E., van Den Heuvel H., van Leeuwen D.A. Acoustic and textual data augmentation for improved ASR of code-switching speech. Proc. of the 19^th Annual Conference of the International Speech Communication Association Interspeech, 2018, pp. 1933–1937. doi: 10.21437/Interspeech.2018-52

10. Chang C.-T., Chuang S.-P., Lee H.-Y. Code-switching sentence generation by generative adversarial networks and its application to data augmentation. Proc. of the 20^th Annual Conference of the International Speech Communication Association Interspeech, 2019, pp. 554–558. doi: 10.21437/Interspeech.2019-3214

11. Şahin G.G. To augment or not to augment? A comparative study on text augmentation techniques for low-resource NLP. Computational Linguistics, 2022, vol. 48, no. 1, pp. 5–42. doi: 10.1162/coli_a_00425

12. Wan Z., Wan X., Peng W., Li R. New datasets and controllable iterative data augmentation method for code-switching ASR error correction. Findings of the Association for Computational Linguistics: EMNLP, 2023, pp. 8075–8087. doi: 10.18653/v1/2023.findings-emnlp.543

13. Kobayashi S. Contextual augmentation: Data augmentation by words with paradigmatic relations. Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, vol. 2, pp. 452–457. doi: 10.18653/v1/N18-2072

14. Hamed I., DenisovP., Li C.-Y., ElmahdyM., Abdennadher S, Vu N.T. Investigations on speech recognition systems for low-resource dialectal Arabic–English code-switching speech. Computer Speech & Language, 2022, vol. 72, pp. 101278. doi: 10.1016/j.csl.2021.101278

15. Ubskii D., Matveev Y., Minker W. Impact of using a bilingual model on Kazakh-Russian code-switching speech recognition. CEUR Workshop Proceedings, 2020, vol. 2590, pp. 1-6.

16. Karpov A.A., Kipyatkova I.S. Methodology for estimation of automatic speech recognition system performance. Journal ofInstrument Engineering, 2012, vol. 55, no. 11, pp. 38–43. (in Russian)

17. Arkhangelskiy T. Low-resource ASR with an augmented language model. Proc. of the 7^th International Workshop on Computational Linguistics of Uralic Languages, 2021, pp. 40–46.

18. Wolf T., Debut L., Sanh V., Chaumond J., Delangue C., Moi A., et al. Transformers: State-of-the-art natural language processing. Proc. of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45. doi: 10.18653/v1/2020.emnlp-demos.6

19. Kipyatkova I., Kagirov I., Dolgushin M. Use of pre-trained multilingual models for Karelian speech recognition. Informatics and Automation, 2025, vol. 24, no. 2, pp. 604–630. (in Russian). doi: 10.15622/ia.24.2.9

20. Kagirov I.A., Kiseleva K.O., Kipyatkova I.S. Code-switching analysis in Karelian language speakers' speech. Terra Linguistica, 2025, vol. 16, no. 2, pp. 22–40. (in Russian). doi: 10.18721/JHSS.16202

21. Kipyatkova I., Kiseleva K., Dolgushin M., Kagirov I. Modeling intra-word code-switching for Karelian ASR. Lecture Notes in Computer Science, 2025, vol. 16188, pp. 104–117. doi: 10.1007/978-3-032-07959-6_8

22. Kipyatkova I. Experimenting with hybrid TDNN/HMM acoustic models for Russian speech recognition. Lecture Notes in Computer Science, 2017, vol. 10458, pp. 362–369. doi: 10.1007/978-3-319-66429-3_35

23. Kipyatkova I.S., Rodionova A.P., Kagirov I.A., Krizhanovsky I.A. Speech and text data preparation for development of an automatic speech recognition system for the Karelian language. Proceedings of Petrozavodsk State University, 2023, vol. 45, no. 5, pp. 89–98. (in Russian). doi: 10.15393/uchz.art.2023.924

24. Kipyatkova I., Karpov A. Lexicon size and language model order optimization for Russian LVCSR. Lecture Notes in Computer Science, 2013, vol. 8113, pp. 219–226. doi: 10.1007/978-3-319-01931-4_29

25. Boyko T., Zaitseva N., Krizhanovskaya N., Krizhanovsky A., Novak I., Pellinen N., et al. The open corpus of the Veps and Karelian languages: Overview and applications. KnE Social Sciences, 2022, pp. 29–40. doi: 10.18502/kss.v7i3.10419

26. Stolcke A., Zheng J., Wang W., Abrash V. SRILM at sixteen: update and outlook. Proc. of the IEEE Automatic Speech Recognition and Understanding Workshop, 2011, pp. 5–9.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License