Automatic machine translation from spoken-language text to sign language gloss sequences

Polyakov A.M., Ryumin D.A. Automatic machine translation from spoken-language text to sign language gloss sequences. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2026, vol. 26, no. 3, pp. 475–485 (in Russian). doi: 10.17586/2226-1494-2026-26-3-475-485

This article addresses the task of machine translation from a spoken-language to Russian Sign Language in the form of an intermediate textual representation — a gloss sequence. The goal of this work is to develop a data preparation method and an automatic translation method for mapping verbal text to a sign-language gloss sequence. A manually annotated parallel corpus of “spoken-language text–gloss sequence” pairs is constructed. A gloss vocabulary is defined based on examples from a sign-language corpus and is used to constrain the set of admissible output tokens. Two model classes are compared: Transformers with an encoder-decoder architecture, adapted to the target task on the parallel corpus; and Large Language Models with a decoder-only architecture applied via In-Context Learning with a few examples and a prompt that includes instructions, the gloss vocabulary, and output-format constraints. Translation quality is evaluated using the BLEU metric on the test split of the parallel corpus. Experimental results show that Transformer-based models provide higher machine translation quality than Large Language Models; the best Transformer result is achieved by mT5-small (0.84). Among Large Language Models, the best value of 0.60 is obtained for GPT-5.2. The proposed method can be applied as part of a system for enabling digital bidirectional communication between sign-language users and spoken-language users. The method translates spoken-language text into a gloss sequence which can subsequently be synthesized using digital avatars to allow sign-language users to understand information that is spoken or written by spoken-language users. Source materials, the parallel corpus, and instructions for reproducing the experiments are available in the public repository dedicated to the method of automatic machine translation of texts from verbal language into a sequence of glosses.

1. Kapitanov A., Karina K., Nagaev A., Elizaveta P. Slovo: Russian sign language dataset. Lecture Notes in Computer Science, 2023, vol. 14253, pp. 63–73. doi: 10.1007/978-3-031-44137-0_6

2. Kagirov I., Ivanko D., Ryumin D., Axyonov A., Karpov A. TheRuSLan: Database of Russian sign language. Proc. of the 12^th Language Resources and Evaluation Conference, 2020, pp. 6079–6085.

3. Ryumin D., Ivanko D., Axyonov A., Kagirov I., Karpov A., Zelezny M. Human-robot interaction with smart shopping trolley using sign language: data collection. Proc. of the IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), 2019, pp. 949–954. doi: 10.1109/percomw.2019.8730886

4. Kagirov I.A., Ryumin D.A. Russian sign language database for clinical use: data and annotation peculiarities. NSU Vestnik. Series: Linguistics and Intercultural Communication, 2022, vol. 20, no. 3, pp. 90–108. (in Russian). doi: 10.25205/1818-7935-2022-20-3-90-108

5. Li D., Opazo C.R., Yu X., Li H. Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1448–1458. doi: 10.1109/wacv45572.2020.9093512

6. Sincan O.M., Keles H.Y. AUTSL: A Large scale multi-modal Turkish sign language dataset and baseline methods. IEEE Access, 2020, vol. 8, pp. 181340–181355. doi: 10.1109/ACCESS.2020.3028072

7. Kapitanov A., Kvanchiani K., Nagaev A., Kraynov R., Makhliarchuk A. HaGRID-HAnd Gesture Recognition Image Dataset. Proc. of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 4560–4569.  doi: 10.1109/WACV57701.2024.00451

8. Zhu D., Czehmann V., Avramidis E. Neural machine translation methods for translating text to sign language glosses. Proc. of the 61^st Annual Meeting of the Association for Computational Linguistics, 2023, pp. 12523–12541. doi: 10.18653/v1/2023.acl-long.700

9. Rust P., Shi B., Wang S., Camgoz N.C., Maillard J. Towards privacy-aware sign language translation at scale. Proc. of the 62^nd Annual Meeting of the Association for Computational Linguistics, 2024, pp. 8624–8641. doi: 10.18653/v1/2024.acl-long.467

10. Zhang B., Tanzer G., Firat O. Scaling sign language translation. Proc. of the 38^th International Conference on Neural Information Processing Systems, 2024, pp. 114018–114047.

11. Li Z., Zhou W., Zhao W., Wu K., Hu H., Li H. Uni-Sign: Toward unified sign language understanding at scale. Proc. of the 13^th International Conference on Learning Representations (ICLR), 2025, pp. 1–20.

12. Hamidullah Y., van Genabith J., España-Bonet C. Sign language translation with sentence embedding supervision. Proc. of the 62^nd Annual Meeting of the Association for Computational Linguistics, 2024, pp. 425–434. doi: 10.18653/v1/2024.acl-short.40

13. Hamidullah Y., Yazdani S., Oguz C., van Genabith J., España-Bonet C. SONAR-SLT: Multilingual sign language translation via language-agnostic sentence embedding supervision. Proc. of the 10^th Conference on Machine Translation, 2025, pp. 301–313. doi: 10.18653/v1/2025.wmt-1.18

14. Toshpulatov M., Lee W., Jun J., Lee S. Deep learning pathways for automatic sign language processing. Pattern Recognition, 2025, vol. 164, pp. 111475. doi: 10.1016/j.patcog.2025.111475

15. Gong J., Foo L.G., He Y., Rahmani H., Liu J. LLMs are good sign language translators. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 18362–18372. doi: 10.1109/CVPR52733.2024.01738

16. Hwang E.J., Cho S., Lee J., Park J.C. An efficient gloss-free sign language translation using spatial configurations and motion dynamics with LLMs. Proc. of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, 2025, pp. 3901–3920. doi: 10.18653/v1/2025.naacl-long.197

17. Guo J., Li P., Cohn T. Bridging sign and spoken languages: pseudo gloss generation for sign language translation. Proc. of the 39^th Conference on Neural Information Processing Systems (NeurIPS), 2025, pp. 1–29.

18. Baltatzis V., Potamias R.A., Ververas E., Sun G., Deng J., Zafeiriou S. Neural sign actors: a diffusion model for 3D sign language production from text. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 1985–1995. doi: 10.1109/CVPR52733.2024.00194

19. Tang S., Xue F., Wu J., Wang S., Hong R. Gloss-driven conditional diffusion models for sign language production. ACM Transactions on Multimedia Computing, Communications, and Applications, 2025, vol. 21, no. 4, pp. 105. doi: 10.1145/3663572

20. Ivanko D., Ryumin D. Intelligent system for automatic bidirectional sign language translation based on recognition and synthesis of audiovisual and sign speech. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2025, vol. XLVIII-2/W9-2025, pp. 131–136. doi: 10.5194/isprs-archives-xlviii-2-w9-2025-131-2025

21. Chipman H.A., George E.I., McCulloch R.E., Shively T.S. mBART: Multidimensional Monotone BART. Bayesian Analysis, 2022, vol. 17, no. 2, pp. 515–544. doi: 10.1214/21-BA1259

22. Tiedemann J., Aulamo M., Bakshandaeva D., Boggia M., Grönroos S.-A., Nieminen T., et al. Democratizing neural machine translation with OPUS-MT. Language Resources and Evaluation, 2024, vol. 58, no. 2, pp. 713–755. doi: 10.1007/s10579-023-09704-w

23. Xue L., Constant N., Roberts A., Kale M., Al-Rfou R., Siddhant A., et al. mT5: A Massively multilingual pre-trained text-to-text transformer. Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2021, pp. 483–498. doi: 10.18653/v1/2021.naacl-main.41

24. Papineni K., Roukos S., Ward T., Zhu W.-J. BLEU: a method for automatic evaluation of machine translation. Proc. of the 40^th Annual Meeting on Association for Computational Linguistics, 2002, pp. 311–318. doi: 10.3115/1073083.1073135

25. Post M. A call for clarity in reporting BLEU scores. Proc. of the 3^rd Conference on Machine Translation: Research Papers, 2018, pp. 186–191. doi: 10.18653/v1/W18-6319

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License