Menu
Publications
2024
2023
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
Editor-in-Chief
Nikiforov
Vladimir O.
D.Sc., Prof.
Partners
doi: 10.17586/2226-1494-2022-22-3-559-566
Improving sign language processing via few-shot machine learning
Read the full article
';
Article in English
For citation:
Abstract
For citation:
Shovkoplias G.F., Strokov D.A., Kasantsev D.V., Vatian A.S., Asadulaev A.A., Tomilov I.V., Shalyto A.A., Gusarova N.F. Improving sign language processing via few-shot machine learning. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2022, vol. 22, no. 3, pp. 559–566. doi: 10.17586/2226-1494-2022-22-3-559-566
Abstract
Improving the efficiency of communication of deaf and hard of hearing people by processing sign language using artificial intelligence is an important task both socially and technologically. One of the ways to solve this problem is a fairly cheap and accessible marker method. The method is based on the registration of electromyographic (EMG) muscle signals using bracelets worn on the arm. To improve the quality of recognition of gestures recorded by the marker method, a modification of the marker method is proposed — duplication of EMG sensors in combination with a low-frame machine learning approach. We experimentally study the possibilities of improving the quality of processing of sign language by duplicating EMG sensors as well as by reducing the volume of the dataset required for training machine learning tools. In the latter case, we compare several technologies of the few-shot approach. Our experiments show that training with few-shot neural nets on 56k samples we can achieve better results than training on random forest with 160k samples. The use of a minimum number of sensors in combination with few-shot signal processing techniques provides the possibility of organizing quick and cost-effective interaction with people with hearing and speech disabilities.
Keywords: sign language processing, few-shot machine learning, bracelets, marker methods
Acknowledgements. This research was supported by Priority 2030 Federal Academic Leadership Program.
References
Acknowledgements. This research was supported by Priority 2030 Federal Academic Leadership Program.
References
1. Agrawal S.C., Jalal A.S., Tripathi R.K. A survey on manual and non-manual sign language recognition for isolated and continuous sign // International Journal of Applied Pattern Recognition. 2016. V. 3. N 2. P. 99–134. https://doi.org/10.1504/ijapr.2016.079048
2. Bragg D., Koller O., Bellard M., Berke L., Boudrealt P., Braffort A., Caselli N., Huenerfauth M., Kacorri H., Verhoef T., Vogler C., Morris M.R. Sign language recognition, generation, and translation: An interdisciplinary perspective // Proc. of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). 2019. P. 16–31. https://doi.org/10.1145/3308561.3353774
3. Kamal S.M., Chen Y., Li S., Shi X., Zheng J. Technical approaches to Chinese sign language processing: A review // IEEE Access. 2019. V. 7. P. 96926–96935. https://doi.org/10.1109/ACCESS.2019.2929174
4. O’Connor T.F., Fach M.E., Miller R., Root S.E., Mercier P.P., Lipomi D.J. The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics // PLoS ONE. 2017. V. 12. N 7. P. e0179766. https://doi.org/10.1371/journal.pone.0179766
5. Song Y., Lee S., Choi Y., Han S., Won H., Sung T.-H., Choi Y., Bae J. Design framework for a seamless smart glove using a digital knitting system // Fashion and Textiles. 2021. V. 8. N 1. P. 6. https://doi.org/10.1186/s40691-020-00237-2
6. Zhou Z., Chen K., Li X., Zhang S., Wu Y., Zhou Y., Meng K., Sun C., He Q., Fan W., Fan E., Lin Z., Tan X., Deng W., Yang J., Chen J. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays // Nature Electronics. 2020. V. 3. N 9. P. 571–578. https://doi.org/10.1038/s41928-020-0428-6
7. Bernhardt P. Myo SDK Beta 7 [Электронный ресурс]. URL: https://developerblog.myo.com/myo-sdk-beta-7/ (дата обращения: 10.02.2022).
8. Abreu J.G., Teixeira J.M., Figueiredo L.S., Teichrieb V. Evaluating sign language recognition using the Myo armband // Proc. of the 18th Symposium on Virtual and Augmented Reality (SVR). 2016. P. 64–70. https://doi.org/10.1109/SVR.2016.21
9. Wang Y., Yao Q., Kwok J., Ni L.M. Generalizing from a few examples: A survey on few-shot learning // ACM Computing Surveys. 2020. V. 53. N 3. P. 63. https://dl.acm.org/doi/10.1145/3386252
10. Wang F., Zhao S., Zhou X., Li C., Li M., Zeng Z. An recognition–verification mechanism for real-time Chinese sign language recognition based on multi-information fusion // Sensors. 2019. V. 19. N 11. P. 2495. https://doi.org/10.3390/s19112495
11. Kim S., Kim J., Ahn S., Kim Y. Finger language recognition based on ensemble artificial neural network learning using armband EMG sensors // Technology and Health Care. 2018. V. 26. S. 1. P. 249–258. https://doi.org/10.3233/THC-174602
12. Paudyal P., Lee J., Banerjee A., Sandeep K.S. A comparison of techniques for sign language alphabet recognition using armband wearables // ACM Transactions on Interactive Intelligent Systems. 2019. V. 9. N 2-3. P. 1–26. https://doi.org/10.1145/3150974
13. Tateno S., Liu H., Ou J. Development of sign language motion recognition system for hearing-impaired people using electromyography signal // Sensors. 2020. V. 20. N 20. P. 5807. https://doi.org/10.3390/s20205807
14. Sheng X., Lv B., Guo W., Zhu X. Common spatial-spectral analysis of EMG signals for multiday and multiuser myoelectric interface // Biomedical Signal Processing and Control. 2019. V. 53. P. 101572. https://doi.org/10.1016/j.bspc.2019.101572
15. Zhang L., Shi Y., Wang W., Chu Y., Yuan X. Real-time and user-independent feature classification of forearm using EMG signals // Journal of the Society for Information Display. 2019. V. 27. N 2. P. 101–107. https://doi.org/10.1002/jsid.749
16. Das P., Paul S., Ghosh J., Palbhowmik S., Neo-Gi B., Ganguly A. An approach towards the representation of sign language by electromyography signals with fuzzy implementation // International Journal of Sensors, Wireless Communications and Control. 2017. V. 7. N 1. P. 26–32. https://doi.org/10.2174/2210327907666170222093839
17. Côté-Allard U., Fall C.L., Drouin A., Campeau-Lecours A., Gosselin C., Glette K., Laviolette F., Gosselin B. Deep learning for electromyographic hand gesture signal classification using transfer learning // IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2019. V. 27. N 4. P. 760–771. https://doi.org/10.1109/TNSRE.2019.2896269
18. Tsinganos P., Cornelis B., Cornelis J., Jansen B., Skodras A. Data augmentation of surface electromyography for hand gesture recognition // Sensors. 2020. V. 20. N 17. P. 4892. https://doi.org/10.3390/s20174892
19. Li W., Shi P., Yu H. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-of-the-art, challenges, and future // Frontiers in Neuroscience. 2021. V. 15. P. 621885 https://doi.org/10.3389/fnins.2021.621885
20. Rahimian E., Zabihi S., Asif A., Farina D., Atashzar S.F., Mohammadi A. FS-HGR: Few-shot learning for hand gesture recognition via electromyography // IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2021. V. 29. P. 1004–1015. https://doi.org/10.1109/TNSRE.2021.3077413
21. Finn C., Abbeel P., Levine S. Model-agnostic meta-learning for fast adaptation of deep networks // Proceedings of Machine Learning Research. 2017. V. 70. P. 1126–1135.
22. Lee Y., Choi S. Gradient-based meta-learning with learned layerwise metric and subspace // Proc. of the 35th International Conference on Machine Learning (ICML). V. 7. 2018. P. 4574–4586.
23. Koch G., Zemel R., Salakhutdinov R. Siamese neural networks for one-shot image recognition // ICML Deep Learning Workshop. V. 2. 2015.
24. Snell J., Swersky K., Zemel R. Prototypical networks for few-shot learning // Advances in Neural Information Processing Systems. 2017. P. 4077–4087.
25. Vaezi Joze H.R., Koller O. MS-ASL: A large-scale data set and benchmark for understanding American sign language // Proc. of the 30th British Machine Vision Conference (BMVC). 2019.
26. De Coster M., Van Herreweghe M., Dambre J. Sign language recognition with transformer networks // Proc. 12th International Conference on Language Resources and Evaluation. LREC. 2020. P. 6018–6024.
27. Pigou L., Van Herreweghe M., Dambre J. Sign classification in sign language corpora with deep neural networks // Proc. of the International Conference on Language Resources and Evaluation (LREC), Workshop. 2016. P. 175–178.
28. Pradhan A., He J., Jiang N. Performance optimization of surface electromyography based biometric sensing system for both verification and identification // IEEE Sensors Journal. 2021. V. 21. N 19. P. 21718–21729. https://doi.org/10.1109/JSEN.2021.3079428
29. Young A.J., Hargrove L.J., Kuike T.A. Improving myoelectric pattern recognition robustness to electrode shift by changing interelectrode distance and electrode configuration // IEEE Transactions on Biomedical Engineering. 2012. V. 59. N 3. P. 645–652. https://doi.org/10.1109/TBME.2011.2177662
30. Benatti S., Farella E., Gruppioni E., Benini L. Analysis of robust implementation of an EMG pattern recognition based control // Proc. of the Analysis of Robust Implementation of an EMG Pattern Recognition Based Control (BIOSIGNALS). 2014. P. 45–54. https://doi.org/10.5220/0004800300450054
31. George J.A., Neibling A., Paskett M.D., Clark G.A. Inexpensive surface electromyography sleeve with consistent electrode placement enables dexterous and stable prosthetic control through deep learning // arXiv. 2003. arXiv:2003.00070. https://doi.org/10.48550/arXiv.2003.00070
32. Vinyals O., Blundell C., Lillicrap T., Kavukcuoglu K., Wierstra D. Matching networks for one shot learning // Advances in Neural Information Processing Systems. 2016. P. 3637–3645.
33. Kaczmarek P., Mańkowski T., Tomczyński J. putEMG–A surface electromyography hand gesture recognition dataset // Sensors. 2019. V. 19. N 16. P. 3548. https://doi.org/10.3390/s19163548
