АНАЛИЗ МЕТОДОВ МНОГОМОДАЛЬНОГО ОБЪЕДИНЕНИЯ ИНФОРМАЦИИ ДЛЯ АУДИОВИЗУАЛЬНОГО РАСПОЗНАВАНИЯ РЕЧИ

В статье представлен аналитической обзор, охватывающий последние результаты, достигнутые в области аудиовизуального объединения (интеграции) многомодальной информации. Рассматриваются основные проблемы и обсуждаются методы их решения. Одной из важнейших задач аудиовизуальной интеграции является понимание того, как именно модальности взаимодействуют и влияют друг на друга. В данной работе этот вопрос рассматривается в контексте аудиовизуальной обработки речи, в особенности распознавания речи. В первой части обзора изложены базовые принципы аудиовизуального распознавания речи, приводится классификация типов аудио- и визуальных признаков речи. Отдельное внимание уделяется систематизации существующих способов и методов объединения аудиовизуальной информации. Во второй части, на основе проведенного анализа области исследований, приводится сводный список задач и приложений, использующих аудиовизуальное объединение с указанием методов, способов объединения информации и используемых аудио- и видеопризнаков. Предлагается структуризация методов аудиовизуальной интеграции по типам решаемых задач, а также обсуждаются преимущества и недостатки различных подходов. Приведены выводы, предложена оценка будущего развития области. В ходе дальнейших исследований планируется реализация системы аудиовизуального распознавания слитной русской речи с применением современных методов объединения многомодальной информации.

1. Katsaggelos A.K., Bahaadini S., Molina R. Audiovisual fusion: challenges and new approaches // Proc. of the IEEE. 2015. V. 103. N 9. P. 1635–1653. doi: 10.1109/JPROC.2015.2459017
2. Narayanan S., Alwan A. Noise source models for fricative consonants // IEEE Transactions on Speech and Audio Processing. 2000. V. 8. N 3. P. 328–344. doi: 10.1109/89.841215
3. Yehia H., Rubin P., Vatikiotis-Bateson E. Quantitative association of vocal-tract and facial behavior // Speech Communication. 1998. V. 26. N 1–2. P. 23–43.
4. McGurk H., MacDonald J. Hearing lips and seeing voices // Nature. 1976. V. 264. N 5588. P. 746–748.
5. Hershey J., Attias H., Jojic N., Kristjansson T. Audio-visual graphical models for speech processing // Proc. IEEE International Conference Acoustics, Speech and Signal Processing. 2004. V. 5. P. 649–652.
6. Nock H.J., Iyengar G., Neti C. Speaker localisation using audio-visual synchrony: an empirical study // Lec-ture Notes in Computer Science. 2003. V. 2728. P. 488–499.
7. Ngiam J., Khosla A., Kim M., Nam J., Lee H., Ng A.Y. Multimodal deep learning // Proc. 28th International Conference on Machine Learning. Bellevue, USA, 2011. P. 689–696.
8. Noda K., Yamaguchi Y., Nakadai K., Okuno H.G., Ogata T. Audio-visual speech recognition using deep learning // Application Intelligence. 2015. V. 42. N 4. P. 722–737. doi: 10.1007/s10489-014-0629-7
9. Nefian A.V., Liang L., Pi X., Liu X., Murphy K. Dynamic Bayesian networks for audio-visual speech recog-nition // EURASIP Journal on Advanced Signal Processing. 2002. V. 2002. N 11. P. 1274–1288. doi: 10.1155/S1110865702206083
10. Terry L., Katsaggelos A.K. A phone-viseme dynamic Bayesian network for audio-visual automatic speech recognition // Proc. 19th International Conference Pattern Recognition. 2008. Art. 4761927.
11. Ninomiya H., Kitaoka N., Tamura S., Iribe Y., Takeda K. Integration of deep bottleneck features for audio-visual speech recognition // Proc. 16th Annual Conference of the International Speech Communication Asso-ciation (Interspeech 2015). Dresden, Germany, 2015. P. 563–567.
12. Kalantari S., Dean D., Ghaemmaghami H., Sridharan S., Fookes C. Cross database training of audio-visual hidden Markov models for phone recognition // Proc. 16th Annual Conference of the International Speech Communication Association (Interspeech 2015). Dresden, Germany, 2015. P. 553–557.
13. Biswas A., Sahu P.K., Bhowmick A., Chandra M. AAM based features for multiple camera visual speech recognition in car environment // Procedia Computer Science. 2015. V. 57. P. 614–621. doi: 10.1016/j.procs.2015.07.417
14. Mroueh Y., Marcheret E., Goel V. Deep multimodal learning for audio-visual speech recognition // Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing. Brisbane, Australia, 2015. P. 2130–2134. doi: 10.1109/ICASSP.2015.7178347
15. Navarathna R., Dean D., Sridharan S., Lucey P. Multiple cameras for audio-visual speech recognition in an automotive environment // Computer Speech and Language. 2013. V. 27. N 4. P. 911–927. doi: 10.1016/j.csl.2012.07.005
16. Marcheret E., Potamianos G., Vopicka J., Goel V. Detecting audio-visual synchrony using deep neural net-works // Proc. 16th Annual Conference of the International Speech Communication Association (Interspeech 2015). Dresden, Germany, 2015. P. 548–552.
17. Aleksic P., Katsaggelos A. An audio-visual person identification and verification system using FAPS as visual features // Proc. ACM Workshop Multimodal User Authentication. 2003. P. 80–84.
18. Keating P.A. Underspecification in phonetics // Phonology. 1988. V. 5. N 2. P. 275–292.
19. Bengio S. Multimodal authentication using asynchronous HMMs // Lecture Notes in Computer Science. 2003. V. 2688. P. 770–777.
20. Kanak A., Erzin E., Yemez Y., Tekalp A.M. Joint audio-video processing for biometric speaker identification // Proc. IEEE International Conference on Acoustic Speech and Signal Processing. Hong Kong, 2003. V. 2. P. 377–380.
21. Chetty G., Wagner M. Audio-visual multimodal fusion for biometric person authentication and liveness veri-fication // Proc. NICTA-HCSNet Multimodal User Interaction Workshop. 2006. V. 57. P. 17–24.
22. Atrey P.K., Kankanhalli M.S., Jain R. Information assimilation framework for event detection in multimedia surveillance systems // Multimedia Systems. 2006. V. 12. N 3. P. 239–253. doi: 10.1007/s00530-006-0063-8
23. Xu H., Chua T.-S. Fusion of AV features and external information sources for event detection in team sports video // ACM Transactions on Multimedia Computing, Communications and Applications. 2006. V. 2. N 1. P. 44–67.
24. Shao X., Barker J. Stream weight estimation for multistream audio-visual speech recognition in a multispeaker environment // Speech Communication. 2008. V. 50. N 4. P. 337–353. doi: 10.1016/j.specom.2007.11.002
25. Petridis S., Rajgarhia V., Pantic M. Comparison of single-model and multiple-model prediction-based audio-visual fusion // Facial Analysis, Animation and Auditory-Visual Speech Processing (FAAVSP). Vienna, Aus-tria, 2015. P. 109–114.
26. Zou X., Bhanu B. Tracking humans using multi-modal fusion // Proc. IEEE Computer Society Conference Computer Vision and Pattern Recognition Workshops. San Diego, USA, 2005. P. 4–11. doi: 10.1109/CVPR.2005.545
27. Talantzis F., Pnevmatikakis A., Polymenakos L.C. Real time audio-visual person tracking // Proc. IEEE 8th Workshop Multimedia Signal Process. Victoria, Canada, 2006. P. 243–247. doi: 10.1109/MMSP.2006.285306
28. Vermaak J., Gangnet M., Blake A., Perez P. Sequential Monte Carlo fusion of sound and vision for speaker tracking // Proc. IEEE International Conference on Computer Vision. Vancouver, Canada, 2001. V. 1. P. 741–745.
29. Gatica-Perez D., Lathoud G., McCowan I., Odobez J.M., Moore D. Audio-visual speaker tracking with im-portance particle filters // Proc. IEEE International Conference on Image Processing. Barcelona, Spain, 2003. V. 3. P. 25–28.
30. Crisan D., Doucet A. A survey of convergence results on particle filtering methods for practitioners // IEEE Transactions on Signal Processing. 2002. V. 50. N 3. P. 736–746. doi: 10.1109/78.984773
31. Zotkin D.N., Duraiswami R., Davis L.S. Joint audio-visual tracking using particle filters // EURASIP Journal on Applied Signal Processing. 2002. V. 2002. N 11. P. 1154–1164. doi: 10.1155/S1110865702206058
32. Gehrig T., Nickel K., Ekenel H. K., Klee U., McDonough J. Kalman filters for audio-video source localization // Proc. IEEE Workshop on Applied Signal Processing to Audio and Acoustics. New Paltz, USA, 2005. P. 118–121. doi: 10.1109/ASPAA.2005.1540183
33. Nock H.J., Iyengar G., Neti C. Speaker localisation using audio-visual synchrony: an empirical study // Lec-ture Notes in Computer Science. 2003. V. 2728. P. 488–499.
34. Wu Y., Chang K.C., Chang E.Y., Smith J.R. Optimal multimodal fusion for multimedia data analysis // Proc. 12th ACM International Conference on Multimedia. New York, 2004. P. 572–579.
35. Adams W.H., Iyengar G., Lin C.-Y., Naphade M.R., Neti C., Nock H.J., Smith J.R. Semantic indexing of multimedia content using visual, audio, text cues // EURASIP Journal on Advanced Signal Processing. V. 2003. N 2. P. 170–185. doi: 10.1155/S1110865703211173
36. Iyengar G., Nock H.J., Neti C. Discriminative model fusion for semantic concept detection and annotation in video // Proc. 11th ACM International Conference on Multimedia. Berkeley, USA, 2003. P. 255–258.
37. Anderson B.D.O., Moore J.B. Optimal Filtering. NY: Courier Dover, 2012. 368 p.
38. Estellers V., Gurban M., Thiran J.-P. On dynamic stream weighting for audio-visual speech recognition // IEEE Transactions on Audio, Speech and Language Processing. 2012. V. 20. N 4. P. 1145–1157. doi: 10.1109/TASL.2011.2172427
39. Hsu W.H.-M., Chang S.-F. Generative, discriminative, ensemble learning on multi-modal perceptual fusion toward news video story segmentation // Proc. IEEE International Conference on Multimedia and Expo. Tai-pei, Taiwan, 2004. V. 2. P. 1091–1094.
40. Terry L.H., Livescu K., Pierrehumbert J.B., Katsaggelos A.K. Audio-visual anticipatory coarticulation mod-eling by human and machin // Proc. 11th Annual Conference of the International Speech Communication Association (Interspeech 2010). Makuhari, Japan, 2010. P. 2682–2685.
41. Terry L. Audio-Visual Asynchrony Modeling and Analysis for Speech Alignment and Recognition. Ph.D. dissertation. Evanston, USA, Northwestern University, 2011.
42. Кривонос Ю.Г., Крак Ю.В., Бармак А.В., Шкильнюк Д.В. Конструирование и идентификация элемен-тов жестовой коммуникации // Кибернетика и системный анализ. 2013. № 2. С. 3–14.
43. Zhou Z., Zhao G., Hong X., Pietikainen M. A review of recent advances in visual speech decoding // Image and Vision Computing. 2014. V. 32. N 9. P. 590–605. doi: 10.1016/j.imavis.2014.06.004
44. Dupont S., Luettin J. Audio-visual speech modeling for continuous speech recognition // IEEE Transactions on Multimedia. 2000. V. 2. N 3. P. 141–151. doi: 10.1109/6046.865479
45. Karpov A., Ronzhin A., Kipyatkova I. Designing a multimodal corpus of audio-visual speech using a high-speed camera // Proc. 11th IEEE Int. Conf. on Signal Processing. Beijing, China, 2012. P. 519–522. doi: 10.1109/ICoSP.2012.6491539
46. Karpov A., Kipyatkova I., Zelezny M. A framework for recording audio-visual speech corpora with a micro-phone and a high-speed camera // Lecture Notes in Computer Science. 2014. V. 8773. P. 50–57.
47. Карпов А.А. Реализация автоматической системы многомодального распознавания речи по аудио- и видеоинформации // Автоматика и телемеханика. 2014. Т. 75. № 12. С. 125–138.
48. Басов О.О., Карпов А.А. Анализ стратегий и методов объединения многомодальной информации // Информационно-управляющие системы. 2015. № 2(75). С. 7–14.
49. Ковшов Е.Е., Завистовская Т.А. Система обработки движения губ человека для речевого ввода ин-формации // Cloud of Science. 2014. Т. 1. № 2. С. 279–291.
50. Крак Ю.В., Тернов А.С. Чтение по губам в жестовой речи: синтез и анализ // Речевые технологии. 2014. № 2. С. 121–131.
51. Snoek C.G., Worring M., Smeulders A.W. Early versus late fusion in semantic video analysis // Proc. 13th Annual ACM International Conference on Multimedia. Singapore, 2005. P. 399–402. doi: 10.1145/1101149.1101236
52. Wu Z., Cai L., Meng H. Multi-level fusion of audio and visual features for speaker identification // Lecture Notes in Computer Science. 2005. V. 3832. P. 493–499.
53. Atrey P.K., Hossain M.A., Saddik A.E., Kankanhalli M.S. Multimodal fusion for multimedia analysis: a sur-vey // Multimedia Systems. 2010. V. 16. N 6. P. 345–379. doi: 10.1007/s00530-010-0182-0
54. Barnard M., Koniusz P., Wang W., Kittler J., Naqvi S.M., Chambers J. Robust multi-speaker tracking via dictionary learning and identity modeling // IEEE Transactions on Multimedia. 2014. V. 16. N 3. P. 864–880. doi: 10.1109/TMM.2014.2301977
55. Bayesian Network [Электронный ресурс]. Режим доступа: https://en.wikipedia.org/wiki/Bayesian_network, свободный. Яз. англ. (дата обращения 20.12.2015).
56. Zhao Y., Wang H., Ji Q. Audio-visual Tibetan speech recognition based on a deep dynamic Bayesian net-work for natural human robot interaction // International Journal of Advanced Robotic Systems. 2012. V. 9. N 258. P. 57–72. doi: 10.5772/54000
57. Noulas A.K., Krose B.J. EM detection of common origin of multi-modal cues // Proc. 8th ACM International Conference on Multimodal Interfaces. Banff, Canada, 2006. P. 201–208. doi: 10.1145/1180995.1181037
58. Dielmann A., Renals S. Automatic meeting segmentation using dynamic Bayesian networks // IEEE Transactions on Multimedia. 2007. V. 9. N 1. P. 25–36. doi: 10.1109/TMM.2006.886337
59. Bilmes J.A., Bartels C. Graphical model architectures for speech recognition // IEEE Signal Processing Mag-azine. 2005. V. 22. N 5. P. 89–100. doi: 10.1109/MSP.2005.1511827
60. Bengio S. Multimodal speech processing using asynchronous hidden Markov models // Information Fusion. 2004. V. 5. N 2. P. 81–89. doi: 10.1016/j.inffus.2003.04.001
61. Morency L.-P., de Kok I., Gratch J. A probabilistic multimodal approach for predicting listener backchannels // Autonomous Agents and Multi-Agent Systems. 2010. V. 20. N 1. P. 70–84. doi: 10.1007/s10458-009-9092-y
62. Casanovas A.L., Monaci G., Vandergheynst P., Gribonval R. Blind audiovisual source separation based on sparse redundant representations // IEEE Transactions on Multimedia. 2010. V. 12. N 5. P. 358–371. doi: 10.1109/TMM.2010.2050650
63. Liu Q., Wang W., Jackson P.J.B., Barnard M., Kittler J., Chambers J. Source separation of convolutive and noisy mixtures using audio-visual dictionary learning and probabilistic time-frequency masking // IEEE Transactions on Signal Processing. 2013. V. 61. N 22. P. 5520–5535. doi: 10.1109/TSP.2013.2277834
 

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License