ENG
РУС
Search
Menu
About the Edition
Open Access Statement
Editorial Board
Editorial Policy
Editorial Ethics
Rules for peer-review
Rules for presentation of papers
Forms of Submitted Documents
Recommendations for the Design of References to Our Journal
Editorial Staff
Information for Subscribers
Summaries of the Issue
List of Authors
Issues
Publishers license agreement
Procedure of work with personal data
Publications
2024
2023
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
Editor-in-Chief

Nikiforov
Vladimir O.
D.Sc., Prof.
Partners













































doi: 10.17586/2226-1494-2020-20-2-163-176

ANALYTICAL REVIEW OF METHODS FOR EMOTION RECOGNITION BY HUMAN FACE EXPRESSIONS

E. V. Ryumina, A. A. Karpov


Read the full article  ';
Article in Russian

For citation:
Ryumina E.V., Karpov A.A. Analytical review of methods for emotion recognition by human face expressions. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2020, vol. 20, no. 2, pp. 163–176 (in Russian). doi: 10.17586/2226-1494-2020-20-2-163-176


Abstract
Recognition of human emotions by facial expressions is an important research problem that covers many areas and disciplines, such as computer vision, artificial intelligence, medicine, psychology and security. This paper provides an analytical overview of video facial expression databases and approaches to recognition emotions by facial expressions, which include three main stages of image analysis, such as pre-processing, feature extraction and classification. The paper presents both traditional approaches to recognition of human emotions by visual facial features, and approaches based on deep learning using deep neural networks. We give the current results of some existing algorithms. In the review of scientific and technical literature we empathized mainly the sources containing theoretical and research information of the methods under consideration, as well as comparison of traditional methods and methods based on deep neural networks, which were supported by experimental studies. Analysis of scientific and technical literature describing methods and algorithms for study and recognition of facial expressions, as well as the results of world scientific research, have shown that traditional methods for classification of facial expressions are second in speed and accuracy to artificial neural networks. The main contribution of this review is providing a common understanding of modern approaches to recognition of facial expressions, which will enable new researchers to understand the main components and trends in the field of recognition of facial expressions. Moreover, comparison of world scientific findings has shown that a combination of traditional approaches and approaches based on deep neural networks achieves better classification accuracy, but artificial neural networks are the best classification methods. The paper may be useful to specialists and researchers in the field of computer vision.

Keywords: digital image processing, classification, facial expression recognition, feature extraction, deep neural networks, computational paralinguistics

Acknowledgements. This research was supported by the Russian Science Foundation (project No.18-11-00145).

References
  1. Varma S., Shinde M., Chavan S.S. Analysis of PCA and LDA features for facial expression recognition using SVM and HMM classifiers.Techno-Societal2018. Proc. 2nd International Conference on Advanced Technologies for Societal Applications,vol. 1,2020, pp. 109–119. doi: 10.1007/978-3-030-16848-3_11
  2. Yin D.B.M., Mukhlas A.A., Chik R.Z.W., Othman A.T., Omar S. A proposed approach for biometric-based authentication using of face and facial expression recognition. Proc. IEEE 3rd International Conference on Communication and Information Systems (ICCIS 2018),Singapore, 2018, pp. 28–33. doi: 10.1109/ICOMIS.2018.8644974
  3. Dino H.I., Abdulrazzaq M.B. Facial expression classification based on SVM, KNN and MLP classifiers. Proc. International Conference on Advanced Science and Engineering (ICOASE 2019), Zakho-Duhok, Iraq, 2019, pp. 70–75. doi: 10.1109/ICOASE.2019.8723728
  4. Tripathi A., Pandey S. Efficient facial expression recognition system based on geometric features using neural network. Lecture Notes in Networks and Systems, 2018, vol. 10, pp. 181–190. doi: 10.1007/978-981-10-3920-1_18
  5. Greche L., Es-Sbai N., Lavendelis E. Histogram of oriented gradient and multi layer feed forward neural network for facial expression identification. Proc. International Conference on Control, Automation and Diagnosis (ICCAD 2017), Hammamet, Tunisia, 2017,pp. 333–337. doi: 10.1109/CADIAG.2017.8075680
  6. Tselikova S.O., Gorozhankin Ya.P., Ivanov A.O., Mironov A.A., Akhremchik Ya.V. Neural network technologies in automatic recognition of emotions. Young Scientist, 2019, no. 26, pp. 59–61. Available at: https://moluch.ru/archive/264/61173/ (accessed: 12.12.2019). (in Russian)
  7. Stepanova O., Ivanovsky L., Khryashchev V. Deep learning and convolutional neural networks for facial expression analysis. DSPA, 2018, vol. 8, no. 4, pp. 170–173. (in Russian).
  8. Talegaonkar I., Joshi K., Valunj S., Kohok R. Kulkarni A. Real time facial expression recognition using deep learning. Proc. of International Conference on Communication and Information Processing (ICCIP), 2019. Available at:https://ssrn.com/abstract=3421486 (accessed: 13.12.2019). doi: 10.2139/ssrn.3421486
  9. Jumani S.Z., Ali F., Guriro S., Kandhro I.A., Khan A., Zaidi A. Facial expression recognition with histogram of oriented gradients using CNN. Indian Journal of Science and Technology, 2019, vol. 12, no. 24, pp. 1–8.  doi: 10.17485/ijst/2019/v12i24/145093
  10. Babu D.R., Shankar R.S., Mahesh G., Murthy K.V.S.S. Facial expression recognition using bezier curves with hausdorff distance.Proc. IEEE International Conference on IoT and Application (ICIOT 2017), Nagapattinam, India, 2017, pp. 8073622. doi: 10.1109/ICIOTA.2017.8073622
  11. Cao H., Cooper D.G., Keutmann M.K., Gur R.C., Nenkova A., Verma R.CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 2014, vol. 5, no. 4, pp. 377–390. doi: 10.1109/TAFFC.2014.2336244
  12. Perepelkina O., Kazimirova E., Konstantinova М.RAMAS: Russian multimodal corpus of dyadic interaction for affective computing. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018, vol. 11096, pp. 501–510.doi:10.1007/978-3-319-99579-3_52
  13. Livingstone S.R., Russo F.A. The Ryerson audio-visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 2018, vol. 13, no. 5, pp. e0196391. doi: 10.1371/journal.pone.0196391
  14. Viola P., Jones M.J. Robust real-time face detection.International Journal of Computer Vision, 2004, vol. 57, no. 2, pp. 137–154. doi: 10.1023/B:VISI.0000013087.49260.fb
  15. Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C.-Y., Berg A.C. SSD: single shot multibox detector. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016, vol. 9905, pp. 21–37. doi:10.1007/978-3-319-46448-0_2
  16. Déniz O., Bueno G., Salido J., De la Torre F. Face recognition using histograms of oriented gradients. Pattern Recognition Letters, 2011, vol. 32, no. 12, pp. 1598–1603. doi: 10.1016/j.patrec.2011.01.004
  17. King D.E Max-margin object detection.Available at: https://arxiv.org/pdf/1502.00046.pdf (accessed: 13.12.2019).
  18. Alexandrov A.A., Kirpichnikov A.P., Lyasheva S.A., Shleymovich M.P. Analyzing the emotional states of a person in an image. Herald of Technological University, 2019, vol. 22, no. 8, pp. 120–123. (in Russian)
  19. Voronov V., Strelnikov V., Voronova L., Trunov A., Vovik A. Faces 2D-recognition аnd identification using the HOG descriptors method. Proc. 24th Conference of Open Innovations Association FRUCT, 2019, pp. 783–789.
  20. Mohan P.G., Prakash C., Gangashetty S.V. Bessel transform for image resizing. Proc. 18th International Conference on Systems, Signals and Image Processing (IWSSIP 2011),Sarajevo, Bosnia-Herzegovina, 2011, pp. 75–78.
  21. Owusu E., Abdulai J.-D., Zhan Y. Face detection based on multilayer feed‐forward neural network and Haar features. Software: Practice and Experience, 2019, vol. 49, no. 1, pp. 120–129. doi:10.1002/spe.2646
  22. Su J., Gao L., Li W., Xia Y., Cao N., Wang R. Fast face tracking-by-detection algorithm for secure monitoring. Applied Sciences, 2019, vol. 9, no. 18, pp. 3774. doi:10.3390/app9183774
  23. Lowe D.G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, vol. 60, no. 2, pp. 91–110. doi:10.1023/B:VISI.0000029664.99615.94
  24. Hernandez-Matamoros A., Bonarini A., Escamilla-Hernandez E., Nakano-Miyatake M., Perez-Meana H. A facial expression recognition with automatic segmentation of face regions. Communications in Computer and Information Science, 2015, vol. 532, pp. 529–540. doi: 10.1007/978-3-319-22689-7_41
  25. Naz S., Ziauddin S., Shahid A.R. Driver fatigue detection using mean intensity, SVM, and SIFT. International Journal of Interactive Multimedia and Artificial Intelligence, 2019, vol. 5, no. 4, pp. 86–93. doi: 10.9781/ijimai.2017.10.002
  26. Priya R.V. Emotion recognition from geometric fuzzy membership functions. Multimedia Tools and Applications, 2019, vol. 78, no. 13, pp. 17847–17878. doi: 10.1007/s11042-018-6954-9
  27. Wang X., Chen L. Contrast enhancement using feature-preserving bi-histogram equalization. Signal Image and Video Processing, 2018, vol. 12, no. 4, pp. 685–692. doi: 10.1007/s11760-017-1208-2
  28. Mustapha A., Oulefki A., Bengherabi M., Boutellaa E., Algaet M.A. Towards nonuniform illumination face enhancement via adaptive contrast stretching. Multimedia Tools and Applications, 2017, vol. 76, no. 21, pp. 21961–21999. doi: 10.1007/s11042-017-4665-2
  29. Oloyede M., Hancke G., Myburgh H., Onumanyi A. A new evaluation function for face image enhancement in unconstrained environments using metaheuristic algorithms. EURASIP Journal on Image and Video Processing, 2019, no. 1, pp. 27. doi: 10.1186/s13640-019-0418-7
  30. Gao Y., Leung M.K.H. Face recognition using line edge map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, vol. 24, no. 6, pp. 764–779. doi: 10.1109/TPAMI.2002.1008383
  31. Fawwad Hussain M., Wang H., Santosh K.C. Gray level face recognition using spatial features. Communications in Computer and Information Science, 2019, vol. 1035, pp. 216–229. doi: 10.1007/978-981-13-9181-1_20
  32. Cootes T.F., Taylor C.J., Cooper D.H., Graham J. Active shape models-their training and application. Computer Vision and Image Understanding, 1995, vol. 61, no. 1, pp. 38–59. doi: 10.1006/cviu.1995.1004
  33. Cootes T.F., Edwards G.J., Taylor C.J. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, vol. 23, no. 6, pp. 681–685. doi: 10.1109/34.927467
  34. Iqtait M., Mohamad F.S., Mamat M. Feature extraction for face recognition via active shape model (ASM) and active appearance model (AAM). IOP Conference Series: Materials Science and Engineering, 2018, vol. 332, no. 1, pp. 012032. doi:10.1088/1757-899X/332/1/012032
  35. Bobe A.S., Konyshev D.V., Vorotnikov S.A. Emotion recognition system based on the facial motor units’ analysis. Engineering Journal: Science and Innovation, 2016, no. 9, pp. 7. doi: 10.18698/2308-6033-2016-9-1530. (in Russian)
  36. Candès E., Demanet L., Donoho D., Ying L. Fast discrete curvelet transforms. Multiscale Modeling & Simulation, 2006,vol. 5, no. 3, pp. 861–899. doi: 10.1137/05064182X
  37. Fu X., Fu K., Zhang Y., Zhou Q., Fu X. Facial expression recognition based on Curvelet transform and sparse representation. Proc. 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNN-FSKD 2018),Huangshan, China, 2018, pp. 257–263. doi: 10.1109/FSKD.2018.8686989
  38. Ahsan T., Jabid T., Chong U.P. Facial expression recognition using local transitional pattern on Gabor filtered facial images. IETE Technical Review, 2013, vol. 30, no. 1, pp. 47–52. doi: 10.4103/0256-4602.107339
  39. Shan C., Gong S., McOwan P.W. Facial expression recognition based on Local Binary Patterns: A comprehensive study. Image and Vision Computing, 2009, vol. 27, no. 6, pp. 803–816. doi: 10.1016/j.imavis.2008.08.005
  40. Fan J., Tie Y., Qi L. Facial expression recognition based on multiple feature fusion in video. Proc. International Conference on Computing and Pattern Recognition (ICCPR 2018), Shenzhen, China, 2018, pp. 86–92. doi: 10.1145/3232829.3232839
  41. Li S., Gong D., Yuan Y. Face recognition using Weber local descriptors. Neurocomputing, 2013, vol. 122, pp. 272–283. doi: 10.1016/j.neucom.2013.05.038
  42. Revina I.M., Emmanuel W.R.S. Face expression recognition using weber local descriptor and F-RBFNN. Proc. 2nd International Conference on Intelligent Computing and Control Systems (ICICCS 2018),Madurai, India, 2018, pp. 196–199. doi: 10.1109/ICCONS.2018.8662891
  43. Addison P.S. The illustrated wavelet transform handbook: introductory theory and applications in science, engineering, medicine and finance. CRC Press, 2017, 464 p.doi: 10.1201/9781315372556
  44. Nigam S., Singh R., Misra A.K. Efficient facial expression recognition using histogram of oriented gradients in wavelet domain. Multimedia Tools and Applications, 2018, vol. 77, no. 21, pp. 28725–28747. doi: 10.1007/s11042-018-6040-3
  45. Martínez A.M., Kak A.C. PCA versus LDA. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, vol. 23, no. 2, pp. 228–233. doi: 10.1109/34.908974
  46. Negahdaripour S. Revised definition of optical flow: Integration of radiometric and geometric cues for dynamic scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, vol. 20, no. 9, pp. 961–979. doi: 10.1109/34.713362
  47. Zhao J., Mao X., Zhang J. Learning deep facial expression features from image and optical flow sequences using 3D CNN. Visual Computer, 2018, vol. 34, no. 10, pp. 1461–1475. doi: 10.1007/s00371-018-1477-y
  48. Guo B., Lam K.-M., Siu W.-C., Yang S. Human face recognition using a spatially weighted Hausdorff distance. Proc. IEEE International Symposium on Circuits and Systems (ISCAS 2001), Hong Kong, China, 2001, vol. 2, pp. 145–148. doi: 10.1109/ISCAS.2001.921027
  49. Meftah I.T., Le Thanh N., Amar C.B. Emotion recognition using KNN classification for user modeling and sharing of affect states. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012, vol. 7663, pp. 234–242. doi: 10.1007/978-3-642-34475-6_29
  50. Greche L., Akil M., Kachouri R., Es-Sbai N. A new pipeline for the recognition of universal expressions of multiple faces in a video sequence. Journal of Real-Time Image Processing, 2019, in press. doi: 10.1007/s11554-019-00896-5
  51. Abdulrahman M., Eleyan A. Facial expression recognition using support vector machines. Proc. 23nd Signal Processing and Communications Applications Conference (SIU 2015), Malatya, Turkey, 2015, pp. 276–279. doi: 10.1109/SIU.2015.7129813
  52. Aleksic P.S., Katsaggelos A.K. Automatic facial expression recognition using facial animation parameters and multistream HMMs. IEEE Transactions on Information Forensics and Security, 2006, vol. 1, no. 1, pp. 3–11. doi: 10.1109/TIFS.2005.863510
  53. Safavian S.R., Landgrebe D. A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man and Cybernetics, 1991, vol. 21, no. 3, pp. 660–674. doi: 10.1109/21.97458
  54. Burkert P., Trier F., Afzal M.Z., Dengel A., Liwicki M. Dexpression: Deep convolutional neural network for expression recognition. Available at:  https://arxiv.org/abs/1509.05371.pdf(accessed: 13.12.2019).
  55. Svozil D., Kvasnička V., Pospichal J. Introduction to multi-layer feed-forward neural networks. Chemometrics and Intelligent Laboratory Systems, 1997, vol. 39, no. 1, pp. 43–62. doi: 10.1016/S0169-7439(97)00061-0
  56. An F., Liu Z. Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM. Visual Computer, 2020, vol. 36, no. 3, pp. 483–498. doi:10.1007/s00371-019-01635-4
  57. Akhremchik Y.V., Gorozhankin Y.P., Ivanov A.O., Mironov A.A., Tselikova S.O. Recognition and extraction of 3D models from two-dimensional images. Young Scientist, 2019, no. 26, pp. 25–28. Available at: https://moluch.ru/archive/264/61169/ (accessed: 02.01.2020).(in Russian)
  58. Cao T., Li M. Facial Expression Recognition Algorithm Based on the Combination of CNN and K-Means. Proc.11th International Conference on Machine Learning and Computing (ICMLC 2019), 2019, pp. 400–404. doi: 10.1145/3318299.3318344
  59. Karadeniz A.S., Karadeniz M.F., Weber G.W., Husein I. Improving CNN features for facial expression recognition. Zero: Jurnal Sains, Matematika dan Terapan, 2019, vol. 3, no. 1, pp. 1–11. doi: 10.30829/zero.v3i1.5881
  60. Kasraoui S., Lachiri Z., Madani K. Tandem modelling based emotion recognition in videos. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2019, vol. 11507, pp. 325–336. doi: 10.1007/978-3-030-20518-8_28


Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Copyright 2001-2024 ©
Scientific and Technical Journal
of Information Technologies, Mechanics and Optics.
All rights reserved.

Яндекс.Метрика