doi: 10.17586/2226-1494-2022-22-1-33-46


Detection of yawning in driver behavior based a convolutional neural network

I. B. Lashkov


Read the full article  ';
Article in Russian

For citation:
Lashkov I.B. Detection of yawning in driver behavior based a convolutional neural network. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2022, vol. 22, no. 1, pp. 33–46 (in Russian). doi: 10.17586/2226-1494-2022-22-1-33-46


Abstract
Among the factors that usually cause road accidents in the world is driver fatigue, which accumulates during the trip or is present even before it begins. One of the most common signs of fatigue or tiredness of a vehicle driver is yawning. The detection of signs of yawning in human behavior is potentially able to further characterize its state of fatigue. Computer image processing methods are actively used to detect the openness of the mouth and yawning for a person. However, this approach has many disadvantages, which include different environmental conditions and a variety of situational yawning options for different people. The paper presents a scheme of a detector for determining signs of yawning, which is focused on processing images of the driver’s face using data analysis methods, computer image processing, and a convolutional neural network model. The essence of the proposed method is to detect yawning in the driver’s behavior in the cabin of a vehicle based on the analysis of a sequence of images obtained from a video camera. It is shown that the driver’s yawning state is accompanied by a wide and prolonged openness of the mouth. Prolonged openness of the mouth signals the appearance of signs of yawning. A conceptual model for detecting the openness of the mouth for a vehicle driver is presented and a scheme for processing and labeling the YawDD and Kaggle Drowsiness Dataset datasets is developed. The developed convolutional neural network model showed an accuracy of 0.992 and recall of 0.871 on a test 10 % data set. The proposed scheme for detecting the yawning state has been validated on a test video subset extracted from the YawDD: Yawning Detection Dataset. This detection scheme successfully detected 124 yawns among all video files from the test dataset. The proportion of correctly classified objects is 98.2 % accuracy, precision is equal to 96.1 %, recall is 98.4 %, and F score is 97.3 % while detecting signs of yawning in driver behavior. Detecting signs of yawning in the driver’s behavior allows one to clarify information about the driver and thereby to increase the effectiveness of existing driver monitoring systems in the vehicle cabin, aimed at preventing and reducing the risk of road accidents. The proposed approach can be combined with other technologies for monitoring driver behavior when building an intelligent driver support system.

Keywords: vehicle, driver, yawning, camera, monitoring, information processing

Acknowledgements. The research was carried out with the financial support of the RFBR within the scientific project No. 19-29-06099 (development of methods for finding vulnerabilities in human interaction interfaces with artificial intelligence of the “smart city” transport environment). The conceptual model of driver behavior monitoring was supported by the budget theme of the SPC RAS No. FFZF-2022-0005.

References
  1. Rau P.S. Drowsy driver detection and warning system for commercial vehicle drivers: Field operational test design, data analyses, and progress. Proc. Nat. Highway Traffic Saf. Admin., 2005, pp. 05–0192.
  2. Dua M., Singla R., Singla R., Raj S., Jangra A. Deep CNN models-based ensemble approach to driver drowsiness detection. Neural Computing and Applications, 2021, vol. 33, no. 8, pp. 3155–3168. https://doi.org/10.1007/s00521-020-05209-7
  3. Bakheet S., Al-Hamadi A. A framework for instantaneous driver drowsiness detection based on improved HOG features and naïve bayesian classification. Brain Sciences, 2021, vol. 11, no. 2, pp. 240. https://doi.org/10.3390/brainsci11020240
  4. Li L., Chen Y., Li Z. Yawning detection for monitoring driver fatigue based on two cameras. Proc. 12th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2009, pp. 12–17. https://doi.org/10.1109/ITSC.2009.5309841
  5. Daquin G., Micallef J., Blin O. Yawning. Sleep Medicine Reviews, 2001, vol. 5, no. 4, pp. 299–312. https://doi.org/10.1053/smrv.2001.0175
  6. Jackson P., Hilditch C., Holmes A., Reed N., Merat N., Smith L. Fatigue and Road Safety: A Critical Analysis of Recent Evidence. Department for Transport, London, 2011, 88 p.
  7. Resendes R., Martin K.H. Saving Lives Through Advanced Safety Technology. Faderal Highway Administration, US, Washington, DC, 2003.
  8. Wang L., Sun P., Xie M., Ma S., Li B., Shi Y., Su Q. Advanced driver-assistance system (ADAS) for intelligent transportation based on the recognition of traffic cones. Advances in Civil Engineering, 2020, pp. 8883639. https://doi.org/10.1155/2020/8883639
  9. Zhang W., Murphey Y.L., Wang T., Xu Q. Driver yawning detection based on deep convolutional neural learning and robust nose tracking. Proc. of the International Joint Conference on Neural Networks (IJCNN), 2015, pp. 1–8. https://doi.org/10.1109/IJCNN.2015.7280566
  10. Chevalier Y., Fenzl F., Kolomeets M., Rieke R., Chechulin A., Krauss C. Cyberattack detection in vehicles using characteristic functions, artificial neural networks, and visual analysis. Informatics and Automation, 2021, vol. 20, no. 4, pp. 845–868. https://doi.org/10.15622/ia.20.4.4
  11. Hasan F., Kashevnik A. State-of-the-Art Analysis of modern drowsiness detection algorithms based on computer vision. Proc. of the 29th Conference of Open Innovations Association (FRUCT), 2021, pp. 141–149. https://doi.org/10.23919/FRUCT52173.2021.9435480
  12. Kashevnik A., Lashkov I., Parfenov V., Mustafin N., Baraniuc O. Context-based driver support system development: Methodology and case study. Proc. of the 21st Conference of Open Innovations Association (FRUCT), 2017, pp. 162–171. https://doi.org/10.23919/FRUCT.2017.8250179
  13. Kashevnik A., Lashkov I., Gurtov A. Methodology and mobile application for driver behavior analysis and accident prevention. IEEE Transactions on Intelligent Transportation Systems, 2020, vol. 21, no. 6, pp. 2427–2436. https://doi.org/10.1109/TITS.2019.2918328
  14. Ibrahim M.M., Soraghan J.J., Petropoulakis L., Di Caterina G. Yawn analysis with mouth occlusion detection. Biomedical Signal Processing and Control, 2015, vol. 18, pp. 360–369. https://doi.org/10.1016/j.bspc.2015.02.006
  15. Ji Y., Wang S., Lu Y., Wei J., Zhao Y. Eye and mouth state detection algorithm based on contour feature extraction. Journal of Electronic Imaging, 2018, vol. 27, no. 5, pp. 051205. https://doi.org/10.1117/1.JEI.27.5.051205
  16. Omidyeganeh M., Shirmohammadi S., Abtahi S., Khurshid A., Farhan M., Scharcanski J., Hariri B., Laroche D., Martel L. Yawning detection using embedded smart cameras. IEEE Transactions on Instrumentation and Measurement, 2016, vol. 65, no. 3, pp. 570–582. https://doi.org/10.1109/TIM.2015.2507378
  17. Khan S., Akram A., Usman N. Real time automatic attendance system for face recognition using face API and OpenCV. Wireless Personal Communications, 2020, vol. 113, no. 1, pp. 469–480. https://doi.org/10.1007/s11277-020-07224-2
  18. Jose J., Vimali J.S., Ajitha P., Gowri S., Sivasangari A., Jinila B. Drowsiness detection system for drivers using image processing technique. Proc. of the 5th International Conference on Trends in Electronics and Informatics (ICOEI), 2021, pp. 1527–1530. https://doi.org/10.1109/ICOEI51242.2021.9452864
  19. Al-Madani A.M., Gaikwad A.T., Mahale V., Ahmed Z.A., Shareef A.A.A. Real-time driver drowsiness detection based on eye movement and yawning using facial landmark. Proc. of the International Conference on Computer Communication and Informatics (ICCCI), 2021, pp. 1–4. https://doi.org/10.1109/ICCCI50826.2021.9457005
  20. Akrout B., Mahdi W. Yawning detection by the analysis of variational descriptor for monitoring driver drowsiness. Proc. of the 2nd International Image Processing, Applications and Systems Conference (IPAS), 2016, pp. 1–5. https://doi.org/10.1109/IPAS.2016.7880127
  21. Zharmagambetov A., Gabidolla M., Carreira-Perpinán M.A. Improved multiclass AdaBoost for image classification: The role of tree optimization. Proc. of the IEEE International Conference on Image Processing (ICIP), 2021, pp. 19–22. https://doi.org/10.1109/ICIP42928.2021.9506569
  22. Kovshov E.E., Zavistovskaya T.A. Development of automated system of an access control based on analysis of the nasolabial muscles dynamics of the human face. Modern problems of science and education, 2013, no. 2, pp. 185–185. (in Russian)
  23. Kass M., Witkin A., Terzopoulos D. Snakes: Active contour models. International Journal of Computer Vision, 1988, vol. 1, no. 4, pp. 321–331. https://doi.org/10.1007/BF00133570
  24. Yazdi Z.M., Soryani M. Driver drowsiness detection by identification of yawning and eye closure. Automotive Science and Engineering, 2019, vol. 9, no. 3, pp. 3033–3044. https://doi.org/10.22068/ijae.9.3.3033
  25. Kassem H.A., Chowdhury M.U., Abawajy J., Al-Sudani A.R. Yawn based driver fatigue level prediction. EPiC Series in Computing, 2020, vol. 69, pp. 372–382. https://doi.org/10.29007/67kk
  26. Abtahi S., Omidyeganeh M., Shirmohammadi S., Hariri B. YawDD: A yawning detection dataset. Proc. of the 5th ACM Multimedia Systems Conference, 2014, pp. 24–28. https://doi.org/10.1145/2557642.2563678
  27. Yang H., Liu L., Min W., Yang X., Xiong X. Driver yawning detection based on subtle facial action recognition. IEEE Transactions on Multimedia, 2021, vol. 23, pp. 572–583. https://doi.org/10.1109/TMM.2020.2985536
  28. Suwarno S., Kevin K. Analysis of face recognition algorithm: Dlib and openCV. Journal of Informatics and Telecommunication Engineering, 2020, vol. 4, no. 1, pp. 173–184. https://doi.org/10.31289/jite.v4i1.3865
  29. Zhang S., Zhu X., Lei Z., Shi H., Wang X., Li S.Z. FaceBoxes: A CPU real-time face detector with high accuracy. Proc. of the IEEE International Joint Conference on Biometrics (IJCB), 2017, pp. 1–9. https://doi.org/10.1109/BTAS.2017.8272675
  30. Zhang K., Zhang Z., Li Z., Qiao Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 2016, vol. 23, no. 10, pp. 1499–1503. https://doi.org/10.1109/LSP.2016.2603342
  31. Bazarevsky V., Kartynnik Y., Vakunov A., Raveendran K., Grundmann M. Blazeface: Sub-millisecond neural face detection on mobile gpus. arXiv.org, 2019, arXiv:1907.05047.
  32. Bulat A., Tzimiropoulos G. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3d facial landmarks). Proc. of the 16th IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1021–1030. https://doi.org/10.1109/ICCV.2017.116
  33. Cohen G., Afshar S., Tapson J., Van Schaik A. EMNIST: Extending MNIST to handwritten letters. Proc. of the International Joint Conference on Neural Networks (IJCNN), 2017, pp. 2921–2926. https://doi.org/10.1109/IJCNN.2017.7966217
  34. Goodfellow I.J., Erhan D., Carrier P.L., Courville A., Mirza M., Hamner B., Cukierski W., Tang Y., Thaler D., Lee D., Zhou Y., Ramaiah C., Feng F., Li R., Wang X., Athanasakis D., Shawe-Taylor J., Milakov M., Park J., Ionescu R., Popescu M., Grozea C., Bergstra J., Xie J., Romaszko L., Xu B., Chuang Z., Bengio Y. Challenges in representation learning: A report on three machine learning contests. Lecture Notes in Computer Science, 2013, vol. 8228, pp. 117–124. https://doi.org/10.1007/978-3-642-42051-1_16
  35. Ramos A.L., Erandio J.C., Enteria E.M., Del Carmen N., Enriquez L.J., Mangilaya D.H. Driver drowsiness detection based on eye movement and yawning using facial landmark analysis. International Journal of Simulation: Systems, Science & Technology, 2019, vol. 20, no. S2, pp. 37. https://doi.org/10.5013/IJSSST.a.20.S2.37
  36. Savaş B.K., Becerikli Y. Real time driver fatigue detection based on SVM algorithm. Proc. of the 6th International Conference on Control Engineering and Information Technology (CEIT), 2018, pp. 1–4. https://doi.org/10.1109/CEIT.2018.8751886
  37. Saradadevi M., Bajaj P. Driver fatigue detection using mouth and yawning analysis. International Journal of Computer Science and Network Security, 2008, vol. 8, no. 6, pp. 183–188.
  38. Castella F.R., Sliding window detection probabilities. IEEE Transactions on Aerospace and Electronic Systems, 1976, vol. AES-12, no. 6, pp. 815–819. https://doi.org/10.1109/TAES.1976.308363


Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License
Copyright 2001-2024 ©
Scientific and Technical Journal
of Information Technologies, Mechanics and Optics.
All rights reserved.

Яндекс.Метрика