Обзор методов глубокого обучения для обработки видеоданных в фотоплетизмографии

Рубин И.М., Волынский М.А. Обзор методов глубокого обучения для обработки видеоданных в фотоплетизмографии // Научно-технический вестник информационных технологий, механики и оптики. 2026. Т. 26, № 2. С. 236–249. doi: 10.17586/2226-1494-2026-26-2-236-249

Введение. Представлен обзор современных методов глубокого обучения для обработки данных удаленной фотоплетизмографии. Рассмотрены архитектуры сверточных нейронных сетей, трансформеров, рекуррентных и генеративных моделей для предобработки видеосигнала, а также извлечения физиологически значимых параметров в условиях с артефактами, вызванными движением, изменением освещения или низким качеством видео. Выполнен анализ перспектив внедрения алгоритмов глубокого обучения в реальных медицинских сценариях на основе предложенных критериев с учетом существующих проблем интеграции, востребованности решений и валидации результатов. Метод. Выполнен обзор существующих методов глубокого обучения, которые используют видеосигнал для оценки сигнала фотоплетизмографии с использованием новых критериев для оценки методов, включая многомерность выходного сигнала фотоплетизмографии, открытость исходного кода и наличие информации о временных затратах, что является важным для их практического применения в реальном времени в медицинских учреждениях. Основные результаты. Показано, что методы глубокого обучения значительно превосходят традиционные подходы в задачах оценки физиологических параметров, в процессах диагностики сердечно-сосудистых заболеваний, а также предобработке видеосигнала. Выявлено, что большинство существующих решений, основанных на глубоком обучении, ограничиваются одномерным выходным сигналом из-за сложности получения многомерной разметки для обучения с учителем. Дополнительный анализ показал дефицит информации о временных и вычислительных затратах, что ограничивает практическое применение методов глубокого обучения в реальном времени. Представленная систематизация раскрывает ключевые термины, связанные с обработкой сигналов фотоплетизмографии: контактная фотоплетизмография, фотоплетизмография на основе видео, удаленная фотоплетизмография, визуализация фотоплетизмографии. Представлено описание подходов к сбору наборов данных, учитывающих концепции многомерности, многоканальности и мультимодальности сигналов. Обсуждение. Полученные результаты могут быть применены при разработке систем удаленного мониторинга здоровья, включая медицинские и бытовые устройства. Обзор будет полезен специалистам в области биомедицинской инженерии, медицинской информатики, а также разработчикам решений для анализа физиологических сигналов.

 

1. Nie G., Zhu J., Tang G., Zhang D., Geng S., Zhao Q., et al. A Review of deep learning methods for photoplethysmography data // arXiv. 2024. arXiv:2401.12783. https://doi.org/10.48550/arXiv.2401.12783

2. Ray D., Collins T., Woolley S., Ponnapalli P. A review of wearable multi-wavelength photoplethysmography // IEEE Reviews in Biomedical Engineering. 2021. V. 16. P. 136–151. https://doi.org/10.1109/rbme.2021.3121476
3. Vavrinsky E., Esfahani N.E., Hausner M., Kuzma A., Rezo V., Donoval M., et al. The current state of optical sensors in medical wearables // Biosensors. 2022. V. 12. N 4. P. 217. https://doi.org/10.3390/bios12040217
4. Chang C.-C., Wu C.-T., Choi B.I., Fang T.-J. MW-PPG sensor: An on-chip spectrometer approach // Sensors. 2019. V. 19. N 17. P. 3698. https://doi.org/10.3390/s19173698
5. Sugita N., Noro T., Yoshizawa M., Ichiji K., Yamaki S., Homma N. Estimation of absolute blood pressure using video images captured at different heights from the heart // Proc. of the 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2019. P. 4458–4461. https://doi.org/10.1109/embc.2019.8856362
6. Kamshilin A.A., Margaryants N.B. Origin of photoplethysmographic waveform at green light // Physics Procedia. 2017. V. 86. P. 72–80. https://doi.org/10.1016/j.phpro.2017.01.024
7. Alharbi S., Hu S., Mulvaney D., Blanos P. An applicable approach for extracting human heart rate and oxygen saturation during physical movements using a multi-wavelength illumination optoelectronic sensor system // Proceedings of SPIE. 2018. V. 10486. P. 104860S. https://doi.org/10.1117/12.2287854
8. Bousefsaf F., Djeldjli D., Ouzar Y., Maaoui C., Pruski A. iPPG 2 cPPG: reconstructing contact from imaging photoplethysmographic signals using U-Net architectures // Computers in Biology and Medicine. 2021. V. 138. P. 104860. https://doi.org/10.1016/j.compbiomed.2021.104860
9. Goudarzi R.H., Mousavi S.S., Charmi M. Using imaging photoplethysmography (iPPG) signal for blood pressure estimation // Proc. of the International Conference on Machine Vision and Image Processing (MVIP). 2020. P. 1–6. https://doi.org/10.1109/mvip49855.2020.9116902
10. Ni A., Azarang A., Kehtarnavaz N. A review of deep learning-based contactless heart rate measurement methods // Sensors. 2021. V. 21. N 11. P. 3719. https://doi.org/10.3390/s21113719
11. Sokolov A.Y., Volynsky M.A., Potapenko A.V., Iurkova P.M., Zaytsev V.V., Nippolainen E., et al. Duality in response of intracranial vessels to nitroglycerin revealed in rats by imaging photoplethysmography // Scientific Reports. 2023. V. 13. N 1. P. 11928. https://doi.org/10.1038/s41598-023-39171-w
12. Volkov I.Y., Sagaidachnyi A.A., Fomin A.V. Photoplethysmographic imaging of hemodynamics and two-dimensional oximetry // Optics and Spectroscopy. 2022. V. 130. N 7. P. 452–469. https://doi.org/10.1134/s0030400x22080057
13. Wieringa F.P., Mastik F., van der Steen A.F.W. Contactless multiple wavelength photoplethysmographic imaging: A first step toward “SpO2 Camera” technology // Annals of Biomedical Engineering. 2005. V. 33. N 8. P. 1034–1041. https://doi.org/10.1007/s10439-005-5763-2
14. Kumar M., Suliburk J.W., Veeraraghavan A., Sabharwal A. PulseCam: a camera-based, motion-robust and highly sensitive blood perfusion imaging modality // Scientific Reports. 2020. V. 10. N 1. P. 4825. https://doi.org/10.1038/s41598-020-61576-0
15. Kamshilin A.A., Volynsky M.A., Khayrutdinova O., Nurkhametova D., Babayan L., Amelin A.V., et al. Novel capsaicin-induced parameters of microcirculation in migraine patients revealed by imaging photoplethysmography // The Journal of Headache and Pain. 2018. V. 19. P. 43. https://doi.org/10.1186/s10194-018-0872-0
16. Volynsky M.A., Mamontov O.V., Sidorov I.S., Kamshilin A.A. Pulse wave transit time measured by imaging photoplethysmography in upper extremities // Journal of Physics: Conference Series. 2016. V. 737. N 1. P. 012053. https://doi.org/10.1088/1742-6596/737/1/012053
17. Sun Y., Thakor N. Photoplethysmography revisited: from contact to noncontact, from point to imaging // IEEE Transactions on Biomedical Engineering. 2015. V. 63. N 3. P. 463–477. https://doi.org/10.1109/tbme.2015.2476337
18. Rubins U., Upmalis V., Rubenis O., Jakovels D., Spigulis J. Real-time photoplethysmography imaging system // IFMBE Proceedings. 2011. V. 34. P. 183–186. https://doi.org/10.1007/978-3-642-21683-1_46
19. Rubins U., Miscuks A., Qawqzeh Y., Marcinkevics Z., Grabovskis A. Photoplethysmography imaging algorithm for real-time monitoring of skin perfusion maps // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2023. P. 5950–5956. https://doi.org/10.1109/cvprw59228.2023.00633
20. Spigulis J. Multispectral, fluorescent and photoplethysmographic imaging for remote skin assessment // Sensors. 2017. V. 17. N 5. P. 1165. https://doi.org/10.3390/s17051165
21. Rubīns U., Spīgulis J., Miščuks A. Photoplethysmography imaging algorithm for continuous monitoring of regional anesthesia // Proc. of the 14th ACM/IEEE Symposium on Embedded Systems for Real-Time Multimedia. 2016. P. 67–71. https://doi.org/10.1145/2993452.2994308
22. Kyung J., Yang J.-Y., Choi J.-H., Chang J.-H., Bae S., Choi J., et al. Deep-learning-based blood pressure estimation using multi channel photoplethysmogram and finger pressure with attention mechanism // Scientific Reports. 2023. V. 13. N 1. P. 9311. https://doi.org/10.1038/s41598-023-36068-6
23. Mehrgardt P., Khushi M., Poon S., Withana A. Deep learning fused wearable pressure and PPG data for accurate heart rate monitoring // IEEE Sensors Journal. 2021. V. 21. N 23. P. 27106-27115. https://doi.org/10.1109/jsen.2021.3123243
24. Jang D.-G., Park S., Hahn M., Park S.-H. A real-time pulse peak detection algorithm for the photoplethysmogram // International Journal of Electronics and Electrical Engineering. 2014. V 2. N 1. P. 45–49. https://doi.org/10.12720/ijeee.2.1.45-49
25. Argüello-Prada E.J. The mountaineer’s method for peak detection in photoplethysmographic signals // Revista Facultad de Ingeniería Universidad de Antioquia. 2019. N 90. P. 42–50. https://doi.org/10.17533/udea.redin.n90a06
26. Vadrevu S., Manikandan M.S. A robust pulse onset and peak detection method for automated PPG signal analysis system // IEEE Transactions on Instrumentation and Measurement. 2019. V. 68. N 3. P. 807–817. https://doi.org/10.1109/tim.2018.2857878
27. Nowara E.M., Marks T.K., Mansour H., Veeraraghavan A. Near-infrared imaging photoplethysmography during driving // IEEE Transactions on Intelligent Transportation Systems. 2022. V. 23. N 4. P. 3589–3600. https://doi.org/10.1109/tits.2020.3038317
28. Nowara E.M., Marks T.K., Mansour H., Veeraraghavan A. SparsePPG: Towards driver monitoring using camera-based vital signs estimation in near-infrared // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2018. P. 1353–1362. https://doi.org/10.1109/cvprw.2018.00174
29. Kumar M., Veeraraghavan A., Sabharwal A. DistancePPG: Robust non-contact vital signs monitoring using a camera // Biomedical Optics Express. 2015. V. 6. N 5. P. 1565–1588. https://doi.org/10.1364/boe.6.001565
30. Poh M.-Z., McDuff D.J., Picard R.W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation // Optics Express. 2010. V. 18. N 10. P. 10762–10774. https://doi.org/10.1364/oe.18.010762
31. de Haan G., Jeanne V. Robust pulse rate from chrominance-based rPPG // IEEE Transactions on Biomedical Engineering. 2013. V. 60. N 10. P. 2878–2886. https://doi.org/10.1109/tbme.2013.2266196
32. Wang W., den Brinker A.C., Stuijk S., de Haan G. Algorithmic principles of remote PPG // IEEE Transactions on Biomedical Engineering. 2016. V. 64. N 7. P. 1479–1491. https://doi.org/10.1109/tbme.2016.2609282
33. van Gent P., Farah H., van Nes N., van Arem B. HeartPy: A novel heart rate algorithm for the analysis of noisy signals // Transportation Research Part F: Traffic Psychology and Behaviour. 2019. V. 66. P. 368–378. https://doi.org/10.1016/j.trf.2019.09.015
34. Goda M.Á., Charlton P.H., Behar J.A. pyPPG: A Python toolbox for comprehensive photoplethysmography signal analysis // Physiological Measurement. 2024. V.45. N 4. P. 045001. https://doi.org/10.1088/1361-6579/ad33a2
35. Boccignone G., Conte D., Cuculo V., D'Amelio A., Grossi G., Lanzarotti, R., et al. pyVHR: a Python framework for remote photoplethysmography // PeerJ Computer Science. 2022. V. 8. P. e929. https://doi.org/10.7717/peerj-cs.929
36. Kwon S., Hong J., Choi E.K., Lee E., Hostallero D.E., Kang W.J., et al. Deep learning approaches to detect atrial fibrillation using photoplethysmographic signals: algorithms development study // JMIR mHealth and uHealth. 2019. V. 7. N 6. P. e12770. https://doi.org/10.2196/12770
37. Chen Y., Zhang D., Karimi H.R., Deng C., Yin W. A new deep learning framework based on blood pressure range constraint for continuous cuffless BP estimation // Neural Networks. 2022. V. 152. P. 181–190. https://doi.org/10.1016/j.neunet.2022.04.017
38. Maqsood S., Xu S., Springer M., Mohawesh R. A benchmark study of machine learning for analysis of signal feature extraction techniques for blood pressure estimation using photoplethysmography (PPG) // IEEE Access. 2021. V. 9. P. 138817–138833. https://doi.org/10.1109/access.2021.3117969
39. Kim M., Lee H., Kim K.-Y., Kim K.-H. Deep learning model for blood pressure estimation from PPG signal // Proc. of the IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE). 2022. P. 1–5. https://doi.org/10.1109/metroxraine54828.2022.9967606
40. Boukhechba M., Cai L., Wu C., Barnes L.B. ActiPPG: Using deep neural networks for activity recognition from wrist-worn photoplethysmography (PPG) sensors // Smart Health. 2019. V. 14. P. 100082. https://doi.org/10.1016/j.smhl.2019.100082
41. Ordóñez F.J., Roggen D. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition // Sensors. 2016. V. 16. N 1. P. 115. https://doi.org/10.3390/s16010115
42. Wang D., Hu Q., Yang C. Biometric recognition based on scalable end-to-end convolutional neural network using photoplethysmography: A comparative study // Computers in Biology and Medicine. 2022. V. 147. P. 105654. https://doi.org/10.1016/j.compbiomed.2022.105654
43. Panwar M., Gautam A., Biswas D., Acharyya A. PP-Net: A deep learning framework for PPG-based blood pressure and heart rate estimation // IEEE Sensors Journal. 2020. V. 20. N 17. P. 10000-10011. https://doi.org/10.1109/jsen.2020.2990864
44. Burrello A., Pagliari D.J., Risso M., Benatti S., Macii E., Benini L., et al. Q-ppg: Energy-efficient ppg-based heart rate monitoring on wearable devices // IEEE Transactions on Biomedical Circuits and Systems. 2021. V. 15. N 6. P. 1196–1209. https://doi.org/10.1109/tbcas.2021.3122017
45. Chen X., Wang X., Zhang K., Fung K.-M., Thai T.C., Moore K., et al. Recent advances and clinical applications of deep learning in medical image analysis // Medical Image Analysis. 2022. V. 79. P. 102444. https://doi.org/10.1016/j.media.2022.102444
46. Wynants L., Van Calster B., Collins G.S., Riley R.D., Heinze G., Schuit E., et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal // British Medical Journal. 2020. V. 369. P. m1328. https://doi.org/10.1136/bmj.m1328
47. Roberts M, Driggs D., Thorpe M., Gilbey J., Yeung M., Ursprung S., et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans // Nature Machine Intelligence. 2021. V. 3. N 3. P. 199–217. https://doi.org/10.1038/s42256-021-00307-0
48. Kamshilin A.A., Zaytsev V.V., Lodygin A., Kashchenko V.A. Imaging photoplethysmography as an easy-to-use tool for monitoring changes in tissue blood perfusion during abdominal surgery // Scientific Reports. 2022. V. 12. N 1. P. 1143. https://doi.org/10.1038/s41598-022-05080-7
49. Almarshad M.A., Islam M.S., Al-Ahmadi S., BaHammam A.S. Diagnostic features and potential applications of PPG signal in healthcare: A systematic review // Healthcare. 2022. V. 10. N 3. P. 547. https://doi.org/10.3390/healthcare10030547
50. Chen W., Yi Z., Lim L.J.R., Lim R.Q.R., Zhang A., Qian Z., et al. Deep learning and remote photoplethysmography powered advancements in contactless physiological measurement // Frontiers in Bioengineering and Biotechnology. 2024. V. 12. P. 1420100. https://doi.org/10.3389/fbioe.2024.1420100
51. Cheng C.-H., Wong K.-L., Chin J.-W., Chan T.-T., So R.H.Y. Deep learning methods for remote heart rate measurement: a review and future research agenda // Sensors. 2021. V. 21. N 18. P. 6296. https://doi.org/10.3390/s21186296
52. Schrumpf F., Frenzel P., Aust C., Osterhoff G., Fuchs M. Assessment of non-invasive blood pressure prediction from ppg and rppg signals using deep learning // Sensors. 2021. V. 21. N 18. P. 6022. https://doi.org/10.3390/s21186022
53. Волков И.Ю., Сагайдачный А.А., Фомин А.В. Фотоплетизмографическая визуализация гемодинамики и двухмерная оксиметрия // Известия Саратовского университета. Новая серия. Серия: Физика. 2022. Т. 22. №. 1. С. 15–45. https://doi.org/10.18500/1817-3020-2022-22-1-15-45
54. Wang Y., Ren Y., Wang T., Li D., Cai H., Ji B. High‐accuracy heart rate detection using multispectral IPPG technology combined with a deep learning algorithm // Journal of Biophotonics. 2024. V. 17. N 9. P. e202400119. https://doi.org/10.1002/jbio.202400119
55. Comas A., Marks T.K., Mansour H., Lohit S., Ma Y., Liu X. Turnip: Time-series U-Net with recurrence for NIR imaging PPG // Proc. of the IEEE International Conference on Image Processing (ICIP). 2021. P. 309-313. https://doi.org/10.1109/icip42928.2021.9506663
56. Kuang H., Lv F., Ma X., Liu X. Efficient spatiotemporal attention network for remote heart rate variability analysis // Sensors. 2022. V. 22. N 3. P. 1010. https://doi.org/10.3390/s22031010
57. Luguev T., Seuß D., Garbas J.U. Deep learning based affective sensing with remote photoplethysmography // Proc. of the 54th Annual Conference on Information Sciences and Systems (CISS). 2020. P. 360–363. https://doi.org/10.1109/ciss48834.2020.1570617362
58. Yu Z., Li X., Zhao G. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks // arXiv. 2019. arXiv:1905.02419. https://doi.org/10.48550/arXiv.1905.02419
59. Ouzar Y., Djeldjli D., Bousefsaf F., Maaoui C. X-iPPGNet: A novel one stage deep learning architecture based on depthwise separable convolutions for video-based pulse rate estimation // Computers in Biology and Medicine. 2023. V. 154. P. 106592. https://doi.org/10.1016/j.compbiomed.2023.106592
60. Liu S., Yang Y., Jing X., Li B., Liu H., Zhu S. BiFormer: An End-to-End Deep Learning Approach for Enhanced Image-Based Photoplethysmography and Heart Rate Accuracy // Frontiers in Artificial Intelligence and Applications. 2024. P. 205–214. https://doi.org/10.3233/faia231195
61. Lee E., Chen E., Lee C.-Y. Meta-rppg: Remote heart rate estimation using a transductive meta-learner // Lecture Notes in Computer Science. 2020. V. 12372. P. 392–409. https://doi.org/10.1007/978-3-030-58583-9_24
62.  Yu Z., Peng W., Li X., Hong X., Zhao G. Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement // Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV). 2019. P. 151–160. https://doi.org/10.1109/iccv.2019.00024
63. Bousefsaf F., Pruski A., Maaoui C. 3D convolutional neural networks for remote pulse rate measurement and mapping from facial video // Applied Sciences. 2019. V. 9. N 20. P. 4364. https://doi.org/10.3390/app9204364
64. Liu S.-Q., Yuen P.C. A general remote photoplethysmography estimator with spatiotemporal convolutional network // Proc. of the 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). 2020. P. 481–488. https://doi.org/10.1109/fg47880.2020.00109
65. Yue Z., Ding S., Yang S., Yang H., Li Z., Zhang Y., et al. Deep super-resolution network for rPPG information recovery and noncontact heart rate estimation // IEEE Transactions on Instrumentation and Measurement. 2021. V. 70. P. 2513511. https://doi.org/10.1109/tim.2021.3109398
66. McDuff D. Deep super resolution for recovering physiological information from videos // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2018. P. 1448–1455. https://doi.org/10.1109/cvprw.2018.00185
67. Lu H., Han H., Zhou S.K. Dual-GAN: Joint bvp and noise modeling for remote physiological measurement // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021. P. 12399–12408. https://doi.org/10.1109/cvpr46437.2021.01222
68. Lokendra B., Puneet G. AND-rPPG: A novel denoising-rPPG network for improving remote heart rate estimation // Computers in Biology and Medicine. 2022. V. 141. P. 105146. https://doi.org/10.1016/j.compbiomed.2021.105146
69. Qiu Y., Liu Y., Arteaga-Falconi J., Dong H., El Saddik A. EVM-CNN: Real-time contactless heart rate estimation from facial video // IEEE Transactions on Multimedia. 2019. V. 21. N 7. P. 1778–1787. https://doi.org/10.1109/tmm.2018.2883866
70. Paracchini M., Marcon M., Villa F., Zappa F., Tubaro S. Biometric signals estimation using single photon camera and deep learning // Sensors. 2020. V. 20. N 21. P. 6102. https://doi.org/10.3390/s20216102
71. Zhan Q., Wang W., de Haan G. Analysis of CNN-based remote-PPG to understand limitations and sensitivities // Biomedical Optics Express. 2020. V. 11. N 3. P. 1268-1283. https://doi.org/10.1364/boe.382637
72. Chen W., McDuff D. DeepPhys: video-based physiological measurement using convolutional attention networks // Lecture Notes in Computer Science. 2018. V. 11206. P. 356–373. https://doi.org/10.1007/978-3-030-01216-8_22
73. Reiss A., Indlekofer I., Schmidt P., Van Laerhoven K. Deep PPG: Large-scale heart rate estimation with convolutional neural networks // Sensors. 2019. V. 19. N 14. P. 3079. https://doi.org/10.3390/s19143079
74. Perepelkina O., Artemyev M., Churikova M., Grinenko M. HeartTrack: Convolutional neural network for remote video-based heart rate monitoring // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2020. P. 1163–1171. https://doi.org/10.1109/cvprw50498.2020.00152
75. Cen Y., Luo J., Wang H., Chen L., Zhu X., Guo S., et al. OVAR-BPnet: a general pulse wave deep learning approach for cuffless blood pressure measurement // IEEE Journal of Biomedical and Health Informatics. 2024. V. 28. N 10. P. 5829–5841. https://doi.org/10.1109/jbhi.2024.3423461
76. Song R., Chen H., Cheng J., Li C., Liu Y., Chen X. PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography // IEEE Journal of Biomedical and Health Informatics. 2021. V. 25. N 5. P. 1373–1384. https://doi.org/10.1109/jbhi.2021.3051176
77. Estepp J.R., Blackford E.B., Meier C.M. Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography // Proc. of the IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2014. P. 1462–1469. https://doi.org/10.1109/smc.2014.6974121
78. Lie W.-N., Le D.-Q., Lai C.-Y., Fang Y.-S. Heart rate estimation from facial image sequences of a dual-modality RGB-NIR camera // Sensors. 2023. V. 23. N 13. P. 6079. https://doi.org/10.3390/s23136079

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License