DISTRIBUTED CONVOLUTIONAL NEURAL NETWORK MODEL ON RESOURCE-CONSTRAINED CLUSTER 

Khaydarova R.R., Mouromtsev D.I., Lapaev M.V., Fishchenko V.D. Distributed convolutional neural network model on resource-constrained cluster. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2020, vol. 20, no. 5, pp. 739–746 (in Russian). doi: 10.17586/2226-1494-2020-20-5-739-746

Subject of Research. The paper presents the distributed deep learning particularly convolutional neural network problem for resource-constrained devices. General architecture of convolutional neural network and its specificity is considered, existing constraints that appear while the deployment process on such architectures as LeNet, AlexNet, VGG-16/VGG-19 are analyzed. Deployment of convolutional neural network for resource-constrained devices is still a challenging task, as there are no existing and widely-used solutions. Method. The method for distribution of feature maps into smaller pieces is proposed, where each part is a determined problem. General distribution model for overlapped tasks within the scheduler is presented. Main Results. Distributed convolutional neural network model for a resource-constrained cluster and a scheduler for overlapped tasks is developed while carrying out computations mostly on a convolutional layer since this layer is one of the most resource-intensive, containing a large number of hyperparameters. Practical Relevance. Development of distributed convolutional neural network based on proposed methods provides the deployment of the convolutional neural network on a cluster that consists of 24 RockPro64 single board computers performing tasks related to machine vision, natural language processing, and prediction and is applicable in edge computing.

1. Shafique K., Khawaja B.A., Sabir F., Qazi S., Mustaqim M. Internet of Things (IoT) for next-generation smart systems: A review of current challenges, future trends and prospects for emerging 5G-IoT scenarios. IEEE Access, 2020, vol. 8, pp. 23022–23040. doi: 10.1109/ACCESS.2020.2970118

2. Shi W., Cao J., Zhang Q., Li Y., Xu L. Edge computing: Vision and challenges. IEEE Internet of Things Journal, 2016, vol. 3, no. 5, pp. 637–646. doi: 10.1109/JIOT.2016.2579198

3. Tarasenko A.O., Yakimov Y.V., Soloviev V.N. Convolutional neural networks for image classification. CEUR Workshop Proceedings, 2019, vol. 2546, pp. 101–114.

4. Zangeneh E., Rahmati M., Mohsenzadeh Y. Low resolution face recognition using a two-branch deep convolutional neural network architecture. Expert Systems with Applications, 2020, vol. 139, pp. 112854. doi: 10.1016/j.eswa.2019.112854

5. Solovyev R., Kustov A., Telpukhov D., Rukhlov V., Kalinin A. Fixed-point convolutional neural network for real-time video processing in FPGA. Proc. of the 2019 IEEE

Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), 2019, pp. 1605–1611. doi: 10.1109/EIConRus.2019.8656778

6. Widiastuti N.I. Convolution neural network for text mining and natural language processing. IOP Conference Series: Materials Science and Engineering, 2019, vol. 662, no. 5, pp. 052010. doi: 10.1088/1757-899X/662/5/052010

7. Giménez M., Palanca J., Botti V. Semantic-based padding in convolutional neural networks for improving the performance in natural language processing. A case of study in sentiment analysis. Neurocomputing, 2020, vol. 378, pp. 315–323. doi: 10.1016/j.neucom.2019.08.096

8. Sim H.S., Kim H.I., Ahn J.J. Is deep learning for image recognition applicable to stock market prediction? Complexity, 2019, pp. 4324878. doi: 10.1155/2019/4324878

9. Borovykh A., Bohte S., Oosterlee C.W. Conditional time series forecasting with convolutional neural networks. arXiv, arXiv:1703.04691, 2018.

10. Mao J., Chen X., Nixon K.W., Krieger C., Chen Y. MoDNN: Local distributed mobile computing system for deep neural network. Proc. 20th Design, Automation and Test in Europe Conference and Exhibition (DATE 2017), 2017, pp. 1396–1401. doi: 10.23919/DATE.2017.7927211

11. Wang S., Tuor T., Salonidis T., Leung K.K., Makaya C., He T., Chan K. When edge meets learning: Adaptive control for resource-constrained distributed machine learning. Proc. of the IEEE Conference on Computer Communications (INFOCOM 2018), 2018, pp. 63–71. doi: 10.1109/INFOCOM.2018.8486403

12. Motamedi M., Fong D., Ghiasi S. Machine intelligence on resource-constrained IoT devices: The case of thread granularity optimization for CNN inference. ACM Transactions on Embedded Computing Systems, 2017, vol. 16, no. 5s, pp. 151. doi: 10.1145/3126555

13. Khan A., Sohail A., Zahoora U., Qureshi A.S. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 2020, vol. 53, no. 8, pp. 5455–5516. doi: 10.1007/s10462-020-09825-6

14. Krizhevsky A., Sutskever I., Hinton G.E. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, vol. 2, pp. 1097–1105.

15. Alippi C., Disabato S., Roveri M. Moving convolutional neural networks to embedded systems: the alexnet and VGG-16 case. Proc. 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), 2018, pp. 212–223. doi: 10.1109/IPSN.2018.00049

16. Kuchumov R.I. Implementation and analysis of work-stealing task scheduler. Stohasticheskaja optimizacija v informatike, 2016, vol. 12, no. 1, pp. 20–39. (in Russian)

17. Dang H., Liu F., Stehouwer J., Liu X., Jain A. On the detection of digital face manipulation. arXiv, arXiv:1910.01717, 2019.

18. Yang S., Luo P., Loy C.C., Tang X. WIDER FACE: A face detection benchmark. Proc. 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016, pp. 5525–5533. doi: 10.1109/CVPR.2016.596

19. Boyko N., Basystiuk O., Shakhovska N. Performance evaluation and comparison of software for face recognition, based on dlib and opencv library. Proc. 2nd IEEE International Conference on Data Stream Mining and Processing (DSMP), 2018, pp. 478–482. doi: 10.1109/DSMP.2018.8478556

20. Khaydarova R., Fishchenko V., Mouromtsev D., Shmatkov V., Lapaev M. ROCK-CNN: a distributed RockPro64-based convolutional neural network cluster for IoT. Verification and performance analysis. Proc. 26th Conference of Open Innovations Association (FRUCT), 2020, pp. 174–181. doi: 10.23919/FRUCT48808.2020.9087457

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License