<div>
	A multivariate binary decision tree classifier based on shallow neural network</div>

Avazjon R. Marakhimov, Jabbarbergen K. Kudaybergenov, Kabul K. Khudaybergenov, Ulugbek R. Ohundadaev

2022 , VOLUME 22, NUMBER 4 ( July-August )

ISSN 2226-1494 (print), ISSN 2500-0373 (online)

Publications

Editor-in-Chief

Nikiforov
Vladimir O.
D.Sc., Prof.

Partners

doi: 10.17586/2226-1494-2022-22-4-725-733

A multivariate binary decision tree classifier based on shallow neural network

A. R. Marakhimov, J. K. Kudaybergenov, K. K. Khudaybergenov, U. R. Ohundadaev

Read the full article

Article in English

For citation:

Marakhimov A.R., Kudaybergenov J.K., Khudaybergenov K.K., Ohundadaev U.R. A multivariate binary decision tree classifier based on shallow neural network. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2022, vol. 22, no. 4, pp. 725–733. doi: 10.17586/2226-1494-2022-22-4-725-733

Abstract

In this paper, a novel decision tree classifier based on shallow neural networks with piecewise and nonlinear transformation activation functions are presented. A shallow neural network is recursively employed into linear and non-linear multivariate binary decision tree methods which generates splitting nodes and classifier nodes. Firstly, a linear multivariate binary decision tree with a shallow neural network is proposed which employs a rectified linear unit function. Secondly, there is presented a new activation function with non-linear property which has good generalization ability in learning process of neural networks. The presented method shows high generalization ability for linear and non-linear multivariate binary decision tree models which are called a Neural Network Decision Tree (NNDT). The proposed models with high generalization ability ensure the classification accuracy and performance. A novel split criterion of generating the nodes which focuses more on majority objects of classes on the current node is presented and employed in the new NNDT models. Furthermore, a shallow neural network based NNDT models are converted into a hyperplane based linear and non-linear multivariate decision trees which has high speed in the processing classification decisions. Numerical experiments on publicly available datasets have showed that the presented NNDT methods outperform the existing decision tree algorithms and other classifier methods.

Keywords: hierarchical classifier, neural networks, binary tree, multivariate decision tree, activation function

Acknowledgements. This research is supported by the Department of Algorithms and Programming technologies, National University of Uzbekistan, Uzbekistan

References

Morala P., Cifuentes J.A., Lillo R.E., Ucar I. Towards a mathematical framework to inform neural network modelling via polynomial regression. Neural Networks, 2021, vol. 142, pp. 57–72. https://doi.org/10.1016/j.neunet.2021.04.036
Cao W., Mirjalili V., Raschka S. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 2020, vol. 140, pp. 325–331. https://doi.org/10.1016/j.patrec.2020.11.008
Messner E., Fediuk M., Swatek P., Scheidl S., Smolle-Jüttner F.M., Olschewski H., Pernkopf F. Multi-channel lung sound classification with convolutional recurrent neural networks. Computers in Biology and Medicine, 2020, vol. 122, pp. 103831. https://doi.org/10.1016/j.compbiomed.2020.103831
Youling L. A calibration method of computer vision system based on dual attention mechanism. Image and Vision Computing, 2020, vol. 103, pp. 104039. https://doi.org/10.1016/j.imavis.2020.104039
Palmerston J.B., Zhou Y., Chan H.M. Comparing biological and artificial vision systems: Network measures of functional connectivity. Neuroscience Letters, 2020, vol. 739, pp. 135407. https://doi.org/10.1016/j.neulet.2020.135407
Basha S.H.Sh., Dubey Sh.R., Pulabaigari V., Mukherjee S. Impact of fully connected layers on performance of convolutional neural networks for image classification. Neurocomputing, 2020, vol. 378, pp. 112–119. https://doi.org/10.1016/j.neucom.2019.10.008
Shuang K., Tan Y., Cai Zh., Sun Y. Natural language modeling with syntactic structure dependency. Information Sciences, 2020, vol. 523, pp. 220–233. https://doi.org/10.1016/j.ins.2020.03.022
Xu M. WITHDRAWN: Image processing system based on FPGA and convolutional neural network. Microprocessors and Microsystems, 2020, pp. 103379. https://doi.org/10.1016/j.micpro.2020.103379
Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. Proc. of the 26^thAnnual Conference on Neural Information Processing Systems (NIPS), 2012, pp. 1097–1105.
Goodfellow I., Bengio Y., Courville A. Deep Learning. MIT Press, 2016, 775 p.
Zhou Z.H., Feng J. Deep forest: Towards an alternative to deep neural networks. Proc. of the 26^th International Joint Conference on Artificial Intelligence (IJCAI), 2017, pp. 3553–3559. https://doi.org/10.24963/ijcai.2017/497
Frosst N., Hinton G. Distilling a neural network into a soft decision tree. CEUR Workshop Proceedings, 2018, vol. 2070.
Wan A., Dunlap L., Ho D., Yin J., Lee S., Jin H., Petryk S., Bargal S.A., Gonzalez J.E. NBDT: Neural-backed decision trees. arXiv, 2020, arXiv:2004.00221. https://doi.org/10.48550/arXiv.2004.00221
Pinto A., Pereira S., Rasteiro D.M., Silva C. Hierarchical brain tumour segmentation using extremely randomized trees. Pattern Recognition, 2018, vol. 82, pp. 105–117. https://doi.org/10.1016/j.patcog.2018.05.006
Vanli N.D., Sayin M.O., Mohaghegh N.M., Ozkan H., Kozat S.S. Nonlinear regression via incremental decision trees. Pattern Recognition, 2019, vol. 86, pp. 1–13. https://doi.org/10.1016/j.patcog.2018.08.014
Blanco-Justici A., Domingo-Ferrer J., Martínez S., Sánchez D. Machine learning explainability via microaggregation and shallow decision trees. Knowledge-Based Systems, 2020, vol. 194, pp. 105532. https://doi.org/10.1016/j.knosys.2020.105532
Kotsiantis S.B. Decision trees: a recent overview. Artificial Intelligence Review, 2013, vol. 39, no. 4, pp. 261–283. https://doi.org/10.1007/s10462-011-9272-4
Quinlan J.R. Induction of decision trees. Machine Learning, 1986, vol. 1, no. 1, pp. 81–106. https://doi.org/10.1023/A:1022643204877
Breiman L., Friedman J.H., Stone C.J., Olshen R.A. Classification and Regression Trees. Chapman & Hall/CRC, 1984.
Quinlan J.R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., 1993.
Chandra B., Paul Varghese P. Moving towards efficient decision tree construction. Information Sciences, 2009, vol. 179, no. 8, pp. 1059–1069. https://doi.org/10.1016/j.ins.2008.12.006
Wang F., Wang Q., Nie F., Yu W., Wang R. Efficient tree classifiers for large scale datasets. Neurocomputing, 2018, vol. 284, pp. 70–79. https://doi.org/10.1016/j.neucom.2017.12.061
Yildiz C., Alpaydin E. Omnivariate decision trees. IEEE Transactions on Neural Networks, 2001, vol. 12, no. 6, pp. 1539–1546. https://doi.org/10.1109/72.963795
Altinçay H. Decision trees using model ensemble-based nodes. Pattern Recognition, 2007, vol. 40, no. 12, pp. 3540–3551. https ://doi.org/10.1016/j.patcog.2007.03.023
Kumar M.A., Gopal M. A hybrid SVM based decision tree. Pattern Recognition, 2010, vol. 43, no. 12, pp. 3977–3987. https://doi.org/10.1016/j.patcog.2010.06.010
Nie F., Zhu W., Li X. Decision Tree SVM: An extension of linear SVM for non-linear classification. Neurocomputing, 2020, vol. 401, pp. 153–159. https://doi.org/10.1016/j.neucom.2019.10.051

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License