## Menu

## Publications

## Editor-in-Chief

**Nikiforov**

Vladimir O.

D.Sc., Prof.

Vladimir O.

D.Sc., Prof.

## Partners

## IMPROVEMENT OF RECOGNITION QUALITY IN DEEP LEARNING NETWORKS BY SIMULATED ANNEALING METHOD

**Read the full article**

**Article in**Russian

**Abstract**

The subject of this research is deep learning methods, in which automatic construction of feature transforms is taken place in tasks of pattern recognition. Multilayer autoencoders have been taken as the considered type of deep learning networks. Autoencoders perform nonlinear feature transform with logistic regression as an upper classification layer. In order to verify the hypothesis of possibility to improve recognition rate by global optimization of parameters for deep learning networks, which are traditionally trained layer-by-layer by gradient descent, a new method has been designed and implemented. The method applies simulated annealing for tuning connection weights of autoencoders while regression layer is simultaneously trained by stochastic gradient descent. Experiments held by means of standard MNIST handwritten digit database have shown the decrease of recognition error rate from 1.1 to 1.5 times in case of the modified method comparing to the traditional method, which is based on local optimization. Thus, overfitting effect doesn’t appear and the possibility to improve learning rate is confirmed in deep learning networks by global optimization methods (in terms of increasing recognition probability). Research results can be applied for improving the probability of pattern recognition in the fields, which require automatic construction of nonlinear feature transforms, in particular, in the image recognition. Keywords: pattern recognition, deep learning, autoencoder, logistic regression, simulated annealing.

**Keywords:**cognition, deep learning, autoencoder, logistic regression, simulated annealing

**Acknowledgements.**The work is supported by the Ministry of Education and Science of the Russian Federation and the Russian Federation President’s Council for Grants (grant MD-1072.2013.9), and partially financially supported by the Government of the Russian Federation ( grant 074-U01).

**References**

*Unsupervised Feature Learning by Deep Sparse Coding*. 2013.Availableat: http://arxiv.org/pdf/1312.5783v1 (accessed03.07.2014).

*Proc. 19*

^{th}European Symposium on Artificial Neural Networks,*Computational Intelligence and Machine Learning,*

*ESANN 2011*. Bruges, Belgium, 2011, pp. 477–488.

*Neural Networks*, 2012, vol. 32, pp. 333–338. doi: 10.1016/j.neunet.2012.02.023

*Playing Atari with Deep Reinforcement Learning*. 2013. Availableat: http://arxiv.org/pdf/1312.5602v1.pdf(accessed03.07.2014).

*Neural Computation*, 2008, vol. 20, no. 6, pp. 1631–1649. doi: 10.1162/neco.2008.04-07-510

*Deep Autoregressive Networks*. 2013. Availableat:http://arxiv.org/pdf/1310.8499v2(accessed03.07.2014).

*Science*, 2011, vol. 331, no. 6022, pp. 1279–1285. doi: 10.1126/science.1192788

*Intriguing properties of neural networks*. 2014. Available at:http://arxiv.org/pdf/1312.6199v4(accessed 03.07.2014).

*Advances in Neural Information Processing Systems*, 2007, vol. 19, pp. 153–160.

*Neural Computation*, 2006, vol. 18, no. 7, pp. 1527–1554. doi: 10.1162/neco.2006.18.7.1527

*Advances in Neural Information Processing Systems*, 2007, vol. 19, pp. 1137–1144.

*Deep Big Simple Neural Nets Excel on Hand-written Digit Recognition*. 2010. Availableat:http://arxiv.org/pdf/1003.0358 (accessed03.07.2014).

*Scientific and Technical Journal of Information Technologies, Mechanics and Optics*, 2008, no. 8 (53), pp. 42–60.

*Scientific and Technical Journal of Information Technologies, Mechanics and Optics*, 2012, no. 2 (78), pp. 51–55.

*Proc. 25*. Helsinki, Finland, 2008, pp. 1096–1103.

^{th}International Conference on Machine Learning*The MNIST Database of handwritten digits*. Availableat:http://yann.lecun.com/exdb/mnist (accessed03.07.2014).