Menu
Publications
2026
2025
2024
2023
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
Editor-in-Chief
Nikiforov
Vladimir O.
D.Sc., Prof.
Partners
doi: 10.17586/2226-1494-2026-26-5-993-999
Search for promising points of neural networks initialization
Read the full article
Article in Russian
For citation:
Abstract
For citation:
Tolstykh V.N., Yershov V.N. Search for promising points of neural networks initialization. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2026, vol. 26, no. 5, pp. 993–999 (in Russian). doi: 10.17586/2226-1494-2026-26-5-993-999
Abstract
The paper examines the problem of initialization, which consists in determining the starting point in the space of the neural network weights for a qualitative gradient descent. It is known that the learning speed and learning result of a neural network mostly depends on its initialization. The problem is that in the process of execution, the algorithm can get into the anomalous zone of the local minimum and get stuck there for a long time, sometimes at the very beginning of training. In such cases, it is recommended to start the descent with a new initialization. The study proposes to start training from a point at the bottom of the local minimum, from which, as research shows, the descent in a new direction begins. The proposed method is based on the assumption that most of these anomalies are critical points that have a direction to descent. The neural network is initialized every time the learning algorithm encounters an anomaly in which the value of the error function increases. At the same time, a new initialization is determined at the minimum point of the error function in the segment connecting the point before the anomaly and the following one. Successive initializations result in the descent occurring along an optimized trajectory without significant oscillations. The effectiveness of the method is demonstrated by the example of a perceptron with a single hidden layer and one anomaly. It is proved that the descent from the new initialization point is faster; this demonstrates the possibility of gradient correction without calculating the Hessian; a conclusion is made about the expediency of multiple initializations of the neural network when it enters the zone of the next anomaly. It is proposed to use the method of correcting the gradient descent algorithm to reach the bottom of the anomaly in cases where the error begins to grow. The developed method gives a high probability of hitting a region with a low value of the error function than in the case of a random drawing of the initialization point, as well as a benefit in comparison with a Restricted Boltzmann’s machine in terms of execution time.
Keywords: neural network training, gradient descent, neural network initialization, local minimum, error function.
Acknowledgements. The work was carried out with the financial support of the Ministry of Science and Higher Education of the Russian Federation, Agreement No. FSRF-2023-0003, “Fundamental principles for building interference-proof systems for space and satellite communications, relative navigation, technical vision and aerospace monitoring”.
Acknowledgements. The work was carried out with the financial support of the Ministry of Science and Higher Education of the Russian Federation, Agreement No. FSRF-2023-0003, “Fundamental principles for building interference-proof systems for space and satellite communications, relative navigation, technical vision and aerospace monitoring”.

