Analysis of pre-training and initialization methods of neural networks

Bobyl, B. V., Tereschenko, V. M. (2018). Analysis of pre-training and initialization methods of neural networks. Bulletin of Taras Shevchenko National University of Kyiv. Physics and Mathematics(4), 84–91. https://doi.org/10.17721/1812-5409.2018/4.12

In this paper we investigate main pre-training and initialization methods of parameter values of neural networks such as pre-training using restricted Boltzmann machines, deep autoencoders, Glorot and He initialization of parameters, transfer learning and domain adaptation. Given methods are useful for finding of appropriate parameter values and initial initialization of neural network, what is necessary condition for further efficient training of deep models, because it give a possibility during training to reduce negative effects such as vanishing or explosion of gradient, overfitting, stucking in one of local minimums of loss function, etc. These methods belong to group of unsupervised training algorithms and do not need any labeling for data which will be used later for model’s training after parameters initialization. Firstly, in this paper, we analyze all these methods and describe advantages and disadvantages of each of them. Secondly, we describe results of our experiments applying these methods for solving of classification task of MNIST dataset and introduce ideas for further development and improvement of these algorithms.Key words: neural network, pre-training, restricted Boltzmann machines, contrastive divergence algorithm, autoencoder, Glorot initialization, He initialization, domain adaptation, Jaccard index.Pages of the article in the issue: 84 - 91Language of the article: Ukrainian