Call For Papers
Contact Us

  Initialization of Weights in Deep Belief Neural Network Based on Standard Deviation of Feature Values in Training Data Vectors  
  Authors : Nader Rezazadeh
  Cite as:


Nowadays, the feature engineering approach has become very popular in deep neural networks. The purpose of this approach is to extract higher-level and more efficient features compared to those of learning data and to improve the learning of machines. One of the common ways in feature engineering is the use of deep belief networks. In addition, one of the problems in deep neural networks' training is the training process. The problems of the training process will be further enhanced in the event of an increase in the dimensions of the features and the complexity of the relationship between the initial features and the higher-level features. In the present paper, we attempt to set the initial weights based on the standard deviation of the feature vector values. Hence, a part of the training process is initially conducted and a better starting point can be provided for the weight training process. However, the impact of this method, to a large extent, depends on the relationship between the training data itself and the degree of independence of the training data's feature values. Experiments conducted in this field have achieved acceptable results.


Published In : IJCSN Journal Volume 6, Issue 6

Date of Publication : December 2017

Pages : 708-715

Figures :05

Tables : --


Nader Rezazadeh : received MSc in Artificial Intelligence, Department of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University. He is currently pursuing the Ph.D. degree in Artificial Intelligence and Robotics Engineering, Science and Research Branch of, Islamic Azad University, Tehran, Iran. His Research Interests is Non Stationary Environment Modeling, Hidden Markov Model, Markov Random Field and Deep Belief Neural Networks.


Neural Network; Restricted Boltzman Machine; Deep Belief Network

One of the major challenges of the deep belief network is the training of network weights for performing feature engineering on the input data sets. This process faces more serious problems with the increasing number of data features. In this paper, a part of the feature engineering of the data is carried out at the initialization stage. The purpose of the proposed method is to provide a better starting point for the weight training process. In this method, the initial weights are based on the standard deviation of the values of each features of the training data.. The proposed method uses the CD algorithm for weight training. The results of the experiments show that the proposed method works well on the data set, whose values are low in standard deviations. Unquestionably, the process of feature engineering is based on the relationship between all features of vector. But this method can be useful for data that has more evident individual features such as the standard deviation of a feature's values of the nodes.


[1] Y. Liu, S. Zhou, Q. Chen, “Discriminative deep belief networks for visual data classification”, Pattern Recognition, vol.44, Issue.10, pp. 2287–2296, 2011. [2] N. Rezazadeh, “A modification of the initial weights in the restricted Boltzman machine to reduce training error of the deep belief neural network”,International Journal of Computer Science and Information Security, vol.15, Issue.7, pp.1-6, 2017. [3] R. Salakhutdinov, G. Hinton, “Deep Boltzmann Machines”, International Conference on Artificial Intelligence and Statistics (AISTATS 2009), Canada, pp.448-455, 2009. [4] H. Lee, C. Ekanadham, and A. Ng, “Sparse deep belief net model for visual area V2,” Advances in neural information processing systems, vol. 20, pp. 873–880, 2008 [5] H. Lee, C. Ekanadham, and A. Ng, “Sparse deep belief net model for visual area V2” Advances in neural information processing systems, vol.20, pp.873–880, 2008. [6] G. E. Hinton, R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks”, Science, vol.313, Issue.578, pp.504–507, 2006. [7] V. Nair and G. Hinton, “3D object recognition with deep belief nets”, Advances in Neural Information Processing Systems, vol.22, pp.1339–1347, 2009. [8] R. Salakhutdinov, G. E. Hinton, “Deep boltzmann machines,” in Proceedings of the international conference on artificial intelligence and statistics, vol.5, pp.448–455, 2009. [9] R. Salakhutdinov, A. Mnih, G. Hinton , “Restricted Boltzman Machine for Collaborative Filtering”, Proceedings of the 24th international conference on Machine learning(2007), pp.791-798, 2007. [10] N. Le Roux, Y. Bengio , “Representation Power of Restricted Boltzman Machines and Deep Beliefe Networks”, Vol.20, Issue.6, pp.1631-1649, 2008. [11] Y. Bengio, “Learning Deep Architectures for AI”,Foundations and Trends in Machine Learning” Vol. 2, Issue.1, pp.1–127, 2009 [12] A. Fischer ,C. Igel, “An Introduction to Restricted Boltzmann Machines”, Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applicationspp(CIARP 2012), pp.14-36, 2012. [13] H. Larochelle, Y. Bengio, “Classification using discriminative restricted Boltzmann machines,” in Proceedings of the 25th international conference on Machine learning, New York, USA, pp. 536–543, 2008. [14] G. Hinton, “A practical guide to training restricted boltzmann machines”, Machine Learning Group, University of Toronto, Technical report, 2010. [15] X. Wang, Vincent Ly, Ruiguo, Chandra Kambhamettu, “2D-3D face recognition via Restricted Boltzmann Machines”, International Conference on Computer Vision Theory and Applications (VISAPP),Lisbon, 2015. [16] S. Iyanaga, Y. Kawada, “Distribution of Typical Random Variables” , Encyclopedic Dictionary of Mathematics. Cambridge MA(MIT Press), pp.1483- 1486, 1980. [17] M. Abramowitz, Stegun, I. A. (Eds.). “Probability Functions” Ch. 26 in Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, Vol.9, pp.925-964, 1972. [18] [17]-Car Evaluation Dataset http://archive.ics.uci.edu/ml/datasets/Car+Evaluation [Online Available]. [19] Heart Disease Dataset http://archive.ics.uci.edu/ml/datasets/Heart+Disease [Online Available]. [20] G. Hinton, S. Osindero, and Y. W. Teh. “A fast learning algorithm for deep belief nets. Neural Computation”, Vol.18, Issue.7,1527–1554, 2006.