Nowadays, the feature engineering approach has become very popular in deep neural networks. The purpose of
this approach is to extract higher-level and more efficient features compared to those of learning data and to improve the
learning of machines. One of the common ways in feature engineering is the use of deep belief networks. In addition, one
of the problems in deep neural networks' training is the training process. The problems of the training process will be
further enhanced in the event of an increase in the dimensions of the features and the complexity of the relationship
between the initial features and the higher-level features. In the present paper, we attempt to set the initial weights based on
the standard deviation of the feature vector values. Hence, a part of the training process is initially conducted and a better
starting point can be provided for the weight training process. However, the impact of this method, to a large extent,
depends on the relationship between the training data itself and the degree of independence of the training data's feature
values. Experiments conducted in this field have achieved acceptable results.
Published In:IJCSN Journal Volume 6, Issue 6
Date of Publication : December 2017
Pages : 708-715
Figures :05
Tables : --
Nader Rezazadeh : received MSc in
Artificial Intelligence, Department of
Computer and Information Technology
Engineering, Qazvin Branch, Islamic
Azad University. He is currently
pursuing the Ph.D. degree in Artificial Intelligence and Robotics Engineering, Science and Research
Branch of, Islamic Azad University, Tehran, Iran. His
Research Interests is Non Stationary Environment Modeling,
Hidden Markov Model, Markov Random Field and Deep
Belief Neural Networks.
Neural Network; Restricted Boltzman Machine; Deep Belief Network
One of the major challenges of the deep belief network is
the training of network weights for performing feature
engineering on the input data sets. This process faces more
serious problems with the increasing number of data
features. In this paper, a part of the feature engineering of
the data is carried out at the initialization stage. The
purpose of the proposed method is to provide a better
starting point for the weight training process. In this
method, the initial weights are based on the standard
deviation of the values of each features of the training
data.. The proposed method uses the CD algorithm for
weight training. The results of the experiments show that
the proposed method works well on the data set, whose
values are low in standard deviations. Unquestionably, the
process of feature engineering is based on the relationship
between all features of vector. But this method can be
useful for data that has more evident individual features
such as the standard deviation of a feature's values of the
nodes.
[1] Y. Liu, S. Zhou, Q. Chen, “Discriminative deep belief
networks for visual data classification”, Pattern
Recognition, vol.44, Issue.10, pp. 2287–2296, 2011.
[2] N. Rezazadeh, “A modification of the initial weights in
the restricted Boltzman machine to reduce training
error of the deep belief neural network”,International
Journal of Computer Science and Information Security,
vol.15, Issue.7, pp.1-6, 2017.
[3] R. Salakhutdinov, G. Hinton, “Deep Boltzmann
Machines”, International Conference on Artificial
Intelligence and Statistics (AISTATS 2009), Canada,
pp.448-455, 2009.
[4] H. Lee, C. Ekanadham, and A. Ng, “Sparse deep belief
net model for visual area V2,” Advances in neural
information processing systems, vol. 20, pp. 873–880,
2008
[5] H. Lee, C. Ekanadham, and A. Ng, “Sparse deep belief
net model for visual area V2” Advances in neural
information processing systems, vol.20, pp.873–880,
2008.
[6] G. E. Hinton, R. R. Salakhutdinov, “Reducing the
dimensionality of data with neural networks”, Science,
vol.313, Issue.578, pp.504–507, 2006.
[7] V. Nair and G. Hinton, “3D object recognition with
deep belief nets”, Advances in Neural Information
Processing Systems, vol.22, pp.1339–1347, 2009.
[8] R. Salakhutdinov, G. E. Hinton, “Deep boltzmann
machines,” in Proceedings of the international
conference on artificial intelligence and statistics, vol.5,
pp.448–455, 2009.
[9] R. Salakhutdinov, A. Mnih, G. Hinton , “Restricted
Boltzman Machine for Collaborative Filtering”,
Proceedings of the 24th international conference on
Machine learning(2007), pp.791-798, 2007.
[10] N. Le Roux, Y. Bengio , “Representation Power of
Restricted Boltzman Machines and Deep Beliefe
Networks”, Vol.20, Issue.6, pp.1631-1649, 2008.
[11] Y. Bengio, “Learning Deep Architectures for
AI”,Foundations and Trends in Machine Learning”
Vol. 2, Issue.1, pp.1–127, 2009
[12] A. Fischer ,C. Igel, “An Introduction to Restricted
Boltzmann Machines”, Progress in Pattern Recognition,
Image Analysis, Computer Vision, and
Applicationspp(CIARP 2012), pp.14-36, 2012.
[13] H. Larochelle, Y. Bengio, “Classification using
discriminative restricted Boltzmann machines,” in
Proceedings of the 25th international conference on
Machine learning, New York, USA, pp. 536–543, 2008.
[14] G. Hinton, “A practical guide to training restricted
boltzmann machines”, Machine Learning Group,
University of Toronto, Technical report, 2010.
[15] X. Wang, Vincent Ly, Ruiguo, Chandra Kambhamettu,
“2D-3D face recognition via Restricted Boltzmann
Machines”, International Conference on Computer
Vision Theory and Applications (VISAPP),Lisbon,
2015.
[16] S. Iyanaga, Y. Kawada, “Distribution of Typical
Random Variables” , Encyclopedic Dictionary of
Mathematics. Cambridge MA(MIT Press), pp.1483-
1486, 1980.
[17] M. Abramowitz, Stegun, I. A. (Eds.). “Probability
Functions” Ch. 26 in Handbook of Mathematical
Functions with Formulas, Graphs, and Mathematical
Tables, Vol.9, pp.925-964, 1972.
[18] [17]-Car Evaluation Dataset
http://archive.ics.uci.edu/ml/datasets/Car+Evaluation
[Online Available].
[19] Heart Disease Dataset
http://archive.ics.uci.edu/ml/datasets/Heart+Disease
[Online Available].
[20] G. Hinton, S. Osindero, and Y. W. Teh. “A fast
learning algorithm for deep belief nets. Neural
Computation”, Vol.18, Issue.7,1527–1554, 2006.
