helpfn

IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020 2237

Generative and Discriminative Fuzzy Restricted Boltzmann Machine Learning for Text and

Image Classification C. L. Philip Chen , Fellow, IEEE, and Shuang Feng

Abstract—The restricted Boltzmann machine (RBM) is an excellent generative learning model for feature extraction. By extending its parameters from real numbers to fuzzy ones, we have developed the fuzzy RBM (FRBM) which is demon- strated to possess better generative capability than RBM. In this paper, we first propose a generative model named Gaussian FRBM (GFRBM) to deal with real-valued inputs. Then, moti- vated by the fact that the discriminative variant of RBM can provide a self-contained framework for classification with com- petitive performance compared with some traditional classifiers, we establish the discriminative FRBM (DFRBM) and discrimina- tive GFRBM (DGFRBM) that combine both the generative and discriminative facility by adding extra neurons next to the input units. Specifically, they can be trained into excellent stand-alone classifiers and retain outstanding generative capability simul- taneously. The experimental results including text and image (both clean and noisy) classification indicate that DFRBM and DGFRBM outperform discriminative RBM models in terms of reconstruction and classification accuracy, and they behave more stable when encountering noisy data. Moreover, the proposed learning models show some promising advantages over other standard classifiers.

Index Terms—Discriminative learning, fuzzy number, Gaussian fuzzy restricted Boltzmann machine (GFRBM), image classification.

I. INTRODUCTION

T HE RESTRICTED Boltzmann machine (RBM) [1] is aneural network with stochastic neurons (or units) which Manuscript received April 5, 2018; revised July 6, 2018; accepted

August 29, 2018. Date of publication October 2, 2018; date of current ver- sion April 15, 2020. This work was supported in part by the National Natural Science Foundation of China under Grant 61751202, Grant 61751205, and Grant 61572540, in part by the Macau Science and Technology Development Fund (FDCT) under Grant 019/2015/A1, Grant 079/2017/A2, and Grant 024/2015/AMJ, in part by the MYRG of University of Macau, and in part by the Teacher Research Capacity Promotion Program of Beijing Normal University, Zhuhai. This paper was recommended by Associate Editor C.-F. Juang. (Corresponding author: Shuang Feng.)

C. L. P. Chen is with the Faculty of Science and Technology, University of Macau, Macau 999078, China, also with the Department of Navigation, Dalian Maritime University, Dalian 116026, China, and also with the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100080, China (e-mail: [email protected]).

S. Feng is with the School of Applied Mathematics, Beijing Normal University, Zhuhai 519087, China, and also with the Faculty of Science and Technology, University of Macau, Macau 999078, China (e-mail: [email protected]).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TCYB.2018.2869902

Fig. 1. RBM with n visible and m hidden neurons.

was first described in [2], where it was referred to as the “harmonium.” The composition of this structure consists two functional parts: 1) visible and 2) hidden units as depicted in Fig. 1. The distinctive feature of RBM is that there are no connections among the visible or hidden neurons, and there are only symmetric connections between the two groups of units. The RBM is capable of modeling a probability distribu- tion when samples from a training data set are clamped onto the visible units. The most common method of training RBM is the k-step contrastive divergence (CD-k) algorithm [3] that approximates the original gradient by sampling the visible and hidden units alternately in k steps.

In most situations, the RBM is trained in an unsupervised way to be a generative model that captures the representative characteristics of inputs. Thus, the trained RBM can be used to reconstruct the input samples, or the values of its hidden units can be regarded as the inputs of a new RBM stacked on top of it which will eventually result in a deep belief network (DBN) [1] or deep Boltzmann machine (DBM) [4]. One other way is to initialize a feedforward neural network with one hidden layer by the parameters of the trained RBM. Both DBN/DBM and NN initialized by RBMs are usually fine- tuned by other supervised algorithms such as BP algorithm so as to perform a classification task. RBMs have been applied successfully to numerous real-world problems involving high- dimensional data, such as image classification [5]–[8]; text identification [9], [10]; and pattern recognition [11].

Nevertheless, it is demonstrated that RBMs can benefit from the incorporation of labels as well as the inputs, and can also be trained by one-step contrastive divergence (CD-1) algorithm to provide a self-contained framework [10] for classification

2168-2267 c© 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

2238 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020

problems. An extra group of neurons representing the labels of inputs is added to the visible units in order to identify differ- ent items. Then this RBM variant named discriminative RBM (DRBM) is able to model the joint distribution of the input and its target (label). It can categorize an unlearned sample by computing a condition probability [see (6)] after training.

It has been well demonstrated that the performance of con- ventional neural networks can be greatly improved by incorpo- rating fuzzy theory, and numerous fuzzy-neuro networks [such as: fuzzy RBF networks [12]–[14] and fuzzy BLS [15] based on BLS [16], [17] and fuzzy deep neural network [18]] have been proposed recently with diverse applications to system controlling and modeling [19]–[22], pattern recognition and classification [23], [24]. However, they are usually applied to low-dimensional data and merely handle classification problems involving raw images.

The fuzzy RBM (FRBM) was first proposed in [25] to deal with the uncertainty and fuzziness in real-world problems by replacing the parameters of an RBM with fuzzy numbers. Further, a more reasonable and enhanced theoretical founda- tion for establishing FRBMs, as well as the novel learning algorithms, is investigated and developed in [26]. The FRBM possesses better generative ability compared with the RBM in image inpainting and reconstruction. However, we always encounter some gray scale and color images for classification and recognition in real-world problems, such as hyperspectral images [27] and human and scene pictures [28], [29]. The existing FRBM is mainly trained to be a generative model by unsupervised algorithms for binary data, or used as a feature extractor to produce good features by its hidden units for other classifiers. Hence, it is aimed to develop some new types of FRBM in this paper, and the contributions are summarized as follows.

1) We propose a new generative model named the Gaussian FRBM (GFRBM) by replacing the binary visible units of FRBM with Gaussian ones to deal with real-valued input variables.

2) We establish the discriminative FRBM (DFRBM) and discriminative GFRBM (DGFRBM) based on FRBM and GFRBM by adding a new component next to the visible units for calculating the labels of input samples, and design their learning algorithms.

3) The DFRBM and DGFRBM can be trained into stand- alone classifiers and retain outstanding generative capa- bility, therefore, they can perform image reconstruction and classification simultaneously. The relation of the RBM and FRBM-based models is depicted in Fig. 2.

It is worth noting that the learning procedure of the proposed fuzzy variants in this paper is essentially generative (like RBM) based on the CD-1 algorithm, and it is much differ- ent from those of MLP and deep neural networks which rely on back-propagating errors between the model outputs and the required targets.

We subsequently use the DFRBM and DGFRBM which are treated as stand-alone classifiers for text and image classifi- cation. We can also perform image reconstruction by them simultaneously. Four representative data sets including the MNIST handwritten data set, 20 newsgroup data set, and

Fig. 2. Relation of the proposed models.

Fig. 3. Example of DRBMs.

two face data sets (Olivetti and Yale) are employed to com- pare our method with DRBM and discriminative Gaussian RBM (DGRBM). The experimental results demonstrate that our proposed models can achieve higher classification accu- racy. When noises are added to the test face samples, the DFRBM and DGFRBM also show better stability in terms of accuracy. Meanwhile, it can obtain competitive classification performances compared to SVM and MLP which are trained in a supervised manner. Thus, we draw a comprehensive con- clusion that the FRBM and its discriminative variants have promising advantages over the RBM and classical classifiers.

Some basic preliminaries about the DRBM, Gaussian RBM (GRBM), and FRBM are provided in Section II. Section III shows how to establish and train a GFRBM. The learning algorithms for the DFRBM and DGFRBM are proposed in Section IV. Three experiments are carried out and compared in Section V. Section VI summarizes our whole work.

II. PRELIMINARIES

We introduce some basic knowledge of the DRBM and GRBM, as well as the FRBM in this section.

A. Discriminative Restricted Boltzmann Machines

An RBM can be represented by a bipartite undirected graph- ical models as shown in Fig. 1. It consists of n visible units x = (x1, x2, . . . , xn) representing observable data and m hid- den units h = (h1, h2, . . . , hm). The RBMs are usually trained to learn the probability distribution of training samples and are treated as generative models, however, they can also learn the joint distribution of training samples and their corresponding labels t (i.e., the classes they belonged to). Thus, this type of RBM with an extra group of neurons representing the labels is called DRBM [10] (see Fig. 3) whose details are described as follows.

Given a training data set S = {(xi, ti)}|S|i=1 which is divided into K classes, the target vector ti = (ti1, ti2, . . . , tiK )

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

CHEN AND FENG: GENERATIVE AND DFRBM LEARNING FOR TEXT AND IMAGE CLASSIFICATION 2239

represents the class of input vector xi where

tik = {

1, if xi ∈ Class k 0, if xi /∈ Class k.

The joint probability distribution under the model is

P(x, t, h) = e −E(x,t,h)∑

x,t,h e −E(x,t,h) (1)

where the energy function

E(x, t, h) = −hWxT − hUtT − xbT − hcT − tdT (2) and the free energy function

FD(x, t) = − n∑

j=1 bjxj −

K∑ k=1

dktk

− m∑

i=1 ln (

1 + eci+ ∑n

j=1 wijxj+ ∑K

k=1 uik tk )

(3)

with parameters θ = (W, U, b, c, d). W and U are the weight matrices connecting hidden units to input variables and the label units, respectively. b, c, and d are bias terms.

The graph of DRBM has connections only between the layer of hidden units and the layer of visible units which means the hidden and visible variables are conditionally independent given the states of each other, i.e.,

P(x|h) = n∏

j=1 P ( xj = 1|h

) = n∏

j=1 sigm

( bj +

m∑ i=1

wijhi

)

and

P(h|x, t) = m∏

i=1 P(hi = 1|x, t)

= sigm ⎛ ⎝ci +

n∑ j=1

wijxj + K∑

k=1 uiktk

⎞ ⎠

where sigm(x) = 1/(1 + e−x). Usually we have the following softmax function to compute label vector t = (t1, t2, . . . , tK ) given the hidden vector h:

P(tk|h) = edk +

∑m i=1 uik hi∑K

k=1 edk + ∑m

i=1 uik hi , k = 1, 2, . . . , K.

In order to train this discriminative model, we have to find θ that minimize the negative log-likelihood function

ln L(θ, S) = − ∑

(xi,ti)∈S ln P(xi, ti). (4)

The stochastic gradient descent method is widely employed to find θ and the accurate gradient is

∂ ln P(xi, ti) ∂θ

= −EP(h|xi,ti) [ ∂E(xi, ti, h)

∂θ

]

+ EP(x,t,h) [ ∂E(x, t, h)

∂θ

]

where the second expectation term is intractable and CD-1 [3] learning algorithm becomes a common way to obtain esti- mation of the log-likelihood gradient so as to facilitate the

Fig. 4. CD-1 algorithm: sampling hidden units given visible and label units (left), sampling visible and label units given hidden units (middle), and sampling hidden units again given visible and label units (right).

computation complexity in training RBMs. The training algo- rithm for DRBM can be referred to [10], and we only briefly illustrate it in Fig. 4. The updated formulas for the weight and bias terms are

�wij = � (⟨

hixj ⟩ data

− ⟨hixj⟩model )

�uik = �(〈hitk〉data − 〈hitk〉model) �bj = �

(⟨ xj ⟩ data

− ⟨xj⟩model )

�ci = �(〈hi〉data − 〈hi〉model) �dk = �(〈tk〉data − 〈tk〉model) (5)

where � is the learning rate. After training, we can employ the above DRBM to per-

form classification by calculating the following conditional probability when a new sample x is given for classifying:

P(tk|x) = edk

∏m i=1

( 1 + eci+uik+

∑n j=1 wijxj

) ∑K

k=1 edk ∏m

i=1 (

1 + eci+uik+ ∑n

j=1 wijxj ) (6)

where k = 1, 2, . . . , K. Then the class of sample x is determined by the index

of the largest conditional probability among {P(tk|x)|k = 1, 2, . . . , K}.

B. Gaussian Restricted Boltzmann Machines

We usually employ an RBM to model distributions over binary visible vectors and binary hidden units which is called the Bernoulli–Bernoulli RBM (or Binary RBM) [30]. However, we often encounter some problems with real-valued data especially when the data for visible units is obtained from a color or gray scale image. Generally, there are two typical ways to deal with real-valued input data [31].

1) We can still adopt the Binary RBM to model a continu- ous distribution but with a simple modification: the input data is scaled to the interval [0, 1], and given the hidden units the values of visible units are set to be the con- ditional probabilities instead of sampling binary values from them. The learning algorithm remains nearly the same as the Binary RBM except there is no sampling in computing visible units.

2) The binary state space of visible units is replaced by real-valued intervals which results in a Gaussian– Bernoulli RBM (also denoted as Gaussian-binary RBM and GRBM) [30], [32], [33]. For instance, if there are

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

2240 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020

n visible units x = (x1, x2, . . . , xn) representing observ- able data and m hidden units h = (h1, h2, . . . , hm). The state space of visible units is Rn while the state space of hidden units is still {0, 1}m.

Hereby, we describe the details of a GRBM as follows. The input data is usually preprocessed and scaled to [0, 1] or transformed to data with zero mean and unit standard deviation.

The energy function of a GRBM is defined as

E(x, h) = n∑

j=1

( xj − bj

)2 2σ 2j

− m∑

i=1 cihi −

m∑ i=1

n∑ j=1

wij σj

hixj (7)

where wij represents the weight connecting the ith hidden unit and the jth visible unit, bj and ci are bias terms associated with the corresponding visible and hidden unit, respectively. σj is the standard deviation associated with Gaussian visible units xj.

The joint probability distribution under the model is

P(x, h) = e −E(x,h)∑

x,h e −E(x,h) . (8)

Since the hidden units of a GRBM remain binary, we can easily derive its free energy function and write it as

FG(x) = n∑

j=1

( xj − bj

)2 2σ 2j

− m∑

i=1 ln

( 1 + eci+

∑n j=1

wij σj

xj )

. (9)

Then the probability distribution of the GRBM also has the following form:

P(x) = e −FG(x)∑

x e −FG(x) . (10)

Similar to the binary RBM, the hidden variables are con- ditionally independent from visible units and vice versa in the GRBM. However, the expressions of the conditional probabilities are a little different which are

P(x|h) = n∏

j=1 P ( xj|h

) = n∏

j=1 N

( xj|bj + σj

m∑ i=1

wijhi, σ 2 j

)

P(h|x) = m∏

i=1 P(hi|x) =

m∏ i=1

sigm

⎛ ⎝ci +

n∑ j=1

wij σj

xj

⎞ ⎠

where N(·|μ, σ 2) denotes the Gaussian probability density function with mean μ and standard deviation σ .

The updated rules for training the parameters of GRBM are

�wij = � (⟨

1

σj hixj

⟩ data

− ⟨

1

σj hixj

⟩ model

)

�bj = � (⟨

1

σ 2j xj

⟩ data

− ⟨

1

σ 2j xj

⟩ model

)

�ci = �(〈hi〉data − 〈hi〉model) (11) where � is the learning rate, i = 1, 2, . . . , m and j = 1, 2, . . . , n. Although, σj is a parameter that needs to be learned in training phase, for simplicity, we usually set it to be 1 when the GRBM is applied to real-world problems.

C. Fuzzy Restricted Boltzmann Machines

The details of establishing an FRBM with symmetric trian- gular fuzzy numbers can be referred to [26], so here we just introduce the process briefly. First of all, we can derive the following explicit expression of the free energy function of a Binary RBM:

FB(x) = − n∑

j=1 bjxj −

m∑ i=1

ln (

1 + eci+ ∑n

j=1 wijxj ) . (12)

Then we replace all the crisp weights W and bias terms b, c with symmetric triangular fuzzy numbers denoted by W , B, and C to construct an FRBM. We define the fuzzy free energy function of an FRBM as

F (x) = − n∑

j=1 b̃jxj −

m∑ i=1

ln (

1 + ec̃i+ ∑n

j=1 w̃ijxj ) . (13)

Here w̃ij, b̃j, and c̃i are set to be symmetric triangular fuzzy numbers. For example, the membership function of w̃ij is

μw̃ij (x) =

⎧⎪⎨ ⎪⎩

x−wLij wwidthij

, wLij ≤ x ≤ wMij wRij −x wwidthij

, wMij < x ≤ wRij (14)

where wLij, w R ij , and w

M ij represent its left bound, right bound,

and center, respectively, and wwidthij = wMij − wLij = wRij − wMij is the width.

Since (13) is a complicated fuzzy function, it is difficult to obtain the derivatives regarding the parameters directly. Therefore, we employ the crisp possibilistic mean value of a fuzzy number [34] to defuzzify this function. And we deduce the following defuzzified free energy function and its approximation:

Fp(x) = ∫ 1

0 α ( F L(α) + F R(α))dα

≈ FB ( x, θ L

) + FB(x, θ R) 2

(15)

where θ L = (WL, bL, cL) and θ R = (WR, bR, cR). The accuracy and validity of this approximation is guaranteed by several theorems proved in [26], thus we define the probability of an FRBM

P (

x, θ̃ )

= e −Fp(x)

Z (16)

where θ̃ = (W , B, C ) and Z = ∑x e−Fp(x) is the partition function.

The update rules for the parameters in an FRBM are

�wLij = �wRij = �

2

(⟨ hixj

⟩ data

− ⟨hixj⟩model )

�bLj = �bRj = �

2

(⟨ xj ⟩ data

− ⟨xj⟩model )

�cLi = �cRi = �

2 (〈hi〉data − 〈hi〉model) (17)

where

hi = hLi + hRi

2 , xj =

xLj + xRj 2

(18)

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

CHEN AND FENG: GENERATIVE AND DFRBM LEARNING FOR TEXT AND IMAGE CLASSIFICATION 2241

Algorithm 1 Training FRBM

Input: Training sample x(0); Learning rate �; Output: Updated parameters: θ L and θ R.

Start positive phase 1: sample hL(0) ∼ sigm(cL + x(0)WLT ); 2: sample hR(0) ∼ sigm(cR + x(0)WRT );

Start negative phase 3: sample xL(1) ∼ sigm(bL + hL(0)WL); 4: sample xR(1) ∼ sigm(bR + hR(0)WR); 5: compute hL(1) = sigm(cL + xL(1)WLT ); 6: compute hR(1) = sigm(cL + xR(1)WRT );

Start updated phase 7: update wLij, b

L j , c

L i and w

R ij , b

R j , c

R i according to Eq. (17)

8: return θ L and θ R.

and � is the learning rate. The training algorithm for FRBM is illustrated in Algorithm 1.

III. GAUSSIAN FUZZY RESTRICTED BOLTZMANN MACHINES

In this section, we will propose the GFRBM by extend- ing the binary visible units of FRBM to real-valued variables for the purpose of modeling some continuous distributions. We will always adopt symmetric triangular fuzzy numbers in our model due to the tradeoff between number of parameters and performance (please refer to [26] for the discussion on different types of fuzzy numbers).

First, we replace all the crisp weights W and bias terms b, c in (9) with symmetric triangular fuzzy numbers denoted as W , B, and C . And we define the fuzzy free energy function of a GFRBM as follows:

F (x) = n∑

j=1

( xj − b̃j

)2 2σ 2j

− m∑

i=1 ln

( 1 + ec̃i+

∑n j=1

w̃ij σj

xj

)

= n∑

j=1

x2j − 2xjb̃j + b̃2j 2σ 2j

− m∑

i=1 ln

( 1 + ec̃i+

∑n j=1

w̃ij σj

xj

)

� n∑

j=1 Ãj −

m∑ i=1

B̃i. (19)

Then we employ the crisp possibilistic mean value of a fuzzy number to defuzzify the above fuzzy free energy func- tion. Similarly, we derive the following defuzzified free energy function:

Fp(x) = ∫ 1

0 α ( F L(α) + F R(α))dα

= n∑

j=1

∫ 1 0

α (

ALj (α) + ARj (α) )

dα

− m∑

i=1

∫ 1 0

α ( BLi (α) + BRi (α)

) dα

= n∑

j=1 Ãjp (x) −

m∑ i=1

B̃ip (x). (20)

We have the following approximation according to [26]:

B̃ip (x) ≈ ln

⎛ ⎝1 + ecLi +

∑n j=1

wLij σj

xj

⎞ ⎠ + ln

⎛ ⎝1 + ecRi +

∑n j=1

wRij σj

xj

⎞ ⎠

2 .

(21)

Since xj is not a fuzzy number and bj is a symmetric triangular fuzzy number in Ãj, then

Ãjp (x) =

( x2j − 2xj

bLj +bRj 2 +

∫ 1 0 α

(( b̃2j

)L (α) +

( b̃2j

)R (α)

) dα

)

2σ 2j (22)

where b̃2j is the multiplication of symmetric triangular fuzzy numbers, and the result is very complicated when fuzzy number b̃j is neither positive nor negative. To simplify the discussion we assume that b̃j is either a positive or a nega- tive fuzzy number, then b̃2j is an approximate triangular fuzzy number with left bound (bLj )

2, right bound (bRj ) 2, and center

(bMj ) 2 if b̃j is positive, or with left bound (b

R j )

2, right bound

(bLj ) 2, and center (bMj )

2 if b̃j is negative [35]. And we give the following approximation:

∫ 1 0

α

(( b̃2j

)L (α) +

( b̃2j

)R (α)

) dα ≈

( bLj

)2 +

( bRj

)2 2

. (23)

Hence, (22) can be approximated and written as

Ãjp (x) ≈ x2j − xj

( bLj + bRj

) +

( bLj

)2 + (

bRj

)2 2

2σ 2j

= 1 2

⎛ ⎜⎝

( xj − bLj

)2 2σ 2j

+ (

xj − bRj )2

2σ 2j

⎞ ⎟⎠. (24)

By (21) and (24), we can deduce the following approxima- tion of the defuzzified free energy function (20):

Fp(x) ≈ 1

2

n∑ j=1

⎛ ⎜⎝

( xj − bLj

)2 2σ 2j

+ (

xj − bRj )2

2σ 2j

⎞ ⎟⎠

− m∑

i=1

ln

⎛ ⎝1 + ecLi +

∑n j=1

wLij σj

xj

⎞ ⎠ + ln

⎛ ⎝1 + ecRi +

∑n j=1

wRij σj

xj

⎞ ⎠

2

= FG ( x, θ L

) + FG(x, θ R) 2

. (25)

The accuracy and validity of this approximation is guar- anteed by initializing b̃j as a small positive or negative symmetric triangular fuzzy number, and we have found that the performance is not affected even when b̃j is initialized as a regular symmetric triangular fuzzy number in the experiments. Now it is interesting that the GFRBM has the same approxi- mation of the defuzzified free energy function as the FRBM,

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

2242 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020

thus we can establish a GFRBM analogously. The probability of a GFRBM is

P (

x, θ̃ )

= e −Fp(x)∑

x e −Fp(x) . (26)

Similarly, we define the following formulas which are generalized from the GRBM:

PL(hi|x) = sigm ⎛ ⎝cLi +

m∑ j=1

wLij σj

xj

⎞ ⎠

PL ( xj|h

) = N (

xj|bLj + σj n∑

i=1 wLijhi, σ

2 j

) (27)

and

PR(hi|x) = sigm ⎛ ⎝cRi +

m∑ j=1

wRij σj

xj

⎞ ⎠

PR ( xj|h

) = N (

xj|bRj + σj n∑

i=1 wRij hi, σ

2 j

) . (28)

Then the derivatives regarding the parameters θ̃ can be easily derived from (25)

∂Fp(x)

∂wLij = − 1

2σj PL(hi|x)xLj

∂Fp(x)

∂bLj = − 1

2σ 2j

( xLj − bLj

)

∂Fp(x)

∂cLi = − 1

2 PL(hi|x) (29)

and ∂Fp(x)

∂W Rij = − 1

2σj PR(hi|x)xRj

∂Fp(x)

∂bRj = − 1

2σ 2j

( xRj − bRj

)

∂Fp(x)

∂cRi = − 1

2 PR(hi|x). (30)

We also employ CD-1 algorithm to learn the parameters in the above equations. In order to maintain the symmetric triangular shape of the parameters, the update rules for the left and right bounds are identical, i.e.,

�wLij = �wRij = �

2σj

(⟨ hixj

⟩ data

− ⟨hixj⟩model )

�bLj = �bRj = �

2σ 2j

(⟨ xj ⟩ data

− ⟨xj⟩model )

�cLi = �cRi = �

2 (〈hi〉data − 〈hi〉model) (31)

where

hi = hLi + hRi

2 , xj =

xLj + xRj 2

(32)

and � is the learning rate. We will always set σj = 1 in the experiment section. The training algorithm for GFRBM with symmetric triangular fuzzy numbers is illustrated in Algorithm 2.

Algorithm 2 Training GFRBM

Input: Training sample x(0); Learning rate �; σ = (σ1, σ2, . . . , σn); Output: Updated parameters: θ L and θ R.

Start positive phase 1: sample hL(0) ∼ sigm(cL + (x(0)/σ )WLT ); 2: sample hR(0) ∼ sigm(cR + (x(0)/σ )WRT );

Start negative phase 3: sample xL(1) ∼ N(xL(1)|bL + σ hL(0)WL, σ 2); 4: sample xR(1) ∼ N(xR(1)|bR + σ hR(0)WR, σ 2); 5: compute hL(1) = sigm(cL + (xL(1)/σ )WLT ); 6: compute hR(1) = sigm(cR + (xR(1)/σ )WRT );

Start updated phase 7: update wLij, b

L j , c

L i and w

R ij , b

R j , c

R i according to Eq. (31)

8: return θ L and θ R.

IV. DISCRIMINATIVE FUZZY RESTRICTED BOLTZMANN MACHINES

The discriminative variants of FRBM and GFRBM are proposed here.

A. Discriminative FRBM Using Softmax Function

The way of extending an FRBM to a DRBM is similar to turning an RBM into a DRBM. We have to add an extra set of input units representing the target class t connected to the hidden units with weight matrix U which consists of symmetric triangular fuzzy numbers ũik, and the bias term of input units for the label vector t is D consisting of symmetric triangular fuzzy numbers d̃k. Hence, the fuzzy free energy function of a DFRBM is

F (x, t) = − n∑

j=1 b̃jxj −

K∑ k=1

d̃ktk

− m∑

i=1 ln (

1 + ec̃i+ ∑n

j=1 w̃ijxj+ ∑K

k=1 ũik tk ) . (33)

We denote UL and UR are the left and right bounds of U , dL and dR are the left and right bounds of D , respectively. Thus, we can employ the same method adopted in FRBM to obtain the following approximate defuzzified free energy function:

Fp(x, t) ≈ FD

( x, t, θ L

) + FD(x, t, θ R) 2

(34)

where θ L = (WL, UL, bL, cL, dL) and θ R = (WR, UR, bR, cR, dR).

Now, we define the probability of a DFRBM

P (

x, t, θ̃ )

= e −Fp(x,t)∑

x,t e −Fp(x,t) (35)

and the negative log-likelihood function is

ln L (

x, t, θ̃ )

= − ∑

x,t∈D ln P

( x, t, θ̃

) (36)

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

CHEN AND FENG: GENERATIVE AND DFRBM LEARNING FOR TEXT AND IMAGE CLASSIFICATION 2243

where θ̃ = (W , U , B, C , D), hence

− ∂ ln P

( x, t, θ̃

) ∂θ L

= ∂Fp(x, t) ∂θ L

− EP [ ∂Fp(x, t)

∂θ L

]

− ∂ ln P

( x, t,θ̃

) ∂θ R

= ∂Fp(x, t) ∂θ R

− EP [ ∂Fp(x, t)

∂θ R

] . (37)

It is straightforward to generalize from FRBM that

PL ( xj|h

) = sigm (

bLj + m∑

i=1 wLijhi

)

PR ( xj|h

) = sigm (

bRj + m∑

i=1 wRij hi

)

PL(tk|h) = ed

L k +

∑m i=1 uLik hi∑K

k=1 e dLk +

∑m i=1 uLik hi

PR(tk|h) = ed

R k +

∑m i=1 uRik hi∑K

k=1 e dRk +

∑m i=1 uRik hi

PL(hi|x, t) = sigm ⎛ ⎝cLi +

n∑ j=1

wLijxj + K∑

k=1 uLiktk

⎞ ⎠

PR(hi|x, t) = sigm ⎛ ⎝cRi +

n∑ j=1

wRij xj + K∑

k=1 uRiktk

⎞ ⎠. (38)

The update rules for the parameters are

�wLij = �wRij = �

2

(〈hixj〉data − 〈hixj〉model) �uLik = �uRik =

�

2 (〈hitk〉data − 〈hitk〉model)

�bLj = �bRj = �

2

(〈xj〉data − 〈xj〉model) �cLi = �cRi =

�

2 (〈hi〉data − 〈hi〉model)

�dLk = �dRk = �

2 (〈tk〉data − 〈tk〉model) (39)

where tk = (tLk + tRk )/2. Although, the conditional probability formulas of t given

hidden vector h are different from that of x, the learning algorithm for DFRBM can be also designed based on CD-1 algorithm and is described detailedly in Algorithm 3.

When the learning process is completed and a new sample x is provided for classifying, we can get the following two conditional probabilities from the DFRBM:

PL(tk|x) = ed

L k ∏m

i=1 (

1 + ecLi +uLik+ ∑n

j=1 wLij xj )

∑K k=1 e

dLk ∏m

i=1 (

1 + ecLi +uLik+ ∑n

j=1 wLij xj )

and

PR(tk|x) = ed

R k ∏m

i=1 (

1 + ecRi +uRik+ ∑n

j=1 wRij xj )

∑K k=1 e

dRk ∏m

i=1 (

1 + ecRi +uRik+ ∑n

j=1 wRij xj )

where k = 1, 2, . . . , K. Consequently, the probabilities of belonging to a certain class for the sample x is

P(tk|x) = PL(tk|x) + PR(tk|x)

2 . (40)

Algorithm 3 Training DFRBM

Input: Training sample (x(0), t(0)); Learning rate �; Output: Updated parameters: θ L and θ R.

Start positive phase 1: sample hL(0) ∼ PL(hL(0) | x(0), t(0)); 2: sample hR(0) ∼ PR(hR(0) | x(0), t(0));

Start negative phase 3: sample xL(1) ∼ PL(xL(1) | hL(0)); 4: sample xR(1) ∼ PR(xR(1) | hR(0)); 5: sample tL(1) ∼ PL(tL(1) | hL(0)); 6: sample tR(1) ∼ PR(tR(1) | hR(0)); 7: compute hL(1) = PL(hL(1) | xL(1), tL(1)); 8: compute hR(1) = PR(hR(1) | xR(1), tR(1));

Start updated phase 9: update wLij, u

L ik, b

L j , c

L i , d

L k and w

R ij , u

R ik, b

R j , c

R i , d

R k according to

Eq. (39). 10: return θ L and θ R.

Then the index of the largest value of {P(tk|x)|k = 1, 2, . . . , K} determines the class of x.

B. Discriminative FRBM Using Sigmoid Function

We have illustrated that an FRBM can be trained to be a much better generative model than RBM [26]. The FRBM has a greater ability in reconstructing input images, therefore, we now consider another type of DFRBM where we replace the softmax function in (38) with the following sigmoid function when calculating the label vector t:

PL(tk|h) = sigm (

dLk + n∑

i=1 uLikhi

)

PR(tk|h) = sigm (

dRk + n∑

i=1 uRikhi

) . (41)

We denote this type of model by DFRBM-sigm while denote the discriminative FRBM discussed above by DFRBM- softmax. The learning algorithm for DFRBM-sigm is almost the same as Algorithm 3 except that the calculating formulas for labels are replaced, so we omit the algorithm here.

C. Discriminative Gaussian FRBM

We can extend the GFRBM to be a discriminative model just like the way we establish the DFRBM. The only change is to replace the binary visible units in a DFRBM with Gaussian units, and sample the visible and hidden units according to (27) and (28). The DGFRBM is mainly applied to classifying color and gray scale images such as face recognition in the latter experiment section.

V. EXPERIMENTS

In the following experiments, the parameters of our proposed models are chosen based on the experience of existing papers and our trials. Specifically, we usually select the learning rate from {0.001, 0.005, 0.01, 0.05, 0.25}, weight decay from {0.0005, 0.001, 0.005} and the number of hid- den units from {100, 200, . . . , 1000}. The parameters of RBM models are set to their optimal values either according to the references or based on our experiments.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

2244 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020

TABLE I COMPARISON OF THE CLASSIFICATION PERFORMANCES ON THE MNIST DATA SET

Fig. 5. MNIST training samples.

A. MNIST Handwritten Database

The MNIST database of handwritten digits [36] consists of a training set of 60 000 examples, and a test set of 10 000 examples. Each sample is an image representing one of the numbers from 0 to 9. The size of every image is 28 × 28 gray- scaled pixels (see Fig. 5). We use this data set to compare the classifying capabilities of DRBM and DFRBM.

When sigmoid function is adopted to compute both the vis- ible and label units of DFRBM-sigm, the images and their labels can be treated as the “visible units” as a whole, i.e., labels are embedded in the corresponding images. Thus, the objective of this model is to reconstruct the correct “images” containing labels as well as the handwriting numbers.

After training the DRBM and DFRBM, we employ (6) and (40) to determine which class the test sample should be, respectively. Each possible label is turned on in turn with the test sample and the one that has the largest probability is chosen to be the most likely class.

The whole training data set is employed to train these dis- criminative models. The DFRBM is trained for 25 epochs while the DRBM is trained for 40 epochs with different num- bers of hidden units. We implement both the models and perform them on the test set five times. For DFRBM the learn- ing rate is 0.25 and weight decay is 0.0005, while for DRBM the learning rate is set to be 0.05 and weight decay is 0.001 [4]. The average classification errors of DRBM and DFRBM are listed in Table I. Meanwhile, the classification error of random forest classifier is 2.94% [10] and the classification error of MLP is 2.61% [37].

We observe that the DFRBM achieves higher classification accuracy in different conditions of hidden units, and it con- sumes less time than DRBM. It seems that the DFRBM with only 500 hidden units can behave almost as well as DRBM with 1000 hidden units in accuracy, but the DFRBM costs only half of the time. Meanwhile, it outperforms the random

TABLE II TOPICS OF THE 20 NEWSGROUP DATA SET

forest classifier and MLP. The learning time of DFRBM- sigm and DFRBM-softmax are nearly the same, however, the DFRBM-softmax has a slightly higher accuracy than DFRBM-sigm.

B. Text Classification

Now, we consider the problem of classifying texts with the 20 newsgroup data set [38]. The 20 newsgroups data set is a collection of 11 269 newsgroup documents, partitioned across 20 different topics (see Table II) while the test data set con- tains 7505 examples. The binary input variables consist of 5000 different words that are most frequently appeared in the documents.

We first adopt a small version denoted by 20 newsgroups- 4 data set [39] of the 20 newsgroups data set which has 16 242 documents belonging to four different classes: comp.*, rec.*, sci.*, and talk.*. One hundred frequently used words are employed as binary inputs. We randomly choose 10 000 exam- ples of the subset as the training data set and the rest 6242 examples as the test set. The number of hidden unit is 100. After training for 40 epochs, the learning errors of DRBM and DFRBM become stable [see Fig. 6(a)]. We implement each of the models ten times and average the classification errors to compare their classification capability. The average errors and standard deviations are listed in Table III.

It can be concluded that DFRBM performs better on this mini data set than DRBM. The performance of DFRBMs using softmax and sigmoid functions in computing labels have no difference.

Moreover, we check their classification errors on the test set every epoch during the learning phase. The comparison result is shown in Fig. 6(b), which indicates that both DFRBM models can achieve lower classification errors than DRBM

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

CHEN AND FENG: GENERATIVE AND DFRBM LEARNING FOR TEXT AND IMAGE CLASSIFICATION 2245

TABLE III COMPARISON OF THE AVERAGE CLASSIFICATION ERRORS ON

20 NEWSGROUPS-4 DATA SET

(a) (b)

Fig. 6. Error comparison on 20 newsgroups-4 data set during 40 learning epochs. (a) Training error. (b) Testing classification error.

TABLE IV COMPARISON OF THE CLASSIFICATION ERRORS ON

20 NEWSGROUPS DATA SET

after the learning process is convergent. It also shows that the testing classification errors of DRBM have more oscillations than the DFRBM during training.

Second, we use the whole 20 newsgroup data set for training and test. The training data set is divided into 191 mini batches and each mini batch contains 59 examples for model learning. The number of hidden neurons is 1000. Both the DRBM and DFRBM are trained for 200 epochs and then tested by the 7505 examples. The learning rate and weight decay for DRBM are set according to [10]. Here, we only train the DFRBM-softmax model since DFRBM-sigm performs nearly the same as it. The comparison results are given in Table IV. The errors of SVM, Random Forest, NNet, and RBM+NNet are cited from [10].

We can also conclude that DFRBM improves the classify- ing result of 20 newsgroup data set compared with the other discriminative models.

C. Facial Recognition

1) Olivetti Face Data Set: The Olivetti face data set [40] contains a set of 400 gray-scaled face images of 40 different persons taken between April 1992 and April 1994 at AT &T Laboratories Cambridge. There are ten different images of each person. We first use the size of 64×64 [41] here instead of the original size and some examples are depicted in Fig. 7(a).

The 400 images are divided into the training set and test set as follows: we randomly choose six pictures of each person

(a) (b)

(c) (d)

Fig. 7. Some samples from Olivetti face data set: (a) original training images, and the test data with: (b) 10% random pixel corruption, (c) 15 dB Gaussian noise, and (d) 20% block corruption.

TABLE V COMPARISON OF THE RECOGNITION ERRORS ON

OLIVETTI FACES DATA SET

and there are 240 images total in the training set, then the test set is composed of the rest 160 pictures.

The whole data set is randomly divided into six equal min- batches for model learning. The learning rate is 0.05, while the weight decay is 0.0005 for DFRBM and 0.001 for DRBM, respectively. We also compare the performances of DGRBM and DGFRBM. The learning rate is 0.001 for DGRBM and 0.005 for DGFRBM. The classification errors listed in Table V are the average of ten experiments. Moreover, the lowest error rate of DFRBM during the 1000 learning epochs is 20% while that of DRBM is 28.13% using 200 hidden units.

It is obvious that the DFRBM has a competitive advantage on face recognition over DRBM. The DFRBM can achieve much lower recognition errors using only 200 hidden units even with a great amount of input attributes (i.e., 64 × 64 = 4094) in every image. Meanwhile, the Gaussian types of RBM and FRBM can greatly reduce the classification errors for that their errors are nearly half of the DRBM and DFRBM. It demonstrates that the Gaussian visible units are more capable of handling real-valued face images. We can conclude that DFRBM and DGFRBM outperform corresponding DRBM and DGRBM in this face recognition data set.

It is interesting that, with more hidden units, binary visi- ble units can also behave comparatively well in real-valued images. Besides, as the number of hidden units increases, the advantages of DFRBM and DGFRBM become smaller since their parameters are nearly twice as many as that of DRBM

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

2246 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020

(a) (b) (c) (d) (e)

Fig. 8. Four faces from Oliveetti data set and their reconstructions: (a) original faces, reconstructed by (b) DRBM, (c) DFRBM, (d) GRBM, and (e) DGFRBM. It is easy to see that DGFRBM produces the best reconstruction.

TABLE VI COMPARISON OF THE RECOGNITION ERRORS ON NOISY OLIVEETTI FACES DATA SET (200 HIDDEN UNITS)

TABLE VII COMPARISON OF THE RECOGNITION ERRORS ON YALE DATABASE (200 HIDDEN UNITS)

and DGRBM, but there are only 240 samples in the training set which is not quite sufficient for a good learning of parameters in FRBM.

Although these models are trained in a discriminative manner, they also learn a good generative capability simul- taneously. Fig. 8 shows four training samples and their reconstructions produced by the four models.

Then we adopt three noisy variants of the data set to see if the performances of these models are robust: two of them are obtained by randomly corrupting 10% pixels of every image with white and black points or replacing 20% block of each image with a monkey face in the test set, the other one is obtained by adding 15 dB Gaussian noises to the test images (see Fig. 7). The results are presented in Table VI, which show that the classification accuracies of our proposed variants of FRBM on the noisy data are even higher than corresponding DRBM and DGRBM on the original data and better than SVM. It demonstrates the superiority of DFRBM and DGFRBM in handling noisy data produced from fuzzy environments.

2) Yale Database: The Yale Face Database [42] contains 165 gray scale images in GIF format of 15 individuals. There are 11 images per subject, one per different facial expression or configuration: center-light, w/glasses, happy, left-light, w/no glasses, normal, right-light, sad, sleepy, surprised, and wink. We use a processed data set with size of 32×32 [41], and three noisy sets produced in a similar way as above section are also used for comparison, except that we apply the 10% random

(a) (b)

(c) (d)

Fig. 9. Some samples from Yale face data set: (a) original training images, and the test images with: (b) 10% random pixel corruption, (c) 15 dB Gaussian noise, and (d) 10% random pixel+20% block corruption.

pixel corruption and 20% block corruption to each image in test set simultaneously to increase the difficulty (see Fig. 9).

The DRBM and DFRBM are trained for 1000 epochs while the DGFRBM and DGFRBM are only trained for 200 epochs for comparison since their errors become stable quickly. The classification error rates are listed in Table VII. We also depict four faces and their reconstructions produced by the four models in Fig. 10.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

CHEN AND FENG: GENERATIVE AND DFRBM LEARNING FOR TEXT AND IMAGE CLASSIFICATION 2247

(a) (b) (c) (d) (e)

Fig. 10. Four faces from Yale database and their reconstructions: (a) original faces, reconstructed by (b) DRBM, (c) DFRBM, (d) GRBM, and (e) DGFRBM. Also the DGFRBM produces the best reconstruction.

Unsurprisingly, the performance of DGFRBM is better than DGRBM, while DFRBM is better than DRBM on both the original and noisy data sets. Moveover, when encountering real-valued data, binary type visible units need more learning epochs to achieve a satisfying accuracy than Gaussian type. In addition, the dimensionality may be the smaller the better if there are not too many labeled samples for model training.

VI. CONCLUSION

We design the discriminative variants of FRBM, including the DFRBM and DGFRBM for document and image classi- fication since they can be trained into stand-alone classifiers without extra classifying layers or BP learning. In order to better handle the real-valued inputs from gray scale and color images, we first propose the GFRBM which is extended from GRBM and FRBM by replacing the binary visible units with Gaussian ones. Then we illustrate how a DFRBM is estab- lished based on the generative FRBM by adding a group of neurons to the visible units. The DGFRBM can be generalized likewise, and the corresponding learning algorithms are also discussed and designed for these models.

We apply the DFRBM and DGFRBM to some represen- tative benchmarks for classification: the MNIST data set, 20 newsgroup data set, and face data sets (i.e., Olivetti, Yale, and their noisy variants). The experimental results demon- strate that the proposed models outperform the corresponding variants of RBM in reconstruction and classification accu- racy, and behave much more robust when encountering noisy data. Meanwhile, they show some promising advantages over classical classifiers, such as SVM and MLP in term of accu- racy. Moreover, the proposed models can retain the excellent generative capability of FRBM simultaneously.

It is worth to mention that the proposed fuzzy models may consume more training time due to the increase of parameters from fuzzy numbers. Therefore, the future work consists of designing a more efficient learning algorithm and developing some fuzzy deep models based on these variants of FRBM which are expected to achieve much higher classification accuracy.

REFERENCES

[1] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, Jul. 2006.

[2] D. E. Rumelhart et al., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vols. 1–2. Cambridge, MA, USA: MIT Press, 1986.

[3] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Comput., vol. 14, no. 8, pp. 1771–1800, 2002.

[4] R. Salakhutdinov and G. Hinton, “An efficient learning procedure for deep Boltzmann machines,” Neural Comput., vol. 24, no. 8, pp. 1967–2006, Aug. 2012.

[5] R. Salakhutdinov, J. B. Tenenbaum, and A. Torralba, “Learning with hierarchical-deep models,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1958–1971, Aug. 2013.

[6] Y. Tang, R. Salakhutdinov, and G. Hinton, “Robust Boltzmann machines for recognition and denoising,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2012, pp. 2264–2271.

[7] T. Schmah, G. E. Hinton, R. S. Zemel, S. L. Small, and S. Strother, “Generative versus discriminative training of RBMs for classification of fMRI images,” in Proc. Adv. Neural Inf. Process. Syst., 2008, pp. 1409–1416.

[8] N. Le Roux, N. Heess, J. Shotton, and J. Winn, “Learning a generative model of images by factoring appearance and shape,” Neural Comput., vol. 23, no. 3, pp. 593–650, 2011.

[9] H. Larochelle and Y. Bengio, “Classification using discriminative restricted Boltzmann machines,” in Proc. 25th Int. Conf. Mach. Learn., 2008, pp. 536–543.

[10] L. Hugo, M. Michael, P. Razvan, and B. Yoshua, “Learning algorithms for the classification restricted Boltzmann machine,” J. Mach. Learn. Res., vol. 13, pp. 643–669, Mar. 2012.

[11] N. Ji, J. Zhang, C. Zhang, and L. Wang, “Discriminative restricted Boltzmann machine for invariant pattern recognition with lin- ear transformations,” Pattern Recognit. Lett., vol. 45, pp. 172–180, Aug. 2014.

[12] S. Mitra and J. Basak, “FRBF: A fuzzy radial basis function network,” Neural Comput. Appl., vol. 10, no. 3, pp. 244–252, 2001.

[13] K. Mali and S. Mitra, “Symbolic classification, clustering and fuzzy radial basis function network,” Fuzzy Sets Syst., vol. 152, no. 3, pp. 553–564, 2005.

[14] S.-K. Oh, W.-D. Kim, W. Pedrycz, and K. Seo, “Fuzzy radial basis function neural networks with information granulation and its par- allel genetic optimization,” Fuzzy Sets Syst., vol. 237, pp. 96–117, Feb. 2014.

[15] S. Feng and C. L. P. Chen, “Fuzzy broad learning system: A novel neuro- fuzzy model for regression and classification,” IEEE Trans. Cybern., to be published.

[16] C. L. P. Chen and Z. Liu, “Broad learning system: An effective and efficient incremental learning system without the need for deep architec- ture,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 1, pp. 10–24, Jan. 2018.

[17] C. L. P. Chen, Z. Liu, and S. Feng, “Universal approximation capabil- ity of broad learning system and its structural variations,” IEEE Trans. Neural Netw. Learn. Syst., to be published.

[18] Y. Deng, Z. Ren, Y. Kong, F. Bao, and Q. Dai, “A hierarchical fused fuzzy deep neural network for data classification,” IEEE Trans. Fuzzy Syst., vol. 25, no. 4, pp. 1006–1012, Aug. 2017.

[19] C. L. P. Chen and Y.-H. Pao, “An integration of neural network and rule-based systems for design and planning of mechanical assemblies,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 23, no. 5, pp. 1359–1371, Sep./Oct. 1993.

[20] C. L. P. Chen and S. Xie, “Freehand drawing system using a fuzzy logic concept,” Comput.-Aided Design, vol. 28, no. 2, pp. 77–89, 1996.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.

2248 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 50, NO. 5, MAY 2020

[21] G. Wen, C. L. P. Chen, Y.-J. Liu, and Z. Liu, “Neural network-based adaptive leader-following consensus control for a class of nonlinear multiagent state-delay systems,” IEEE Trans. Cybern., vol. 47, no. 8, pp. 2151–2160, Aug. 2017.

[22] H. Han, X.-L. Wu, and J.-F. Qiao, “Nonlinear systems modeling based on self-organizing fuzzy-neural-network with adaptive compu- tation algorithm,” IEEE Trans. Cybern., vol. 44, no. 4, pp. 554–564, Apr. 2014.

[23] M. F. Mohammed and C. P. Lim, “An enhanced fuzzy min–max neural network for pattern classification,” IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 3, pp. 417–429, Mar. 2015.

[24] G.-D. Wu and P.-H. Huang, “A vectorization-optimization-method-based type-2 fuzzy neural network for noisy data classification,” IEEE Trans. Fuzzy Syst., vol. 21, no. 1, pp. 1–15, Feb. 2013.

[25] C. L. P. Chen, C.-Y. Zhang, L. Chen, and M. Gan, “Fuzzy restricted Boltzmann machine for the enhancement of deep learning,” IEEE Trans. Fuzzy Syst., vol. 23, no. 6, pp. 2163–2173, Dec. 2015.

[26] S. Feng and C. L. P. Chen, “A fuzzy restricted Boltzmann machine: Novel learning algorithms based on the crisp possibilistic mean value of fuzzy numbers,” IEEE Trans. Fuzzy Syst., vol. 26, no. 1, pp. 117–130, Feb. 2018.

[27] Y. Zhou and Y. Wei, “Learning hierarchical spectral–spatial features for hyperspectral image classification,” IEEE Trans. Cybern., vol. 46, no. 7, pp. 1667–1678, Jul. 2016.

[28] X. Lu, X. Li, and L. Mou, “Semi-supervised multitask learning for scene recognition,” IEEE Trans. Cybern., vol. 45, no. 9, pp. 1967–1976, Sep. 2015.

[29] B. Ni, Y. Pei, P. Moulin, and S. Yan, “Multilevel depth and image fusion for human activity detection,” IEEE Trans. Cybern., vol. 43, no. 5, pp. 1383–1394, Oct. 2013.

[30] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.

[31] A. Fischer and C. Igel, “Training restricted Boltzmann machines: An introduction,” Pattern Recognit., vol. 47, no. 1, pp. 25–39, 2014.

[32] V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proc. 27th Int. Conf. Mach. Learn. (ICML-10), 2010, pp. 807–814.

[33] R. D. Hjelm et al., “Restricted Boltzmann machines for neuroimaging: An application in identifying intrinsic networks,” NeuroImage, vol. 96, pp. 245–260, Aug. 2014.

[34] C. Carlsson and R. Fullér, “On possibilistic mean value and variance of fuzzy numbers,” Fuzzy Sets Syst., vol. 122, no. 2, pp. 315–326, 2001.

[35] A. Irion, “Fuzzy rules and fuzzy functions: A combination of logic and arithmetic operations for fuzzy numbers,” Fuzzy Sets Syst., vol. 99, no. 1, pp. 49–56, 1998.

[36] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learn- ing applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.

[37] C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics). New York, NY, USA: Springer-Verlag, 2006.

[38] K. Lang, “NewsWeeder: Learning to filter netnews,” in Proc. 12th Int. Conf. Mach. Learn., 1995, pp. 331–339.

[39] S. Roweis. Data for MATLAB Hackers. Accessed: Dec. 2008. [Online]. Available: https://cs.nyu.edu/∼roweis/data.html

[40] F. S. Samaria and A. C. Harter, “Parameterisation of a stochastic model for human face identification,” in Proc. 2nd IEEE Workshop Appl. Comput. Vis., 1994, pp. 138–142.

[41] D. Cai, X. He, J. Han, and H.-J. Zhang, “Orthogonal Laplacianfaces for face recognition,” IEEE Trans. Image Process., vol. 15, no. 11, pp. 3608–3614, Nov. 2006.

[42] P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 711–720, Jul. 1997.

C. L. Philip Chen (S’88–M’88–SM’94–F’07) received the M.S. degree in electrical engineering from the University of Michigan, Ann Arbor, MI, USA, in 1985 and the Ph.D. degree in electrical engi- neering from Purdue University, West Lafayette, IN, USA, in 1988.

He is a Chair Professor with the Department of Computer and Information Science, Faculty of Science and Technology, University of Macau, Macau, China. Being a Program Evaluator of the Accreditation Board of Engineering and Technology

Education in the U.S., for computer engineering, electrical engineering, and software engineering programs, he successfully architects the University of Macau’s Engineering and Computer Science programs receiving accreditations from Washington/Seoul Accord through Hong Kong Institute of Engineers (HKIE), of which is considered as his utmost contribution in engineer- ing/computer science education for Macau as the former Dean of the Faculty. His current research interests include systems, cybernetics, and computational intelligence.

Dr. Chen was a recipient of the 2016 Outstanding Electrical and Computer Engineers Award from his alma mater, Purdue University, after he grad- uated from the University of Michigan at Ann Arbor, Ann Arbor, MI, USA. He was the IEEE SMC Society President from 2012 to 2013 and is currently a Vice President of Chinese Association of Automation (CAA). He is the Editor-in-Chief of the IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS: SYSTEMS and an Associate Editor of the IEEE TRANSACTIONS ON FUZZY SYSTEMS and IEEE TRANSACTIONS ON CYBERNETICS. He was the Chair of TC 9.1 Economic and Business Systems of International Federation of Automatic Control from 2015 to 2017. He is a fellow of AAAS, IAPR, CAA, and HKIE.

Shuang Feng received the B.S. degree in mathemat- ics and the M.S. degree in applied mathematics from Beijing Normal University, Beijing, China, in 2005 and 2008, respectively. He is currently pursuing the Ph.D. degree in computer science with the Faculty of Science and Technology, University of Macau, Macau, China.

He is currently an Associate Professor with the School of Applied Mathematics, Beijing Normal University, Zhuhai, China. His current research interests include fuzzy systems and fuzzy neural

networks and their applications in computational intelligence.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:31:46 UTC from IEEE Xplore. Restrictions apply.