helpfn

504 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

An Overview of Unsupervised Deep Feature Representation for Text Categorization

Shiping Wang , Jinyu Cai, Qihao Lin, and Wenzhong Guo

Abstract— High-dimensional features are extensively accessible in machine learning and computer vision areas. How to learn an efficient feature representation for specific learning tasks is invariably a crucial issue. Due to the absence of class label information, unsupervised feature representation is exceedingly challenging. In the last decade, deep learning has captured growing attention from researchers in a broad range of areas. Most of the deep learning methods are supervised, which is required to be fed with a large amount of accurately labeled data points. Nevertheless, acquiring sufficient accurately labeled data is unaffordable in numerous real-world applications, which is suggestive of the needs of unsupervised learning. Toward this end, quite a few unsupervised feature representation approaches based on deep learning have been proposed in recent years. In this paper, we attempt to provide a comprehensive overview of unsu- pervised deep learning methods and compare their performances in text categorization. Our survey starts with the autoencoder and its representative variants, including sparse autoencoder, stacked autoencoder, contractive autoencoder, denoising autoen- coder, variational autoencoder, graph autoencoder, convolutional autoencoder, adversarial autoencoder, and residual autoencoder. Aside from autoencoders, deconvolutional networks, restricted Boltzmann machines, and deep belief nets are introduced. Then, the reviewed unsupervised feature representation methods are compared in terms of text clustering. Extensive experiments in eight publicly available data sets of text documents are conducted to provide a fair test bed for the compared methods.

Index Terms— Autoencoder, deconvolutional network, deep belief nets, deep learning, feature representation, text catego- rization, unsupervised learning.

I. INTRODUCTION

WITH the development of Internet and multimedia tech-nology, the amount of data comes with a rapid and steady growth, which contains a great deal of redundant, irrelevant, and inconsistent data. These undesired data may largely deteriorate the performance of specific learning tasks. Consequently, how to learn an efficient feature representation

Manuscript received November 28, 2018; revised February 26, 2019; accepted April 1, 2019. Date of publication April 26, 2019; date of current version June 10, 2019. This work was supported in part by the National Natural Science Foundation of China under Grant U1705262 and Grant 61672159, in part by the Technology Innovation Platform Project of Fujian Province under Grant 2014H2005 and Grant 2009J1007, in part by the Fujian Collaborative Innovation Center for Big Data Application in Governments, and in part by the Fujian Engineering Research Center of Big Data Analysis and Processing. (Corresponding author: Shiping Wang.)

The authors are with the College of Mathematics and Computer Science, Fuzhou University, Fuzhou 350116, China, and also with the Fujian Provincial Key Laboratory of Network Computing and Intelligent Infor- mation Processing, Fuzhou University, Fuzhou 350116, China (e-mail: [email protected]; [email protected]).

Digital Object Identifier 10.1109/TCSS.2019.2910599

from massive data is extremely critical and urgent. It is known that an effective feature representation comes with satisfac- tory effects of learning tasks [1], [2]. Feature representation can be categorized as supervised or unsupervised methods. In supervised learning, it is about finding a low-dimensional representation by a trained model with a set of known class labels [3]–[5]. In contrast, unsupervised methods attempt to construct a feature representation by the evaluation of sample similarities [6]–[8].

The concept of deep learning was put forward by Hinton et al. [9] and Hinton and Salakhutdinov [10], and it has made persistent advances in recent years [11]–[14]. It is actually a multilayered neural network to simulate and analyze the operation mechanism of human brains. In such a scheme, deep learning attempts to represent, decode, and interpret all kinds of data, including images [15]–[18], sounds [19]–[21], texts [22]–[24], and videos [25]–[28]. Like other machine learning methods, deep learning can also be categorized as supervised and unsupervised approaches. As an example, convolutional neural networks [29]–[31] are supervised deep learning models, while deep belief nets tend to work in unsupervised scenes [32]–[34].

In contrast, shallow neural networks come with a smaller number of layers (below three layers) [35], [36]. This type of networks possesses favorable performances in some specific applications, especially for the cases of insufficient training data [37], [38]. However, the limitations of shallow neural networks are the weak approximation ability and generaliza- tion capacity of complex data distributions [39]. Conversely, deep neural networks consist of a large number of hidden layers, which are able to provide well-approximated solutions to varying complex functions [40], [41]. In other words, some complex nonlinear functions are expressible by deep neural networks but cannot be well approximated by any shallow neural network with the same number of neurons [42]. From a theoretic viewpoint, the former works with the lower bound of approximation performance. Simultaneously, deep nets exhibit considerable merits in learning hierarchical semantic feature representations, such as learning parts of objects [43], [44]. The semantic features act frequently as high-level characteri- zations of visual objects [45], [46].

Nonetheless, a variety of practical applications frequently suffer from the lack of sufficient accurately labeled data. Naturally, learning beneficial patterns and feature represen- tations from this type of data is of great importance. Toward this end, there have been a number of unsupervised learning

2329-924X © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 505

paradigms, such as autoencoder networks [47]–[50], restricted Boltzmann machines (RBMs) [51]–[54], deconvolutional net- works [55]–[58], and deep belief nets [59]–[61]. These methods focus more on dimensionality reduction, aiming at providing a series of rewarding solutions to an unsupervised learning.

In this paper, we present a comprehensive overview of unsupervised deep feature representation methods for the problem of text categorization. In the first place, we begin with one widely used type of unsupervised neural networks, namely autoencoder networks, and describe their intrinsic mechanisms and distinguished invariants. Then, other typ- ical representatives of unsupervised deep feature represen- tations are introduced, including deconvolutional networks, deep belief nets, and RBMs. Finally, numerous experiments are conducted in eight publicly available data sets of text documents from machine learning repositories. Extensively experimental results demonstrate the respective benefits of varying unsupervised deep feature representation methods. It is expected that this paper would provide some enlightenments and insights for who are fresh in this area.

The remaining part of this paper is arranged as fol- lows. A number of unsupervised deep feature representation approaches are presented in Section II. And the extensive com- parative experiments of the reviewed methods are provided in Section III. Finally, this paper is concluded with remarking work in Section IV.

II. OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION

To begin with, some frequently used notations are revisited in this paper. For an unsupervised learning task, the set of data points is denoted by {xi}ni=1, where xi ∈ Rd is a d -dimensional column vector. The input data matrix is assumed to be X = [x1, . . . , xn] ∈ Rd×n . For an arbitrary matrix M = [Mi j]m×n , its Frobenius norm (simply called F -norm) is defined as ||M||F =

√∑n j=1

∑m i=1 M

2 i j , and its

L 1-norm is denoted as ||M||1 = ∑n

j=1 ∑m

i=1 |Mi j |. This section puts more emphasis on four categories of unsu-

pervised deep feature representations, including autoencoders- like networks, deconvolutional networks, RBMs, and deep belief nets.

A. Autoencoder Network and its Variants

Autoencoder neural network is a well-known unsupervised feature representation method that learns low-dimensional hidden variables with the minimum reconstruction error to the input matrix. In order to capture the certain geometrical struc- tures of the input data, a number of variations of autoencoder have emerged.

1) Autoencoder: An autoencoder neural network can be divided into two symmetric steps: encoder and decoder. The former aims to learn a low-dimensional feature representation of the input data, whereas the latter is to recover the data with the minimum reconstruction error. An encoder can be regarded as a feedforward bottom-up step, while a decoder can be viewed as a feedback top-down generative step. An illustration

Fig. 1. Framework of autoencoder neural network.

of autoencoder networks is demonstrated in Fig. 1. From Fig. 1, it is observed that each data point x = [x (1); · · · ;x (d)] in the input layer is encoded as y = [y(1); · · · ; y(r)] with r � d in the hidden layer, which is then decoded as x̂ = [̂x (1); · · · ; x̂ (d)] in the output layer.

The output x̂ is actually an estimator of the input sample x, and loss(x, x̂) serves as the performance loss function. Specif- ically, the hidden layer y represents the activated neurons by a certain linear transformation of the input layer x. The transformation matrix and bias from the input layer to the hidden layer are denoted by W(1) ∈ Rd×r and b(1) ∈ Rr , and those from the hidden layer to the output layer are denoted by W(2) ∈ Rr×d and b(2) ∈ Rd . With the aforementioned notation, hidden units y can be computed by the following canonical form:

y = f (x) = hW(1),b(1) (x) = σ (W(1) T

x + b(1)) (1) where σ is a predefined activation function, typically a logistic sigmoid function σ (z) = (1/1 + e−z) for any z ∈ R. Analogously, the output layer can be represented by

x̂ = g( f (x)) = hW(2),b(2) (y) = σ (W(2) T

y + b(2)) = σ (W(2)T σ (W(1)T x + b(1)) + b(2)) (2)

in which the activation functions in encoding and decoding modes are assumed to be the same for model description simplicity. The mappings f : Rd → Rr and g : Rr → Rd are called the encoder and the decoder, respectively.

Without loss of generality, it is assumed that the encoder and the decoder share the same weighted matrix, i.e., W(1) = W(2)

T = W and b = [b f ;bg] with b(1) = b f and b(2) = bg . The optimization objective function of an autoencoder neural network is defined as

min W,b

J (W, b) = n∑

i=1 �(xi , g( f (xi ))) =

n∑

i=1 �(xi , x̂i ) (3)

where �(·, ·) is a loss function to measure the mutual difference between two variables and x̂i = g( f (xi )) = σ (WT σ (WT xi + b f )+ bg). For simplicity, �(·, ·) is specified as the Euclidean distance. Therefore, the above-mentioned optimization objec- tive is written as

min W,b

J (W, b) = n∑

i=1 ||xi − g( f (xi ))||22. (4)

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

506 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

Algorithm 1 Algorithm for Autoencoder Neural Network Input: Data points {xi}ni=1, the number of hidden units r , and learning rate α. Output: Shared weighted matrix W and bias vectors b f , bg , denoted by b = [b f ;bg]. 1: Initialize W and b; 2: Randomly generate mini-batches {Xi}ki=1 with Xi ∈ Rd×ni

and n = ∑kj=1 n j , forming a partition of X; 3: repeat 4: for each epoch (i = 1, . . . , k) do 5: Update W with W ← W − α ∂

∂W J (W, b); 6: Update b with b ← b − α ∂

∂b J (W, b); 7: end for 8: until convergence 9: return W and b.

Assuming that X̂ = [̂x1, . . . , x̂n], the optimization problem of autoencoder neural networks is then rewritten as

min W,b

J (W, b) = ||X − g( f (X))||2F = ||X − X̂||2F (5)

with g( f (X)) = X̂ = σ (WT σ (WT X + b f ) + bg). The aforementioned matrix characterization for the autoen-

coder network provides a batched optimization technique. The above-mentioned optimization problem can be solved by the mini-batch gradient descent method, as demonstrated in Algorithm 1.

2) Sparse Autoencoder: For certain learning tasks, most neurons of an autoencoder neural network are not activated. Assuming that a weighted matrix W and two bias vectors {b(l)}2l=1 will be learned, there are d × r + d + r parameters to be solved, where the parameter number may be greater than the known sample number, being indicative of the high risk of overfitting. Besides, the simplest form of autoencoders may lead to weight-decay, suggesting that the corresponding neurons are optimized by small weights. Sparse autoencoder neural network is based on the assumption that only a small number of neurons are activated [62], [63] when addressing specific learning tasks. It requires that both weighted matri- ces and output hidden units are constrained to be sparse. Therefore, the optimization problem of the sparse autoencoder neural network is formulated as

min W,b

J (W, b) + λ||W||1 + μ r∑

j=1 K L(ρ|ρ̂ j ) (6)

where λ and μ are the two regularized coefficients to bal- ance the fitting term and the sparsity term, ρ is a prede- fined sparsity parameter, and K L(ρ|ρ̂ j ) = ρlog((ρ/ρ̂ j )) + (1 − ρ)log((1 − ρ/1 − ρ̂ j )) is the Kullback–Leibler (KL)- divergence to guarantee the sparsity of the output hidden variables. Herein

ρ̂ j = 1

n

n∑

i=1 σ ( wTj X + b(1)j

) xi (7)

with W =[w1, . . . , wr] ∈ Rd×r .

Fig. 2. Illustration of stacked autoencoder neural network wherein only two hidden layers are demonstrated.

3) Stacked Autoencoder: The aforementioned autoencoder neural network indicates that a low-dimensional representation Y ∈ Rr×n is available for a given data matrix X ∈ Rd×n . However, the autoencoder network just contains one hid- den layer, which implies weak abilities to learn a nonlinear feature representation, though nonlinear activation function is employed in each layer. Naturally, adding more hidden layers is a prior choice to learn nonlinear high-level features. A stacked autoencoder [64], [65] is to construct a network with multiple hidden layers based on greedy strategies. The stacked autoencoder network with m hidden layers can be simply regarded as a combination of m autoencoders. It can also be viewed as m pairs of encoder and decoder, as shown in Fig. 2.

In the hidden layers, the encoding parameter from the (l−1)th layer to the lth layer is denoted by (W(l), b(l)f ), and its corresponding decoding parameter is written as (W(l)

T , b(l)g ).

Hence, the hyperparameter space of a stacked autoencoder can be denoted as {W(l), b(l)}ml=1 with b(l) = [b(l)f ;b(l)g ]. With an input data matrix X ∈ Rd×n , the mth hidden layer to be solved can be represented as

Y = fm ( fm−1 · · ·( f1(X))) = σ (W(m)T · · ·σ (W(1)T X + b(1)f

)+ b(m)f )

(8)

and the reconstructed data matrix X̂ is expressed as

X̂ = g1(g2 · · ·(gm(Y))) = σ (W(1)T · · ·σ (W(m)T Y + b(m)g

)+ b(1)g ) . (9)

Therefore, the overall cost loss of a stacked autoencoder neural network is J (W, b) = ||X − g1(g2 · · ·(gm ( fm ( fm−1 · · ·( f1(X)))))||2F , where W � {W(l)}ml=1 and b � {b(l)}ml=1. With a layerwise greedy strategy, the optimization objective function of a stacked autoencoder can be represented as

min W,b

J (W, b) = m∑

l=1 ||Ol−1 − gl ( fl (Ol−1))||2F (10)

where O0 is the input layer, i.e., O0 = X, and Ol = fl (Ol−1) is the lth layer output for any l ∈ {1, . . . , m}.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 507

Fig. 3. Architecture of denoising autoencoder.

4) Contractive Autoencoder: Beyond sparsity and multilay- ers, the robustness of the hidden variables is an indispensable point for a neural network. Toward this end, a measure was proposed to encourage the robustness of the learned low-dimensional feature representation f (x). This measure is represented as the Frobenius norm of the Jacobian matrix J f (x) of a nonlinear mapping f [66], defined by

||J f (x)||2F = ∑

i, j

( ∂h j (x)

∂xi

)2 (11)

where h j (x) is the j th element of the low-dimensional hidden representation h(x) encoded by f . Particularly, the Jacobian matrix can be represented by

J f (x) =

⎡ ⎢⎢⎢⎢⎣

∂h1(x) ∂x1

· · · ∂hr (x) ∂x1

... ...

∂h1(x) ∂xn

· · · ∂hr (x) ∂xn

⎤ ⎥⎥⎥⎥⎦

(12)

where r is the number of hidden units. Adding a regulariza- tion term associated with the Jacobian matrix of nonlinear mappings, a contractive autoencoder neural network [67] is formalized as

min W,b

J (W, b) = n∑

i=1 �(xi , g( f (xi ))) + λ||J f (xi )||2F (13)

where λ is a regularization parameter. By a batched matrix for- mulation, the aforementioned optimization objective problem is equivalently transformed into

min W,b

J (W, b) = �(X, g( f (X))) + λ||J f (X)||2F (14)

in which J f (X) ∈ Rn×n×r is a 3-D tensor of which the i th entry is equal to J f (xi ) for any i ∈ {1, . . . , n}.

5) Denoising Autoencoder: Deep neural networks tend to rely largely on accurately labeled input data, which implies that these networks frequently fail to learn effective dis- criminative features from partially destroyed data. Denoising autoencoder neural network [68], [69] aims at reconstructing clean data from given noisy data.

The architecture of denoising autoencoders is shown in Fig. 3. It is observed from Fig. 3 that a clean input data point x is preprocessed as corrupted x by adding an amount of noises obeying certain probability distribution, such as binomial noises or Gaussian noises [70]. Each corrupted data point x is mapped onto a low-dimensional hidden represen- tation y = hW,b(x) = σ (WT x + b f ) using an autoencoder.

Fig. 4. Illustration of variational autoencoder neural networks.

Using hidden units {yi}ni=1, we reconstruct the original input data points {xi}ni=1, instead of the corrupted data points {xi}ni=1. Therefore, the optimization objective function of a denoising autoencoder neural network can be represented as

min W,b

J (W, b) = �(X, X̂) = �(X, g( f (X))) (15)

where X̂ = σ (WT σ (WT X+b f ))+bg with X = [x1, . . . , xn]. Since x is assumed to be a random variable by adding noise to x, �(·, ·) tends to be defined as a probability loss function, such as cross entropy.

6) Variational Autoencoder: Given any high-dimensional input data point, autoencoders learn a low-dimensional hid- den representation with minimum reconstruction error. It is observed that the learned hidden variable is closely related to the input data matrix, indicating that it fails to gener- ate an arbitrary sample. Hence, the hidden layer is further constrained with certain conditions and regularizations, such that it can produce any sample at a relatively independent manner. Toward this end, a variational autoencoder neural network [71], [72] provides a probabilistic view to learn regularized hidden units.

Given an input data set {xi}ni=1 of n independent identically distributed (i.i.d) samples from a random variable x, it is assumed that the given data points are generated by a given random process with an unobserved hidden random variable y. The variational autoencoder aims to learn a hidden random variable (also called a latent variable) y, as shown in Fig. 4. It is observed from Fig. 4 that variational autoencoders consist of two steps, namely a recognition model (a probabilistic encoder) and a generative model (a probabilistic decoder). The former serves as an encoder to learn the hidden variable y with conditional probability qφ(y|x), while the latter acts as a decoder to produce x̂ with a generative model pθ (̂x|y) with minimum reconstruction loss to x. Herein, parameters φ and θ need to be specified into certain distribution forms. When specifying the distribution (e.g., Gaussian distribution) that pθ (̂x|y) obeys and the estimated parameter θ , pθ (·) can be used to generate any adversarial sample x̂.

Given an input random variable x, variational autoencoders aim at learning a hidden random variable y, satisfying the following two conditions: 1) the reconstruction loss �(·, ·) should be minimized and 2) the hidden variable y should obey a given distribution. According to Bayes’ formula, the true

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

508 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

posterior density can be represented as

pθ (y|x) = pθ (x|y) pθ (y)

pθ (x) . (16)

Naturally, the conditional distribution qφ(y|x) should be the best estimation of the true posterior distribution pθ (x|y). The KL-divergence is employed to measure the difference between both distributions. Using maximum likelihood estimation, the marginal likelihood is expressed as

log pθ (x1, . . . , xn ) = n∑

i=1 log pθ (xi ). (17)

Here, the marginal likelihood of each data point can be written as

log pθ (xi ) = DK L (qφ(y|xi )||pθ (y|xi )) + �(φ, θ;xi ) (18) where �(φ, θ;xi ) is a cost loss to be minimized, also called the variational lower bound of on the marginal likelihood of data point xi . Therefore, the optimization objective function of variational autoencoders is formulated as

min φ,θ −DK L (qφ(y|xi )||pθ (y)) + Eqφ (y|xi )[log pθ (xi|y)] (19)

where φ and θ to be optimized are called the variational and generative parameters, respectively. Frequently, we suppose that the hidden variable is a centered isotropic multivariate Gaussian, i.e., pθ (y) = N (y;0, I), and pθ (x|y) is a multivari- ate Gaussian for real-value data or Bernoulli for binary data. Simultaneously, the variational posterior is assumed to be a multivariate Gaussian with a diagonal covariance matrix, that is

qφ(y|xi ) = N ( y;μi , σ 2i I

) . (20)

With the aforementioned parameterization trick, a variational autoencoder neural network is trained, and the corresponding hidden variables are computed.

7) Graph Autoencoder: Compared with other autoencoder variants, graph autoencoder includes both the data matrix X ∈ Rd×n and the graph adjacency matrix A ∈ Rn×n as an input pair [73], [74]. By encoder function f (◦), the input two tuples (X, A) are mapped into a low-dimensional embedding Y ∈ Rk×n with k � d , while a pairwise decoder model g(◦) recovers A with the minimum reconstruction error [75].

In this section, we introduce a local invariant deep nonlinear mapping algorithm, namely a graph regularized autoencoder. With an input data matrix X ∈ Rd×n , this algorithm is to search a feature representation Y ∈ Rk×n with an encoder function f (◦), i.e., Y = fW,b(X, A), in which k is the number of reduced dimensions and θ is a parameter to be solved. Naturally, there is a decoder function g(◦) to project the hidden units onto a reconstructed feature space, X̂ = g(Y).

From a perspective of manifold learning, the local structure of the input feature space is preserved in the embedded low- dimensional representation. Accordingly, the graph regularized autoencoder aims at finding a best low-dimensional embedding in the hidden layer. Its optimization objective problem is defined as the form of

arg min W,b ||X − g( f (X))||2F + λtr (YLAYT ) (21)

Algorithm 2 Algorithm for Graph Regularized Autoencoders Require: The original data X, adjacency matrix A, the number

l of layers, and the numbers of units in the hidden layers. Ensure: Weighted matrix W and bias vector b of each layer. 1: for i ∈ {1, . . . , n} do 2: Update W and b by solving Problem (22); 3: Obtain the i -th layer of data representation Yi ; 4: end for 5: return W and b.

where λ > 0 is suggestive of a regularization parameter of the training algorithm and LA is a regularizer of structured learning. In many circumstances, LA is specified as a graph Laplacian, i.e., LA = D − A, where D is a diagonal matrix with Dii =

∑n j=1 Ai j .

As training all layers simultaneously in multiple-layer autoencoders may be stacked, graph regularized autoencoders train the multilayered neural networks layer-by-layer. It is worth pointing out that the above-mentioned optimization problem can be rewritten as

arg min W,b

n∑

i=1 ||xi − g( f (xi ))||22 + λ

n∑

i=1 tr (yi LAy

T i ) (22)

where Y = [y1, . . . , yn]. With the aforementioned analyses, the procedure for graph

regularized autoencoders is summarized in Algorithm II. 8) Convolutional Autoencoder: A convolutional autoen-

coder is also comprised of encoding and decoding. Compared with other invariants of autoencoder neural networks, it is dis- tinctive in feature generation using convolutional operations, instead of nonlinear activation functions of linear transforma- tions.

For encoding, a convolution neural network architec- ture [76] is used to produce feature maps, which is widely used in image data processing. It is assumed to contain L layers where the former L − 1 layers are convolutional, and the Lth layer is fully connected. Here, the hidden feature representation in the lth layer is denoted by Yl for l ∈ {1, . . . , L}. Suppose that the lth layer consists of pl feature filters for any l ∈ {1, . . . , L}. For example, as to the i th filter in the first layer, a convolutional operation with stride length r produces a latent feature map

yi,1 = σ (X ∗ Wi,1 + bi,1f )

(23)

where σ (·) is a specific nonlinear activation function and ∗ denotes the convolutional operator. In real-world applica- tions, σ (·) is frequently represented by a rectified linear unit (ReLU) [77]. Concatenating all feature maps generated by all filters, the set of all feature maps in the lth layer is denoted Y(l) = [y(1,l), . . . , y( pl ,l)]. For simplicity, we denote Wl = [W1,l , . . . , W pl ,l] and blf = [b1,lf , . . . , b

pl ,l f ]. Hence,

the aforementioned latent feature maps are formulated as the recursive form of

Yl = σ (Yl−1 ∗ Wl + blf )

(24)

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 509

Fig. 5. Architecture of adversarial autoencoder neural network.

where Y1 = σ (X ∗ W1 + b1f ) and l ∈ {1, . . . , L}. As a result, the final hidden layer YL serves as a high-level semantic representation for learning tasks.

For decoding, the hidden low-dimensional feature represen- tation is hierarchically reconstructed to the original feature space. Consequently, the estimated data matrix is represented by

X̂ � X̂L = gL (gL−1 · · ·(g1(YL ))) (25) with the recursive formulation of

X̂l = σ (X̂l−1 ∗ Ŵl + b̂lg )

(26)

in which X̂1 = σ (ŶL ∗ Ŵ1 + b̂1g ) and l ∈ {1, . . . , L}. With the formulated architecture, convolutional autoencoders are represented as the following optimization objective function:

arg min W,b

1

2 ||X − X̂||2F (27)

where a unified framework is provided for all variants of autoencoders.

9) Adversarial Autoencoder: Adversarial autoencoders use an “adversarial" mechanism to learn a low-dimensional rep- resentation. It is comprised of two main models, namely the generation model and the discriminative model. Frequently, it can be combined with generative adversarial networks [78], [79] and variational autoencoders. The structure of adversarial autoencoder networks [80] is shown in Fig. 5.

Let x be an input data point and y be a hidden unit of an adversarial autoencoder. Let p(y) be an arbitrary prior distribution, p(x|y) be the decoding distribution, and q(y|x) be the encoding distribution. Meanwhile, we let p(x) be the model distribution and pd (x) be the data distribution. By q(y|x), we can define q(y) on a hidden encoding vector as

q(y) = ∫

x q(y|x) pd(x)d x. (28)

Actually, q(y) can be regarded as an aggregated posterior dis- tribution on hidden units. An adversarial autoencoder matches q(y) to p(y) for regularization purposes as well as adversarial sample generators. In addition, the generator of adversarial

network is the encoder q(y|x), which guarantees that the aggregated posterior distribution could confuse the discrim- inative network between q(y) and p(y).

From the perspective of optimization algorithms, an adver- sarial autoencoder could be divided into two phases in each mini-batch training process, solved by stochastic gradient descent (SGD). Both the phases are regarded as the reconstruc- tion and the regularization, respectively. In the reconstruction phase, adversarial autoencoder attempts to minimize the recon- struction error of inputs by updating the encoding parameters φ and θ , defined by

arg min θ,φ ‖x − x̂‖22. (29)

There are two tasks in the regularization phase. On one hand, the adversarial network distinguishes the true samples gener- ated by the prior knowledge from the generated samples in the hidden layer through updating its discriminative network. On the other hand, it updates its generator to confuse the discriminative network, formulated as

min G

max D

Ex∼pdata[log D(x)]+Ey∼p(y)[log(1−D(G(y)))] (30) where G(◦) is the generator and D(◦) is the discriminator, respectively. Apart from this, D(x) is a neural network to calculate the possibility that a variable x from the data space is a sampling from positive samples that we attempt to model instead of negative samples. Meanwhile, G(y) is a mapping to project sample y in the prior p(y) to the data space. The task of G(y) is to infer the discriminative network into a trusted mechanism, such that it generates positive samples as much as possible in the training process. And the training is realized by updating the gradients of D(x) with regard to x.

Concerning the selection of the encoding q(y|x), adversarial autoencoders have several beneficial choices, such as deter- ministic, Gaussian posterior [81], and universal approximator posterior. This type of mechanism can generate rewarding models when addressing specific practical scenarios. All in all, an adversarial autoencoder aims at integrating the idea of confrontation into autoencoders, and its merit is the usage of modeling discrete true samples using a continuous infinite hidden variable. In other words, sufficient training data are generated by adversarial autoencoders, which is effective in filling the gap that generative adversarial networks could not produce discrete samples.

10) Residual Autoencoder: He et al. [82] proposed a resid- ual learning framework for object recognition. The aim of this method is not to learn unreferenced functions, instead of residual functions about the layer inputs. Due to the multiple downsampling operations, deep residual neural networks may miss some beneficial details. Nevertheless, this type of neural networks can ease the training processing and is effective in exploiting substantially deeper networks. Let H(◦) be the underlying function to map the original data point x into the reconstructed unit x̃ = H(x) with the same dimension. It is based on the assumption that any complicated nonlinear functions could be asymptotically approximated by multi- layered neural networks with enough layers. Naturally, it is hypothesized that the residual function x � H(x)−x can be

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

510 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

well approximated. Residual networks are to explicitly learn a surrogate of the residual function x, rather than the mapping H(x). Combining the weighted matrices {Wi}li=1 to be learned with the layer number l, the residual function is then defined as (x;{Wi}li=1). Therefore, the output vector of deep residual neural networks is computed by y = x̃ = (x;{Wi}li=1) + x, where

( x;{Wi}li=1

) = WTl σ ( · · ·WT2 σ

( WT1 x

)) . (31)

It is noted that the input feature vector x is assumed to have the same dimension with the output (x;{Wi}li=1), which is not necessarily true in varying practical scenarios. Toward this end, a linear projection Ws by the shortcut connections is performed to uniform the dimensions, with the output defined by

y = (x;{Wi}li=1 )+ WTs x. (32)

In a supervised learning, the given class label can serve as an estimated output y = x̃, which guides the searching directions of the optimization algorithms. Nonetheless, residual networks are tough to address unsupervised learning tasks due to the absence of class label information.

Residual autoencoders [83] aim at learning an efficient low-dimensional representation in an unsupervised manner. The residual function of an unsupervised task is defined as x = x − x̃, where x̃ is an estimate of the input feature vector x. Therefore, the loss function of residual autoencoders is represented as the canonical form of

L =‖ x − x̂‖22 =‖(x − x̃) − θW,b(x̃)‖22 (33) where x̂ = θW,b(x̃) is the estimated output that residual autoencoders produce. The principle of residual autoencoders is to make x̂ as close to the expected output as possible.

B. Deconvolutional Network

Deconvolutional network [84] is a framework that allows an unsupervised construction of hierarchical image represen- tations. These feature representations are available to provide effective features for object recognition. Each hierarchical layer collects useful information to yield more complex rep- resentations for midlevel and high-level feature learning [58]. Taking image data as an example, it can automatically extract rich features that correspond to midlevel features, such as edge junctions, parallel lines, and curves and high-level features, such as rectangles and semantics. Compared with the convo- lutional autoencoders, each layer in deconvolutional networks is top-down. It manages to reconstruct the input data by a sum over convolutions of the feature maps with the learned filters.

The architecture of deconvolutional neural networks is shown in Fig. 6. Each deconvolutional network consists of pairs of convolution and deconvolution operations. The former is to generate features using convolution kernels, while the latter aims to reconstruct the input data using feature filters. Naturally, these two operations correspond to the encoder and the decoder of an autoencoder neural network, respectively.

As an example of single deconvolutional network, let the hidden variable be yi consisting of c channels yi1, . . . , y

i c.

Fig. 6. Illustration of convolution and deconvolution.

In this network, each channel is represented as a linear sum of latent feature maps zij convolved with filters W j,k , defined by

yik = c′∑

j=1 zij ∗ W j,k (34)

where k ∈ {1, . . . , c} and c′ is the number of latent feature maps. It is worth noting that c′ is jointly determined by the input image size and filter size. For instance, there are (W − h +1)×(H −h +1) feature maps for an image of resolution W ×H and a filter of h×h size. Accordingly, the optimization problem is represented by

arg min yi

c∑

k=1

∥∥∥∥∥∥ yik −

c′∑

j=1 zij ∗ W j,k

∥∥∥∥∥∥

2

2

+ λ c′∑

j=1

∣∣zij ∣∣p (35)

where λ is a regularization parameter to keep a tradeoff between the first fitting accuracy (ACC) and the second regularization term. Besides, this regularization term is added to provide a well-defined solution to the underdetermined optimization problem. Then, the deconvolutional networks assume that Gaussian noises are imposed on the reconstruction term [85] and encourage certain properties of the learned fea- ture maps by p-norm. As an example, λ balances the relative contributions of the reconstruction of yi and the sparsity of the feature maps zij when fixing p = 1. Finally, the model of deconvolutional networks is top-down. It is suggested from this network that an image is well restored when the latent feature maps are available. Unlike sparse autoencoders or deep belief nets, there is a tailored mechanism for feature map generation, whose effectiveness has been validated in varying practical applications.

C. Restricted Boltzmann Machine

A Boltzmann machine is a kind of stochastic recur- rent neural network constructed by Ackley and Hinton [86] in 1985. The Boltzmann machine is named from the Boltz- mann distribution in statistical mechanics. RBM is regarded as a special topological structure of the Boltzmann machine, proposed by Hinton et al. [87], [88] in 1986. The term “restricted" refers to the model constrained with the bipartite graph, which is indicative of the disconnectivity of neurons in the same layer. Therefore, RBMs consume less computational resources than traditional ones [89].

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 511

Fig. 7. Framework of RBM.

RBM has a visible layer in the input layer and a hidden layer for feature representations [90], [91], and its architecture is shown in Fig. 7. It is comprised of visible units and hidden units, where the former is the input data points and the latter corresponds to feature representations. Actually, RBMs come with binary-valued visible and hidden variables, aiming at modeling a probability distribution of the input data points.

Let xi and y j be the binary states of visible neuron i and hidden neuron j . An energy function E(x, y) of the visible and hidden units (x, y) is formulated as

E(x, y) =− ∑

i

ai xi − ∑

j

b j y j − ∑

i

∑

j

xi y j Wi j (36)

where ai and b j are the biases and Wi j is the weighted parameter between xi and y j . With the form of matrix representation, the aforementioned energy function is written as

E(x, y) =−aT x − bT y − yT Wx. (37) Alternatively, there is another energy function E(x, y) to assign a probability to the pair (x, y) of visible and hidden vectors, defined by

p(x, y) = 1 Z

e−E(x,y) (38)

where Z is the partition function that is formalized by sum- ming over all possible pairs of visible and hidden vectors, that is

Z = ∑

x

∑ y

e−E(x,y). (39)

Besides, the marginal probability p(x) of p(x, y) with respect to visible variable x is defined as

p(x) = 1 Z

∑ y

e−E(x,y). (40)

Since RBMs are structured with bipartite graphs, which is suggestive of no intralayer connection in the visible or hidden layers. Therefore, the conditional probabilities p(x|y) and p(y|x) are written as

p(x|y) = ∏

i

p(xi|y) and p(y|x) = ∏

i

p(yi|x). (41)

In the training process, RBMs are to search the optimal W value while maximizing the product of probabilities of a given training set X, that is

arg max W

∏

x∈X p(x) (42)

which is equivalently written as

arg max W

E

[∑

x∈X log p(x)

] . (43)

A widely used optimization algorithm for the above- mentioned problem is Gibbs sampling-based gradient descent method [92]. It is observed that E(x, y) is a quadratic form, and hence, the derivative of the log probability of a training vector with regard to a weight is computed by

∂ log p(x) ∂Wi j

=〈xi y j〉data −〈xi y j〉model (44)

where 〈xi y j〉data and 〈xi y j〉model represent the expectations under the distribution defined by the subscript that follows. The updated weights using stochastic descent method is solved by

Wi j = α(〈xi y j〉data −〈xi y j〉model) (45) where α > 0 is the learning rate. It is noted that for any visible unit x, we have

p(xi = 1|y) = σ ⎛ ⎝ai +

∑

j

y j Wi j

⎞ ⎠ (46)

where σ (◦) is the logistic sigmoid function. Analogously, for any hidden unit y, we know

p(y j = 1|x) = σ (

b j + ∑

i

xi Wi j

) . (47)

It is suggested that 〈xi y j〉data is easy to compute using the aforementioned formulations. Unfortunately, it is tough to obtain an unbiased sample of 〈xi y j〉model, which would consume a large amount of computational cost if performing alternating Gibbs sampling by starting at any stochastic state of the visible units. Toward this end, Hinton [93] proposed a faster optimization method in 2002, replacing 〈xi y j〉model with 〈xi y j〉recon, indicating that the optimization problem is rewritten as

Wi j = α(〈xi y j〉data −〈xi y j〉recon). (48) Overall, RBMs have been capturing growing attention in

recent years. Nevertheless, how to improve the training effi- ciency is still an open issue.

D. Deep Belief Nets

As a type representative of probabilistic generative model, deep belief nets were proposed by Hinton et al. [9] in 2006. This network consists of multiple layers of stochastic and latent units, where the latent variables are frequently binary- valued. The top two layers are connected with an undirected

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

512 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

Fig. 8. Illustration of the architecture of deep belief nets.

and symmetric graph, constructing an associative memory. The architecture of deep belief nets is demonstrated in Fig. 8.

It is observed from Fig. 8 that deep belief nets are treated as one type of RBMs. The crucial procedure behind deep belief nets is to optimize W with the learned conditional distribution p(x|y, W) and prior distribution p(y|W) from an RBM. Therefore, the probability of generating a visible unit x is formalized as

p(x) = ∑

y

p(y|W) p(x|y, W). (49)

While fixing p(x|y, W), the weighted matrix W is updated by replacing p(y|W) with a better model of the aggregated posterior distribution over the hidden variables. Actually, the complementary prior at each layer guarantees that the posterior distribution could be factorial.

For feature representation and dimensionality reduction, deep belief nets are regarded as a nonlinear model as well. Indeed, deep belief nets act as a model that could produce complex nonlinear features in the last layer, in which these features are justified by minimal reconstruction error to the visible units.

III. EXPERIMENTS ANALYSES

Extensive experiments are conducted to provide a fair comparison for the aforementioned unsupervised deep feature representation methods. The selective experimental databases are characterized as follows.

A. Data Sets

CNAE is a collection of 1080 text documents with nine categories from Brazilian companies for free text business characterizations.1 Each document is represented as an 857-D feature vector, where each feature is the frequency of some certain word.

20Newsgroups consists of 18821 text documents, partitioned nearly evenly across 20 newsgroups, leading to 20 categories in total.2 Each document corresponds to one different topic, forming a 61 188-D feature vector with word frequencies.

1https://archive.ics.uci.edu/ml/datasets/CNAE-9 2http://qwone.com/ jason/20Newsgroups/

TABLE I

BRI EF DES CRI P TION OF THE TES TED DATA SETS

Reuters is a subset of the database Reuters21578 of text documents. This data set is preprocessed as 8293 documents in 65 categories by discarding those data with multiple cat- egory labels.3 Each document contains 18 933 distinct terms, corresponding to a feature vector.

RCV1 is cut from the distinguished RCV text document corpus, containing 9625 documents.4 Each document is rep- resented by a 29 992-D feature vector of distinct words.

TOX is a gene database of 171 genetic toxicology (muta- genicity) text documents with four categories from expert peer review of open scientific.5 Each document is represented as a 5748-D feature vector.

HAR is a database from the recordings of 30 subjects performing activities of daily living.6 The data set is composed of 10 299 documents in six categories, where each document is extracted as a 561-D feature vector.

TDT2 is a subset of NIST topic detection and tracking corpus. In this subset, those documents occurring two or more categories were eliminated, and only the top 30 categories were preserved, leaving totally 9394 documents of 36 771-D features.7

DBWorld is a data collection of 64 e-mails manually pro- duced from DBWorld mailing list with two categories. Each document is represented as a 4702-D feature vector, in which each feature value corresponds to a precise word or stem in the entire data set vocabulary.8

With the aforementioned descriptions, a brief characteriza- tion to all tested data sets is summarized in Table I.

B. Parameter Settings

For all compared feature representation algorithms, the learning rate is fixed as α = 0.1. As to those multilayered neural networks, the numbers of layers are set as 3, and the numbers of hidden variables in the first and second layers are tuned as 400 and 500, respectively. In order to provide a fair comparison, the number of units in the last hidden layer ranges in {20, 30, . . . , 200}. Meanwhile, the numbers of epochs for all algorithms are fixed as 50, and the batch sizes are set as 100. Besides, the default settings are adopted for other algorithmic parameters.

3 http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html 4 http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html 5 https://toxnet.nlm.nih.gov/newtoxnet/genetox.htm 6 http://archive.ics.uci.edu/ml/datasets 7 http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html 8 http://archive.ics.uci.edu/ml/datasets/DBWorld+e-mails

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 513

Fig. 9. Illustration of varying unsupervised deep feature representation methods on the data set TOX with four classes, where the dimensionality-reduced samples from the same class are marked with the same color. Naturally, the better method is that the samples having the same color are clustered together, while ones having different colors are scattered.

TABLE II

CLUS TERI NG ACC (MEAN% ± S TD%) WI TH DI FFERENT UNS UP ERVISED FEATURE REP RES ENTATI ON METHODS. THE BES T RES ULTS ARE MARKED I N BOLD (THE HI GHER THE BETTER)

All compared unsupervised methods are evaluated by their clustering performances of the feature representations in the last hidden layer. In these circumstances, the K -means method is employed for the clustering performance specification. Due to initialization sensitivity, the mean and standard deviation of 20 repeated experiments are reported.

C. Evaluation Metrics

Due to the absence of class label information, the clus- tering evaluation for unsupervised learning tasks is a tough issue. Toward this end, we adopt three evaluation metrics to assess the clustering performance, including clustering

ACC, normalized mutual information (NMI), and adjusted rand index (ARI). Denote the set of data points as {xi}ni=1. Suppose that {li}ni=1 is the given ground truth and {l̂i}ni=1 is the predictive clustering labels. The clustering ACC is calculated by

ACC = ∑n

i=1 δ(li , map(l̂i )) n

(50)

where map(◦) is a permutation mapping that best matches the predictive clustering labels to ground truths, and δ is a Dirac delta function, i.e., δ(li , l j ) = 1 if li = l j ; otherwise, δ(li , l j ) = 0.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

514 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

TABLE III

NMI (MEAN% ± S TD%) WI TH DI FFERENT UNS UP ERVISED FEATURE REP RES ENTATION METHODS. THE BES T RES ULTS ARE MARKED I N BOLD (THE HI GHER THE BETTER)

TABLE IV

ARI (MEAN% ± S TD%) WI TH DI FFERENT UNS UP ERVI SED FEATURE REP RES ENTATI ON METHODS. THE BES T RES ULTS ARE MARKED I N BOLD (THE HI GHER THE BETTER)

Given two random variables P and Q, their NMI is com- puted by

NMI(P, Q) = I (P;Q)√ H (P)H (Q)

(51)

where I (◦;◦) is the mutual information and H (◦) is the information entropy. Let the predictive clustering result be C̃ = {C̃i }̃ci=1 and the ground truth be C = {C j}cj=1. Herein, NMI can be defined by

NMI = ∑c̃

i=1 ∑c

j=1 |C̃i ∩ C j|log n|C̃i∩C j | |C̃i||C j |√(∑c̃

i=1 |C̃i|log|C̃i|n ) (∑c

j=1 |C j|log |C j |

n

). (52)

ARI is a distance metric that can measure the similarity between two data clusterings. Denote ni j = |C̃i ∩ C j|, ai =∑c

j=1 ni j , and b j = ∑c̃

i=1 ni j for any i ∈ {1, . . . , c̃} and j ∈ {1, . . . , c}. Consequently, ARI can be computed with

ARI = �i j

(ni j 2

)− [ �i

(ai 2

) � j

(b j 2

)] / (n

2

)

1 2

[ �i

(ai 2

)+ � j (b j

2

)]− [ �i

(ai 2

) � j

(b j 2

)] / (n

2

). (53)

These three evaluation metrics are positively related to the clustering performance. In other words, higher values of ACC,

NMI, and ARI are suggestive of a better clustering result as well as feature representation performance.

D. Experimental Visualizations

In this section, an experimental visualization is reported to provide an intuitive demonstration of low-dimensional repre- sentations learned by different deep neural networks. Taking the data set TOX as an example, the number of reduced dimensions is set as 2. Varying unsupervised deep feature representation methods are visualized in Fig. 9.

It is observed Fig. 9 that varying unsupervised deep feature representation approaches come with different visualization performances. The best feature representation, in theory, is a mapping with maximal between-cluster distance and mini- mum within-cluster distance simultaneously, where the circles marked with the same color can be regarded as one cluster. From this perspective, almost all tested feature representa- tion methods perform favorably because all data points are projected onto a low-dimensional space with a large mar- gin between the circles with different colors. Nonetheless, the learned low-dimensional distribution by different methods differs. As an example, the sparse autoencoder projects the original data points onto a nearly linear distribution, and so are contractive, variational, and residual autoencoders.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 515

E. Experimental Results

In this section, the compared unsupervised deep feature representation approaches are tested on eight publicly available data sets of text documents. Meanwhile, we report their respec- tive performances in terms of data clustering, as demonstrated in Tables II–IV.

From Tables II–IV, we have the following observations. In the first place, most of the unsupervised deep feature representation methods are capable of learning an effective low-dimensional representation, which is validated by the fact that the performance of the learned representation outperforms the baseline method that uses all original features for clus- tering. Then, the feature representation methods have a pos- itive influence on the improvement of learning performance. As an example, the best compared method exhibits a definite superiority over the baseline method in the data sets, such as 20Newsgroups, RCV1, TDT2, and DBWorld. Certainly, feature representation techniques may fail to improve the clus- tering performance in some specific situations. For instance, stacked autoencoders work unfavorable in the data set CNAE, and deconvolutional networks perform undesirable in the data set Reuters. The learning performance fluctuation is partly due to the fixed parameter settings of tested neural networks for a fair experimental comparison. Finally, the majority of unsupervised feature representation methods could exceed the baseline. From the perspective of structured learning, certain feature representation method attempts to learn an effective low-dimensional embedding with specific patterns. In partic- ular, sparse autoencoders favor the sparsity of the activated neurons in hidden layers, while adversarial autoencoders stress the robustness of the learned low-dimensional representation.

IV. CONCLUSION AND FUTURE WORK

In this paper, we presented an overview of a number of unsupervised deep feature representation methods. These reviewed methods included autoencoders and their vari- ants, deconvolutional networks, RBMs, and deep belief nets. In addition, extensive comparative experiments were con- ducted on eight publicly available data sets of text documents. These experiments provided a comprehensive comparison for all tested feature representation approaches, aiming at exhibiting respective benefits of varying deep feature learning methods in text clustering. It is expected that this paper would provide some insights and enlightenments for who is highly interested in unsupervised deep learning. In our future work, we will explore more efficient unsupervised deep learning methods with given data-driven scenes.

REFERENCES

[1] A. Farasat, G. Gross, R. Nagi, and A. G. Nikolaev, “Social network analysis with data fusion,” IEEE Trans. Comput. Social Syst., vol. 3, no. 2, pp. 88–99, Jun. 2016.

[2] A. Argyriou, T. Evgeniou, and M. Pontil,“Multi-task feature learning,” in Proc. 19th Int. Conf. Neural Inf. Process. Syst., Dec. 2006, pp. 41–48.

[3] G. Carneiro, A. B. Chan, P. J. Moreno, and N. Vasconcelos, “Super- vised learning of semantic classes for image annotation and retrieval,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 394–410, Mar. 2007.

[4] N. Foroutan and A. Hamzeh, “Discovering the hidden structure of a social network: A semi supervised approach,” IEEE Trans. Comput. Social Syst., vol. 4, no. 1, pp. 14–25, Mar. 2017.

[5] Y. Xiao, X. Li, H. Wang, M. Xu, and Y. Liu, “3-HBP: A three-level hidden bayesian link prediction model in social networks,” IEEE Trans. Comput. Social Syst., vol. 5, no. 2, pp. 430–443, Jun. 2018.

[6] J. G. Dy and C. E. Brodley, “Feature selection for unsupervised learning,” J. Mach. Learn. Res., vol. 5, pp. 845–889, Aug. 2004.

[7] S. Wang and W. Zhu, “Sparse graph embedding unsupervised feature selection,” IEEE Trans. Syst., Man, Cybern., Syst., vol. 48, no. 3, pp. 329–341, Mar. 2018.

[8] S. Wang and W. Guo, “Sparse multigraph embedding for multi- modal feature representation,” IEEE Trans. Multimedia, vol. 19, no. 7, pp. 1454–1466, Jul. 2017.

[9] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, Jul. 2006.

[10] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, Jul. 2006.

[11] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Netw., vol. 61, pp. 85–117, Jan. 2015.

[12] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.

[13] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in Proc. 28th Int. Conf. Mach. Learn., 2011, pp. 689–696.

[14] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in Proc. 30th Int. Conf. Mach. Learn., Jun. 2013, pp. 1139–1147.

[15] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 580–587.

[16] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real- time object detection with region proposal networks,” in Proc. Adv. Neural Inf. Process Syst., Jun. 2015, pp. 91–99.

[17] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 779–788.

[18] W. Liu et al., “Ssd: Single shot multibox detector,” in Proc. Eur. Conf. Comput. Vis., Sep. 2016, pp. 21–37.

[19] L. Deng et al., “Recent advances in deep learning for speech research at microsoft,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2013, pp. 8604–8608.

[20] P.-S. Huang, M. Kim, M. Hasegawa-Johnson, and P. Smaragdis, “Deep learning for monaural speech separation,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., May 2014, pp. 1562–1566.

[21] D. Amodei et al., “Deep speech 2: End-to-end speech recognition in english and mandarin,” in Proc. 33rd Int. Conf. Mach. Learn., Jun. 2016, pp. 173–182.

[22] A. Severyn and A. Moschitti, “Learning to rank short text pairs with convolutional deep neural networks,” in Proc. 38th Int. ACM SIGIR Conf. Res. Develop. Inf. Retr., Aug. 2015, pp. 373–382.

[23] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional net- works for text classification,” in Proc. Adv. Neural Inf. Process. Syst., Sep. 2015, pp. 649–657.

[24] T. Young, D. Hazarika, S. Poria, and E. Cambria, “Recent trends in deep learning based natural language processing,” IEEE Comput. Intell. Mag., vol. 13, no. 3, pp. 55–75, Aug. 2018.

[25] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei- Fei, “Large-scale video classification with convolutional neural net- works,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 1725–1732.

[26] J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 4694–4702.

[27] D.-A. Huang et al., “What makes a video a video: Analyzing temporal information in video understanding models and datasets,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7366–7375.

[28] D.-A. Huang, S. Buch, L. Dery, A. Garg, L. Fei-Fei, and J. C. Niebles, “Finding ‘it’: Weakly-supervised reference-aware visual grounding in instructional videos,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog- nit., Jun. 2018, pp. 5948–5957.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

516 IEEE TRANSACTIONS ON COMPUTATIONAL SOCIAL SYSTEMS, VOL. 6, NO. 3, JUNE 2019

[29] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, Jun. 2017.

[30] S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks for human action recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 221–231, Jan. 2013.

[31] C. Ding and D. Tao, “Trunk-branch ensemble convolutional neural networks for video-based face recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 1002–1014, Apr. 2018.

[32] H. Lee, C. Ekanadham, and A. Y. Ng, “Sparse deep belief net model for visual area v2,” in Proc. 20th Int. Conf. Neural Inf. Process. Syst., Dec. 2007, pp. 873–880.

[33] N. Srivastava and R. Salakhutdinov, “Learning representations for mul- timodal data with deep belief nets,” in Proc. Int. Conf. Mach. Learn. Workshop, Jul. 2012, pp. 79–87.

[34] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848, Apr. 2018.

[35] A. Coates, A. Y. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in Proc. 14th Int. Conf. Artif. Intell. Statist., Jun. 2011, pp. 215–223.

[36] M. Soltanolkotabi, A. Javanmard, and J. D. Lee, “Theoretical insights into the optimization landscape of over-parameterized shallow neural networks,” IEEE Trans. Inf. Theory, vol. 65, no. 2, pp. 742–769, Feb. 2019.

[37] V. Balntas, E. Riba, D. Ponsa, and K. Mikolajczyk, “Learning local fea- ture descriptors with triplets and shallow convolutional neural networks,” in Proc. Brit. Mach. Vis. Conf., Sep. 2016, pp. 1–11.

[38] V. Kurková and M. Sanguineti, “Probabilistic lower bounds for approx- imation by shallow perceptron networks,” Neural Netw., vol. 91, pp. 34–41, Jul. 2017.

[39] S. Lin, “Limitations of shallow nets approximation,” Neural Netw., vol. 94, pp. 96–102, Oct. 2017.

[40] R. Ranjan, V. M. Patel, and R. Chellappa, “Hyperface: A deep multi- task learning framework for face detection, landmark localization, pose estimation, and gender recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 1, pp. 121–135, Jan. 2019.

[41] E. Sangineto, M. Nabi, D. Culibrk, and N. Sebe, “Self paced deep learning for weakly supervised object detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 3, pp. 712–725, Mar. 2019.

[42] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Modl: Model-based deep learning architecture for inverse problems,” IEEE Trans. Med. Imag., vol. 38, no. 2, pp. 394–405, Feb. 2019.

[43] J. Yang, W. Xiong, S. Li, and C. Xu, “Learning structured and non- redundant representations with deep neural networks,” Pattern Recognit., vol. 86, pp. 224–235, Feb. 2019.

[44] Q. Zou, Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang, “Deepcrack: Learning hierarchical convolutional features for crack detection,” IEEE Trans. Image Process., vol. 28, no. 3, pp. 1498–1512, Mar. 2019.

[45] I. González-Díaz, J. Benois-Pineau, J.-P. Domenger, D. Cattaert, and A. de Rugy, “Perceptually-guided deep neural networks for ego-action prediction: Object grasping,” Pattern Recognit., vol. 88, pp. 223–235, Apr. 2019.

[46] Y. Niu, Z. Lu, J.-R. Wen, T. Xiang, and S.-F. Chang, “Multi-modal multi-scale deep learning for large-scale image annotation,” IEEE Trans. Image Process., vol. 28, no. 4, pp. 1720–1731, Mar. 2019.

[47] C. Jia, M. Shao, S. Li, H. Zhao, and Y. Fu, “Stacked denoising tensor auto-encoder for action recognition with spatiotemporal corruptions,” IEEE Trans. Image Process., vol. 27, no. 4, pp. 1878–1887, Apr. 2018.

[48] M. A. Kiasari, D. S. Moirangthem, and M. Lee, “Coupled generative adversarial stacked auto-encoder: Cogasa,” Neural Netw., vol. 100, pp. 1–9, Apr. 2018.

[49] M. Li, J. Hu, C. Xia, and Y. Zhang, “An implementation of picture compression with A CNN-based auto-encoder,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, Jun. 2018, pp. 2543–2546.

[50] J. Song, H. Zhang, X. Li, L. Gao, M. Wang, and R. Hong, “Self- supervised video hashing with hierarchical binary auto-encoder,” IEEE Trans. Image Process., vol. 27, no. 7, pp. 3210–3221, Jul. 2018.

[51] D. Chen, J. Lv, and Z. Yi, “Graph regularized restricted boltzmann machine,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 6, pp. 2651–2659, Jun. 2018.

[52] J. Chen, S. Cheng, H. Xie, L. Wang, and T. Xiang, “Equivalence of restricted boltzmann machines and tensor network states,” Phys. Rev. B, Condens. Matter, vol. 97, no. 8, Feb. 2018, Art. no. 085104.

[53] L.-W. Kim, “Deepx: Deep learning accelerator for restricted boltzmann machine artificial neural networks,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 5, pp. 1441–1453, May 2018.

[54] L. Ning, R. Pittman, and X. Shen, “Lcd: A fast contrastive divergence based algorithm for restricted Boltzmann machine,” Neural Netw., vol. 108, pp. 399–410, Dec. 2018.

[55] Y. Chen, H. Gao, L. Cai, M. Shi, D. Shen, and S. Ji, “Voxel deconvolutional networks for 3D brain image labeling,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Aug. 2018, pp. 1226–1234.

[56] J. Liu, Y. Wang, Y. Li, J. Fu, J. Li, and H. Lu, “Collaborative deconvolutional neural networks for joint depth estimation and semantic segmentation,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 11, pp. 5655–5666, Nov. 2018.

[57] X. Yu and F. Porikli, “Imagining the unimaginable faces by decon- volutional networks,” IEEE Trans. Image Process., vol. 27, no. 6, pp. 2747–2761, Jun. 2018.

[58] M. D. Zeiler, G. W. Taylor, and R. Fergus, “Adaptive deconvolutional networks for mid and high level feature learning,” in Proc. Int. Conf. Comput. Vis., Nov. 2011, pp. 2018–2025.

[59] J. Qiao, G. Wang, X. Li, and W. Li, “A self-organizing deep belief network for nonlinear system modeling,” Appl. Soft Comput., vol. 65, pp. 170–183, Apr. 2018.

[60] S. N. Tran and A. S. d’A. Garcez, “Deep logic networks: Inserting and extracting knowledge from deep belief networks,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 2, pp. 246–258, Feb. 2018.

[61] C. Zhang, K. C. Tan, H. Li, and G. S. Hong, “A cost-sensitive deep belief network for imbalanced classification,” IEEE Trans. Neural Netw. Learn. Syst., vol. 30, no. 1, pp. 109–122, Jan. 2018.

[62] Q. V. Le, “Building high-level features using large scale unsupervised learning,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Dec. 2013, pp. 8595–8598.

[63] K. Chen, J. Hu, and J. He, “A framework for automatically extracting overvoltage features based on sparse autoencoder,” IEEE Trans. Smart Grid, vol. 9, no. 2, pp. 594–604, Mar. 2018.

[64] H.-C. Shin, M. R. Orton, D. J. Collins, S. J. Doran, and M. O. Leach, “Stacked autoencoders for unsupervised feature learning and multiple organ detection in a pilot study using 4D patient data,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1930–1943, Aug. 2013.

[65] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer- wise training of deep networks,” in Proc. Adv. Neural Inf. Process. Syst., 2007, pp. 153–160.

[66] Y. I. Manin, “Frobenius manifolds, quantum cohomology, and moduli spaces,” Amer. Math. Soc., vol. 38, no. 1999, p. 101–108, 2007.

[67] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio, “Contractive auto-encoders: Explicit invariance during feature extraction,” in Proc. 28th Int. Conf. Int. Conf. Mach. Learn., Jul. 2011, pp. 833–840.

[68] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Proc. 25th Int. Conf. Mach. Learn., Jul. 2008, pp. 1096–1103.

[69] S. Li, J. Kawale, and Y. Fu, “Deep collaborative filtering via marginal- ized denoising auto-encoder,” in Proc. 24th ACM Int. Conf. Inf. Knowl. Manage., Oct. 2015, pp. 811–820.

[70] D. Slepian, “The one-sided barrier problem for gaussian noise,” Bell Syst. Tech. J., vol. 41, no. 2, pp. 463–501, Mar. 1962.

[71] Y. Pu et al., “Variational autoencoder for deep learning of images, labels and captions,” in Proc. Adv. Neural Inf. Process. Syst., Sep. 2016, pp. 2352–2360.

[72] J. Walker, C. Doersch, A. Gupta, and M. Hebert, “An uncertain future: Forecasting from static images using variational autoencoders,” in Proc. Eur. Conf. Comput. Vis. Springer, 2016, pp. 835–851.

[73] T. N. Kipf and M. Welling. (2016). “Variational graph auto-encoders.” [Online]. Available: https://arxiv.org/abs/1611.07308

[74] F. Tian, B. Gao, Q. Cui, E. Chen, and T.-Y. Liu,“Learning deep representations for graph clustering,” in Proc. 28th AAAI Conf. Artif. Intell., Jul. 2014, pp. 1293–1299.

[75] R. van den Berg, T. N. Kipf, and M. Welling. (2017). “Graph convolutional matrix completion” [Online]. Available: https://arxiv.org/abs/1706.02263

[76] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, Jun. 2017.

[77] V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proc. 27th Int. Conf. Mach. Learn., Jun. 2010, pp. 807–814.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.

WANG et al.: OVERVIEW OF UNSUPERVISED DEEP FEATURE REPRESENTATION FOR TEXT CATEGORIZATION 517

[78] I. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv. Neural Inf. Process. Syst., Nov. 2014, pp. 2672–2680.

[79] Y. Yu, Z. Gong P. Zhong, and J. Shan, “Unsupervised representation learning with deep convolutional neural network for remote sensing images,” in Proc. Int. Conf. Image Graph., Dec. 2017, pp. 97–108

[80] A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. (2015). “Adversarial autoencoders.” https://arxiv.org/abs/1511.05644

[81] C. E. Rasmussen, “Gaussian processes in machine learning,” in Proc. Adv. lectures Mach. Learn., Feb. 2003, pp. 63–71.

[82] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 770–778.

[83] L. Tran, X. Liu, J. Zhou, and R. Jin, “Missing modalities imputation via cascaded residual autoencoder,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Aug. 2017, pp. 1405–1414.

[84] M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolu- tional networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 2528–2535.

[85] M. Mäkitalo and A. Foi, “Noise parameter mismatch in variance stabilization, with an application to poisson–gaussian noise estimation,” IEEE Trans. Image Process., vol. 23, no. 12, pp. 5348–5359, Dec. 2014.

[86] D. H. Ackley and G. E. Hinton, “A learning algorithm for Boltzmann machines,” Cognit. Sci., vol. 9, no. 1, pp. 147–169, Jan./Mar. 1985.

[87] P. Smolensky, “Information processing in dynamical systems: Founda- tions of harmony theory,” in Proc. Parallel Distrib. Process., Explor. MicroStruct. Cogn., Feb. 1986, pp. 194–281.

[88] R. R. Salakhutdinov and G. E. Hinton, “Deep boltzmann machines,” in Proc. 12th Int. Conf. Artif. Intell. Statist., Apr. 2009, pp. 448–455.

[89] J. Chu, H. Wang, H. Meng, P. Jin, and T. Li, “Restricted Boltzmann machines with gaussian visible units guided by pairwise constraints,” IEEE Trans. Cybern., to be published.

[90] R. Salakhutdinov, A. Mnih, and G. Hinton, “Restricted boltzmann machines for collaborative filtering,” in Proc. 24th Int. Conf. Mach. Learn., Jun. 2007, pp. 791–798.

[91] N. Srivastava and R. Salakhutdinov, “Multimodal learning with deep boltzmann machines,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 2949–2980, 2014.

[92] C.-J. Kim and C. R. Nelson, State-Space Models With Regime Switch- ing: Classical and Gibbs-Sampling Approaches With Applications. Cambridge, MA, USA: MIT Press, 1999.

[93] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Comput., vol. 14, no. 8, pp. 1771–1800, 2002.

Shiping Wang received the Ph.D. degree from the School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China, in 2014.

He was a Research Fellow at Nanyang Technolog- ical University, Singapore, from 2015 to 2016. He is currently a Full Professor and Qishan Scholar with the College of Mathematics and Computer Science, Fuzhou University, Fuzhou, China. His research interests include machine learning, computer vision, and granular computing.

Jinyu Cai received the B.S. degree in computer science and technology from Fuzhou University, Fuzhou, China, in 2018, where he is currently pursu- ing the M.S. degree with the College of Mathematics and Computer Science.

His research interests include machine learning, computer vision, and pattern recognition.

Qihao Lin received the B.S. degree in network engineering from Fuzhou University, Fuzhou, China, in 2018, where he is currently pursuing the M.S. degree with the College of Mathematics and Com- puter Science.

His research interests include machine learning and computer vision.

Wenzhong Guo received the B.S. and M.S. degrees in computer science from Fuzhou Univer- sity, Fuzhou, China, in 2000 and 2003, respectively, and the Ph.D. degree in communication and infor- mation systems from Fuzhou University, in 2010.

He is currently a Full Professor with the College of Mathematics and Computer Science, Fuzhou Uni- versity. Moreover, he leads the Network Computing and Intelligent Information Processing Laboratory, which is a Key Laboratory of Fujian Province, China. His research interests include cloud comput-

ing, mobile computing, and evolutionary computation.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:09:16 UTC from IEEE Xplore. Restrictions apply.