helpfn
bcs
Short_Text_Clustering_Enhanced_by_Semantic_Matching_Model.pdf
Short Text Clustering Enhanced by Semantic
Matching Model
Zijun Peng
School of computer science and
technology
Harbin Institute of Technology
Weihai, China
Guodong Xin
School of computer science and
technology
Harbin Institute of Technology
Weihai, China
Yuliang Wei
School of computer science and
technology
Harbin Institute of Technology
Weihai, China
Wei Wang
School of computer science and
technology
Harbin Institute of Technology
Weihai, China
Bailing Wang
Harbin Institute of Technology
Harbin Institute of Technology
(Weihai) Innovation and Pioneer Park
Co.,Ltd
Weihai, China
Lianhai Wang
Shandong Provincial Key Laboratory
of Computer Networks
Qilu University of Technology
(Shandong Academy of Science)
Jinan, China
Abstract—With the popularity of social networks, short text
clustering has become a more and more important task that is
widely used. Short text clustering is a challenging problem
because social network short texts are characterized by
irregular words, a lot of noise, and sparse features. We propose
a Short Text Clustering enhanced by Semantic Matching Model
(abbr. to STCSMM). The STCSMM method applies the
knowledge of the tagged text similarity task dataset to the short
text clustering through the semantic matching model, thereby
improving the effect of short text clustering. First, we train a
semantic matching network on the data set of the text similarity
task, where the network contains the feature extraction layer
and the vector distance calculation layer. Then, we use the
learned feature extraction layer to extract short text feature and
use the vector distance calculation layer replaces the commonly
used distance metrics in the traditional K-means algorithm,
such as cosine distance, Euclidean distance and so on. Finally,
the text features obtained by feature extraction layer are applied
to K-means based on vector distance calculation layer. This
improved K-means clustering (STCSMM) has better
performance on the microblog text clustering dataset than some
existing methods such as K-means clustering with LDA, LSI or
average word embedding feature vectors.
Keywords—short text clustering, semantic matching model, K-
means, STCSMM
I. INTRODUCTION
Different from the normal text clustering, Short text clustering are characterized by irregular words, a lot of noise, and sparse features. Especially sparsity, that is, most words appear only once in each short text, so the TF-IDF measure does not work well in the short text clustering. To solve this problem, some researchers are studying how to extend and enrich the context of textual data. This is actually using external knowledge from Wikipedia, Baidu Encyclopedia or ontology, BigCilin to enrich the feature of text. Another way to solve this problem is to explore some complex models for short text clustering. For example, Yin and Wang (2014) proposed a Dirichlet multinomial mixture model-based
approach for short text clustering[1] and Cai et al. (2005) clustered texts using Locality Preserving Indexing (LPI) algorithm[2]. Yet how to design an effective model is an open question, and most of these methods directly trained based on bag-of-words (BoW) are shallow structures which can not preserve the accurate semantic similarities.
With the recent revival of interest in Deep Neural Network (DNN), many researchers have concentrated on using Deep Learning to learn features. Hinton and Salakhutdinov (2006) use deep auto encoder (DAE) to learn text representation from raw text representation[3]. Recently, with the help of word embedding, neural networks demonstrate their great performance in terms of constructing text representation, such as Recursive Neural Network (RecNN) (Socher et al., 2013) [4] and Recurrent Neural Network (RNN) (Mikolov et al., 2011)[5]. Deep Learning has shown excellent performance in many traditional NLP tasks, such as machine translation, emotional analysis, dialogue system, reading comprehension, semantic similarity calculation and so on. However, most of the tasks mentioned above are NLP tasks with supervised learning, and short text clustering tasks in unsupervised learning still have great exploration space. In 2012, Huang et al. (2012) used IDF weighted embedding as the global context in their task . Firstly, a fixed window is used to collect the context information of a word, and then an IDF coefficient is used as the weight to weigh the corresponding word vectors of the context words [6]. In 2015, Xu et al. proposed a short text clustering method based on convolutional neural network, which they called STCC[7]. However, due to the nature of shallow structure, weighted average word embedding method has limited effect in extracting text features. STCC uses self- taught learning framework and does not use any external labels, so that the performance of the trained convolutional neural network still has a lot of room to improve.
To address challenges mentioned above, we propose a Short Text Clustering enhanced by Semantic Matching Model (abbr. to STCSMM). First, we train a semantic matching network on the data set of the text similarity task,
480
2019 IEEE 2nd International Conference on Information Systems and Computer Aided Education (ICISCAE)
978-1-7281-3066-8/19/$31.00 ©2019 IEEE September 28-30•2019 Dalian, China
20 19
2 nd
In te
rn at
io na
l C on
fe re
nc e
on In
fo rm
at io
n Sy
st em
s an
d C
om pu
te r A
id ed
E du
ca tio
n (I
C IS
C A
E ) 9
78 -1
-7 28
1- 30
66 -8
/2 0/
$3 1.
00 ©
20 20
IE E
E 1
0. 11
09 /I
C IS
C A
E 48
44 0.
20 19
.2 21
68 0
Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:14:05 UTC from IEEE Xplore. Restrictions apply.
where the network contains the feature extraction layer and the vector distance calculation layer. Then, we use the learned feature extraction layer for short text feature extraction and representation, and then use the vector distance calculation layer replaces the commonly used distance metrics in the traditional K-means algorithm, such as cosine distance, Euclidean distance and so on. Finally, the text features obtained by feature extraction layer are applied to K-means based on vector distance calculation layer.
The semantic similarity calculation task mentioned above, in simple terms, is to judge whether two texts, text a and text b, are expressing the same meaning. One popular neural network framework for semantic similarity computing tasks is DSSM (Deep Structured Semantic Models) [8]. To put it simply, DSSM performs feature extraction (i.e., feature extraction layer) on the text with the shared parameter neural network after passing through the same embedding layer, so that two representation vectors of text are obtained respectively, and then the vector distance calculation layer is used to calculate the distance between two vectors. The output is the distance between two vectors. When the output is greater than or equal to 0.5, the text a and the text b can be considered to have the same semantics. When less than 0.5, the text a and the text b can be considered to have different semantics. We save the neural network thus learned and load the network parameters of the feature extraction layer and the vector distance calculation layer for short text clustering. Using the K-means method for short text clustering, there are two key points: (1) How to get a valid text representation; (2) The choice of distance metrics, that is, how to calculate the difference between vectors. The feature extraction layer and vector distance calculation layer that we learned on the semantic similarity calculation dataset through the semantic matching network can fully fit the above two points.
The main contributions of this paper are summarized as follows:
1) We attempt to explore the feasibility and effectiveness
of short text clustering with the use of supervised learning
semantic similarity tasks to enhance unsupervised learning
tasks.
2) We proposed Short Text Clustering enhanced by
Semantic Matching Model(abbr. to STCSMM). Our
approach makes good use of the tagged semantic similarity
task dataset. In a sense, our approach migrates the
knowledge of tagged datasets to unsupervised short text
clustering.
3) We conducted experiments on the short text clustering
dataset, the experimental results demonstrate that STCSMM
has good performance. We also performed experimental
comparisons to show that the neural network performs better
on the semantic similarity task dataset, while the
performance on short text cluster will also be better. The remainder of this paper is organized as follows:
In Section 2, we first describe the proposed approach STCSSM and implementation details. Experimental results and analyses are presented in Section 3. In Section 4, we briefly survey several related works. Finally, conclusions are given in the last Section.
II. METHODOLOGY
A. Semantic Matching Model
We will briefly review a very popular semantic similarity calculation model, Deep Structured Semantic Model (DSSM), which is one of the foundations of our proposed method. As shown in Fig.1, DSSM maps high-dimensional sparse text features into low-dimensional dense features in the semantic space by using DNN. The first hidden layer uses a 30k neuron to implement the hash of the word, and then passes the hashed feature through multiple layers of nonlinear layers, and finally the last layer of neurons in the DNN constitutes a features in low-dimensional dense semantic space. Fig.1. shows the semantic similarity calculation of a query Q1 and multiple documents D1-Dn. When only the semantic similarity of a query and a document is calculated, the DSSM model can be simplified as shown in Fig.2 Lecun (2005) proposed the Siamese Network, which is a similar solution to calculate the similarity of two pictures[9]. Because query Q and document D obviously belong to different semantic spaces, DSSM needs to use different feature extraction layers to calculate the similarity between query and document, while Siamese Network is the feature extraction layer of parameter sharing. This is the biggest difference between DSSM and Siamese Network.
Fig. 1. Deep Structured Semantic Model
Fig. 2. Siamese Network
B. Short Text Clustering enhanced by Semantic Matching Model
Combining the ideas of DSSM and Siamese Network, we built a semantic matching network as shown in Fig.3, first dividing the short text, then converting the sequence of words into a sequence of word vectors through the word embedding layer, and then extracting the features with shared weights feature extraction layer. The feature extraction layer that shared weights is to map the short text to a common semantic space, so that it conforms to the vector calculation in the same
481
Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:14:05 UTC from IEEE Xplore. Restrictions apply.
semantic space when the short text clusters. If the network is not designed to share weights, then the network parameters on both sides will produce some differences during the iterative training process. We will concatenate the two short text feature vectors into a double-length vector, calculate it with several nonlinear layers, and output a real number between 0 and 1, representing the semantic similarity between the two short texts.
Fig. 3. Semantic Matching Model
• Feature extraction layer: Here we can actually use a variety of text feature extraction methods, including: DNN, CNN, LSTM, attention mechanism, etc. Because the main purpose of this paper is to explore the enhancement effect of semantic matching model on short text clustering, so CNN with short training time is directly selected, and the comparison of other feature extraction methods will continue in the future. When the input matrix is convoluted by a convolution kernel with width N, the feature information of n-gram (N represents the width of the convolution kernels) can be obtained. The size of N represents the distance of the feature to be captured. In order to capture the characteristics of multiple distances, convolution kernels of various widths can be set up and the input matrix can be convoluted at the same time. According to this idea, we used multi-scale convolution kernels looked like Fig.4 shown. We used multi-scale convolution to construct a feature extraction layer in order to capture the most import semantic features of different word combinations (N-gram combination of words)in the text[10], it similar to the idea of TextCNN. After the convolutional layer, we used Max Pooling to extract the convolved results.
• Vector Distance Calculation Layer: Traditional vector distance calculations often use distance metrics directly. Common distance metrics include: cosine distance, Manhattan distance, Euclidean distance, and so on. In this paper, the vector distance calculation layer uses a multi-layer fully connected layer, the activation function uses ReLU, and finally the output layer activation function uses sigmoid, so that the output value is between 0 and 1. Taking the cosine distance as an example, the calculation of the cosine distance itself is without parameters, which means that it has no ability to learn and cannot cope with various scenes. Faced with this problem, it is traditionally to choose multiple distance metrics to experiment and then select the best metric of distance, which will take a lot of time and effort, and if better distance metrics are not placed in the selected metric set Inside ? The fitting ability of the multi-layer fully connected layer is relatively strong, so we use the multi-layer fully connected layer to replace the traditional distance
metric, so that the multi-layer fully connected layer learns a better distance calculation mapping in the training iteration. We can make such a simple hypothesis: if the cosine distance is the optimal vector distance calculation, the fitting ability of the multi- layer nonlinear layer should be close to fitting the cosine similarity calculation function M1, and if the cosine distance is not suitable The mapping M learned through training should exceed the cosine similarity.
Fig. 4. TextCNN Model
After the training is completed, the parameters of the feature extraction layer F and the vector distance calculation layer D of the semantic matching model are respectively loaded. First, the clustered short text set X is transformed into the feature vector set T by the feature extraction layer F, and then the vector distance calculation layer D is used to replace the distance metric in the conventional K-means. As mentioned above, it is STCSSM.
III. EXPERIMENTS
A. Datasets
There are two data sets used in our experiments, including data sets for semantic similarity tasks and short text cluster data sets, More information is as follows:
1) Semantic similarity task data set: combined with ATEC
Intelligent Financial Service Contest dataset and CCKS 2018
WeBank Intelligent Customer Service Question Match
Contest data set. The data set contains 100,000 levels of
semantic similarity calculation examples, mainly including
the customer service questions encountered by Ant Financial
and WeBank's customer service, including topics such as Ant
Credit Pay, Ant By Chanting, Alipay, and Micro-credit.
2) Short text datasets containing topics such as Meizu,
Xiaomi, Rockets, Jeremy Lin, Evergrande, and Haze, from
Sina Weibo, have been tagged with text categories in advance.
482
Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:14:05 UTC from IEEE Xplore. Restrictions apply.
B. Evaluation Metrics
There are two types of clustering performance metrics. One is to compare the clustering effect with a "reference model", which is called external indicator, and the other is to directly examine the clustering results without using any reference model, which is called internal indicator. Since our short text dataset has prepared the classification labels of cluster clusters, we use the following external indicator to evaluate the effect of the algorithm.
For the dataset 𝑐 = {𝑥1, 𝑥2, … , 𝑥𝑚} is divided into clusters 𝐶 = {𝐶1, 𝐶2, … , 𝐶𝑘 } by a cluster algorithm ,and the dataset 𝐷 is divided into clusters C∗ = {𝐶1,
∗ 𝐶2 ∗, … , 𝐶s
∗} by a reference model. Correspondingly, let λ represent the cluster mark vector C and let 𝜆∗ represents the cluster mark vector of C∗. We consider the pairing of the samples in pairs, as defined below:
a=|SS|,SS={(𝑥𝑖 , 𝑥𝑗 )|𝜆𝑖 = 𝜆𝑗 , 𝜆𝑖 ∗ = 𝜆𝑗
∗, 𝑖 < 𝑗} (1)
b=|SD|,SD={(𝑥𝑖 , 𝑥𝑗 )|𝜆𝑖 = 𝜆𝑗 , 𝜆𝑖 ∗ ≠ 𝜆𝑗
∗, 𝑖 < 𝑗} (2)
c=|DS|,DS={(𝑥𝑖 , 𝑥𝑗 )|𝜆𝑖 ≠ 𝜆𝑗 , 𝜆𝑖 ∗ = 𝜆𝑗
∗, 𝑖 < 𝑗} (3)
d=|DD|,DS={(𝑥𝑖 , 𝑥𝑗 )|𝜆𝑖 ≠ 𝜆𝑗 , 𝜆𝑖 ∗ ≠ 𝜆𝑗
∗, 𝑖 < 𝑗} (4)
The set SS contains sample pairs belonging to the same cluster in C and to the same cluster in 𝐶 ∗ and the set SD contains sample pairs belonging to the same cluster in C but belonging to different clusters in 𝐶∗ . Based on formula (1) - (4) above, Jaccard coefficient and FM index can be derived as external indicators of clustering performance measurement.
Jaccard coefficient (JC):
JC = 𝑎
𝑎+𝑏+𝑐 (5)
FM index (Fowlkes and Mallows Index):
FMI = √ 𝑎
𝑎+𝑏 .
𝑎
𝑎+𝑐 (6)
The results of the above performance measures are between [0,1], and the larger the value, the better.
C. Comparisons
We will compare the proposed method with the following popular clustering algorithms:
1) Conventional methods: K-means algorithm (Wagstaff
et al., 2001), which originally used TF-IDF for text features,
and LDA (AG et al., 2003)[11], LSI stands for K-means
algorithm for text features[12].
2) Average Embedding:K-means on the weighted average
of the word embeddings which are respectively weighted with
TF and TF-IDF. Huang et al. (2012) also used this strategy
as the global context in their task . Firstly, a fixed window is
used to collect the context information of a word, and then an
IDF coefficient is used as the weight to weigh the
corresponding word vectors of the context words[6].
3) STCSMM and cosine distance: K-means and only use
the feature extraction layer of the STCSMM method and the
vector calculation uses cosine distance instead.
D. Hyperparameter Settings
In our experiments, most of the hyperparameters are set uniformly for data sets. For the CNN model we use in feature extraction layer, we adopt the idea of multi-scale convolution core. In our experiments, we set the scale of convolution core to 2, 3, 4, 5, 6, the number of convolution core to 128, and the pooling layer to use Max pooling. We further set the dimension of word embedding to 100, adopt cutting and padding strategy for text length processing. We set the maximum sequence length to 150, because 150 can cover most of the short text data similar to Sina Weibo and Twitter. Different dropout rates affect the effect of model learning. After a variety of dropout rates experiments, we chose 0.3 in our experiment.
And because the initial centroid has a significant impact on the clustering results of the K-means algorithm, we have used multiple repeated random initialization of the centroid. More specifically, the final result of the experiment, we set up a method of averaging 10 experiments for all clustering methods.
Most of the hyperparameters in our experiment are set in Table I.
TABLE I. HYPERPARAMETER SETTINGS
Hyperparameter
Settings
Hyperparameter Settings
hyperparameter numerical value
1 Word embedding
dimensions 100
2 Minimum width of
convolution kernel 6
3 Maximum width of
convolution kernel 2
4 Number of
convolution kernels 128
5 Dropout Rate 0.3
6 Maximum sequence
length 150
7 Number of full
connection layers 3
8 Activation function ReLU\Sigmoid
E. Results
We use the method proposed in this paper to compare the K-means method of LDA and LSI for text feature, the K- means method for weighted averaging of word vectors by TF- IDF, and only use the feature extraction layer of the STCSMM method and the vector calculation uses cosine distance instead (STCSMM+COS). As shown in Table I., the experimental results show that our proposed STCSMM has a good effect, ranking first in the Jaccard coefficient and FM index, and ranking second is the STCSMM+COS method, which shows that the vector calculation layer of the STCSMM method is superior to the traditional distance.
TABLE II. COMPARISON OF DIFFERENT CLUSTERING METHODS
Comparison Evaluation Metrics
Jaccard FM
LDA 0.2567 0.4212
LSI 0.3013 0.4638
TF-IDF 0.2936 0.4825
STCSMM+COS 0.3140 0.5063
STCSMM 0.3536 0.5271
At the same time, as shown in Table Ⅱ., we also use F1 to
483
Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:14:05 UTC from IEEE Xplore. Restrictions apply.
compare the performance of semantic matching models with different performances in STCSMM from the horizontal, indicating that the semantic matching model with better performance has better clustering effect.
TABLE III. COMPARISON OF DIFFERENT STCSMM
Comparison Evaluation Metrics
F1 Jaccard FM
STCSMM 1 0.5320 0.2697 0.4146
STCSMM 2 0.6128 0.2938 0.4421
STCSMM 3 0.6876 0.3329 0.4910
STCSMM 4 0.7593 0.3536 0.5271
IV. RELATED WORK
In this section, we review the related work from three aspects: short text clustering, deep semantic matching model, and semantic similarity calculation.
1) Short text clustering: Several studies have attempted to overcome the sparsity of short text representations. One approach is to extend and enrich the knowledge of data. Among them, Banerjee et al. (2007) propose a method to improve the accuracy of short text clustering--enriching the representation of short texts by additional features from Wikipedia[13]. For example, Yin and Wang (2014) proposed a Dirichlet-based polynomial hybrid model for short text clustering[1].
2)Deep semantic matching: Huang et al. (2013) proposed the DSSM network in order to solve the problem of semantic similarity calculation between query and document (doc) in information retrieval[8]. Shen et al. (2014) proposed CLSM, also known as CDSSM, to change the original DNN presentation layer model to use CNN to extract text features[14]. Palmi et al. (2014) proposed LSTM-DSSM to capture long distances contextual features in text by using LSTM.[15]
3)Semantic similarity calculation:The Semantic similarity calculation has a wide range of applications, including: information retrieval system for matching user queries and matching web pages; for question similarity calculation, matching user questions with common problem databases in the system to obtain high quality Reply. Ant Finantional Services Group, China Conference on Knowledge Graph and Semantic Computing and some institutions and organizations have organized related competitions to promote the development of Semantic similarity calculation. Stanford University also established the Natural Language Inference Task (NLI) evaluation task and provided a long-term task evaluation ranking.
V. CONCLUSIONS
In this paper, we propose STCSSM to learn the semantic matching model and apply it to improve the short text clustering effect. A series of experiments show that our method shows superior performance than the benchmark method. If the data set of the semantic similarity task is larger, the effect of STCSSM will be better. We will further compare the clustering effect of different feature extraction methods
on STCSSM.
ACKNOWLEDGEMENT
The work of this paper is funded by the project of National Key Research and Development Program of China (No. 2016YFB0800802, No. 2017YFB0801804), Frontier Science and Technology Innovation of China (No. 2016QY05X1002-2), National Regional Innovation Center Science and Technology Special Project of China (No. 2017QYCX14), Key Research and Development Program of Shandong Province (No. 2017CXGC0706), and University Co-construction Project in Weihai City.
REFERENCES
[1] Yin, Jianhua & Wang, Jianyong. “A Dirichlet multinomial mixture model-based approach for short text clustering,” Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 10.1145/2623330.2623715.2014.
[2] Deng Cai, Xiaofei He, and Jiawei Han. 2005.“ Document clustering using locality preserving indexing,” Knowl- edge and Data Engineering, IEEE Transactions on, 2017(12):1624–1637.
[3] Geoffrey E Hinton and Ruslan R Salakhutdinov. “Reducing the dimensionality of data with neural networks,” Science, 2006, 313 (5786):504–507.
[4] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. “Recursive deep models for semantic compositionality over a sentiment treebank,” In EMNLP, 2013, volume 1631, page 1642. Citeseer.
[5] Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan H Cernocky, and Sanjeev Khudanpur. “Extensions of recurrent neural network language model,” In ICAS- SP, 2011, pages 5528–5531. IEEE.
[6] Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. “Improving word representations via global context and multiple word prototypes,”In ACL, 2012, pages 873–882. Association for Compu-tational Linguistics.
[7] Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, Hongwei Hao. “Short Text Clustering via Convolutional Neural Networks,” Proceedings of NAACL-HLT 2015, 2015, pages 62–69. Asso-ciation for Computational Linguistics.
[8] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, Larry Heck. “Learning deep structured semantic models for web search using clickthrough data,” Proceedings of the 22nd ACM international con-ference on Conference on information & knowledge managem- ent ACM, 2013.
[9] Sumit Chopra,Raia Hadsell,and Yann LeCun. “Learning a similarity metric discriminatively, with application to face verification,” In Computer Vision and Pattern Recognition, CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 539–546. IEEE, 2005.
[10] Yoon Kim.“Convolutional neural networks for sentence classifi- cation,” arXiv preprint arXiv:1408.5882, 2014.
[11] David M. Blei, Andrew Y. Ng, and Michael Jordan. “Latent Dirichlet Allocation,” Journal of Machine Learning Research 3(2003):993-1022.
[12] Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R, “Indexing By Latent Semantic Analysis. Journal of the American Society For Information Science,” , 1990, 41, 391-407. 10
[13] Somnath Banerjee, Krishnan Ramanathan, and Ajay Gupta. 2007. “Clustering short texts using wikipedia,” In SIGIR, pages 787–788. ACM.
[14] Yelong Shen, Xiaodong He and Jianfeng Gao. “A latent semantic model with convolutional-pooling structure for information retrieval, ”Pro-ceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 2014.
[15] Palangi, H. , Deng, L. , Shen, Y. , Gao, J. , He, X. , & Chen, J. , et al. “Semantic modelling with long-short-term memory for information retrieval,” arXiv preprint arXiv:1412.6629, 2014.
484
Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:14:05 UTC from IEEE Xplore. Restrictions apply.
