help

Abstract – User generated content in the Web is a main channel to monitor customer satisfaction of service. Especially, customer review is recognized as the core of user generated content. Even though previous studies analyzed customer review using sentiment analysis, these studies have limitations in providing practical information for measuring customer satisfaction of service. Multi Criteria Decision Making (MCDM) approach is suitable for supporting an analysis of customer review with sentiment analysis because in service, multiple attributes affect customer satisfaction simultaneously. In response, this study proposes a framework for measuring customer satisfaction of service based on the user generated contents using sentiment analysis and MCDM approach. The proposed framework can provide practical information about customer satisfaction and directions for improvement of service.

Keywords – Customer satisfaction, user generated

contents, sentiment analysis, MCDM approach

I. INTRODUCTION With the rapid technological advance especially in internet, e-business and information sharing has been activated [1]. The fact that many business activities are occurred through the computer and internet reveals the core issue of how e-business can make high customer satisfaction [2]. Especially in the service, this issue has significant importance because customers want to get much customized services in e-business [3]. Thus it has become important to be able to monitor and enhance customer satisfaction in e-business. User generated contents can be recognized as a major channel to listen and monitor a voice of customer in the service [4]. This user generated contents mean all materials on the website that is uploaded, viewed, and downloaded by the users. Among many kinds of user generated contents, customer review is considered as the core of user generated contents because it contains direct experiences of customers. Also, customer review has the merit in terms of reliability since customers write review voluntarily, whereas survey method that is generally used to investigate customer satisfaction can lead to inaccurate results because of problems in contents of survey and attitude of respondent. Some previous research have been focused on their efforts to extract useful information automatically from the customer review by using sentiment analysis and

opinion mining. Some studies classified sentiment of words extracted from review document [5,6,7]. Other research focused on subjective expressions [8,9], subjective sentences [10], and topics [11,12]. However, previous research did not provide practical information for evaluating customer satisfaction based on the customer review because they just focused on the technique itself. Thus, there is a need for using sentiment analysis appropriately to analyze customer satisfaction based on the customer review. Multi Criteria Decision Making (MCDM) approach can be the most suitable method to support an analysis of customer review with sentiment analysis. This is because service attributes that affect customer satisfaction are much different among various categories compared with those of product. Also multiple attributes affect customer satisfaction of service simultaneously and weights of service attributes are different. Because of this, sentiment analysis can’t cover all of these aspects single-handed. It is reasonable to use MCDM approach to support an analysis of customer review with sentiment analysis effectively because MCDM approach can deal with multi attributes for evaluating service simultaneously. Accordingly, this study proposes a framework for measuring customer satisfaction of service based on an analysis of the customer review by using sentiment analysis and MCDM approach. Especially, because of the ease of usage and interpretation, aggregating function based MCDM approach is used in this study. The proposed framework mainly consists of two parts: data collection and preprocessing and measurement of customer satisfaction. In the stage of data collection and preprocessing, dictionaries of service attributes and sentimental words are constructed based on the review data by using text mining. Then, keyword vectors of customers’ opinions are constructed by using sentiment analysis. In the stage of measurement of customer satisfaction, measuring of customer satisfaction is conducted by using three aggregating function based MCDM methods. Lastly, validity test is conducted with various results derived from different MCDM methods. The proposed framework can provide useful information for diagnosing customer satisfaction and propose directions for improvement of service through review data of service. The rest of this paper is organized as follows. In section II, methodological background of this study is provided. The section of first introduces sentiment

Measuring customer satisfaction of service based on an analysis of the user generated contents: sentiment analysis and aggregating function based MCDM

approach

Daekook Kang1, Yongtae Park2 1Department of Industrial Engineering, Seoul National University, Daehak-dong, Gwanak-gu, Seoul, 151-744, Republic of

Korea (phone: 82-10-3152-2537; fax: 82-2-878-3511; e-mail: [email protected]). 2 Department of Industrial Engineering, Seoul National University, Daehak-dong, Gwanak-gu, Seoul, 151-744, Republic of

Korea (phone: 82-10-3152-2537; fax: 82-2-878-3511; e-mail: [email protected]).

978-1-4673-0110-7/12/$31.00 ©2012 IEEE 244

analysis. Also, three aggregating function based MCDM methods, Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS), Serbian: ViseKriterijumsa Optimizacija I Kompromisno Resenje (VIKOR), and Grey Relational Analysis (GRA), are briefly explained in this section. In Section III, the proposed framework is introduced with the overall process and detailed procedures. Lastly, section V concludes the paper with some contributions and observations about future research.

II. METHODOLOGICAL BACKGROUND A. Sentiment analysis Sentiment analysis is the technique for identifying how sentiments are expressed in texts and whether the expressions represent positive negative opinions toward specific subject [13]. This method includes several identifications such as sentiment expressions, polarity and strength of the expression, and their relationship to the subject. Among them, most of work in the field of sentiment analysis focuses on identification of polarities of sentiment expressions. A general approach of sentiment analysis is to find sentiment expression for a given subject and distinguish a lexicon of positive or negative words. There are three types of lexicons. First one is a lexicon that has a positive polarity such as beautiful. Second, words such as horrible are classified as a lexicon that has a negative polarity. Lastly, there is a lexicon that has a contextual polarity which means that a sentiment of the word varies depending on the context. In this study, customer satisfaction of service is measured based on the review data by using sentiment analysis. Polarity of words in review data to the service attribute is analyzed. Then, rating index is measured by using polarity of words with respect to each service attribute for a given service alternative. B. Aggregating function based MCDM approach MCDM are well-known acronyms for multiple criteria decision making that is method for solving decision and planning problems involving multiple criteria. There are many MCDM methods. Among them, aggregating function based MCDM methods such as TOPSIS, VIKOR, and GRA provide simple value to rank each alternative by considering the weight of different criteria. Therefore, it is easy to determine preferences of alternatives. Also, aggregating function based MCDM has the merit on the easiness of mathematical calculation. For these reasons, in this study, three aggregating function based MCDM methods, TOPSIS, VIKOR, and GRA, are used to measure customer satisfaction of service. Brief explanations of each MCDM method are as follows. (1) TOPSIS

TOPSIS is presented by [14] with reference to [15]. The basic concept of TOPSIS is simple. The chosen alternative should be as close to the ideal solution as possible and as far from the negative solution as possible. Ideal solution is constructed as a composite of the best performance values in terms of each criterion in decision matrix while the negative solution is the composite of the worst performance values. In TOPSIS, the n-dimensional Euclidean distance is used to calculate distance from ideal solution or negative solution. (2) VIKOR VIKOR is a compromise ranking method to optimize the multi-response process [16]. This method is used for ranking and selecting from a set of alternatives in the presence of conflicting criteria based on the closeness to the ideal solution. In VIKOR, each alternative is evaluated according to each criterion and ranked by comparing the closeness to ideal alternative. Compromising ranking index that represents preference of each alternative is derived by considering both maximum group utility and minimum individual regret of the opponent. (3) GRA Grey relational space is the concept that is proposed by combining system theory and space theory [17]. The concept of grey relations emphasizes the ‘greyness’ that means incomplete information. As a method for analyzing these grey relations, GRA is proposed to understand the uncertain relations between things, components of systems. GRA assumes that relationships can be identified among complex factors in a system. This method provides the correlations between the reference (desired) factors and other compared (alternative) factors of system based on the degree of similarity [18].

III. RESEARCH FRAMEWORK A. Overview of the proposed framework

245

Fig. 1. Overall research framework The proposed framework for mesuring customer satisfaction based on user generated contents, as shown in Fig. 1, mainly consists of two parts from an overall perspective: data collection and preprocessing and measurement of customer satisfaction. In the stage of data collection and preprocessing, firstly, review data of service is collected from the Web. Then, dictionaries of service attributes and sentimental words are constructed by using text-mining. With reference to dictionaries, review documents are transformed into keyword vectors of customers’ opinions to use in next stage. In the stage of measurement of customer satisfaction, based on the keyword vectors of customers’ opinions, rating of service is calculated with respect to each service attribute and weights of service attributes are also measured. Next, evaluation of final level of customer satisfaction are conducted by using aggregating function based MCDM. In this step, various results of evaluation derived by using three aggregating function based MCDM, TOPSIS, VIKOR, and GRA, are compared. Lastly, with reference to real overall rating of service, validity test is conducted to determine which MCDM approach has the highest performance for measuring customer satisfaction of service. B. Collection of review data In this phase, review data of service is collected with reference to Website. Review data includes overall rating for service and customer’s subjective opinions for service. Especially, customer’s subjective opinions for the service represent function of service and its merits and demerits. Therefore, we can derive important service attributes that customers carefully consider by analyzing customer’s subjective opinions for the service. In this study, review data of some similar services are collected to measure and compare customer satisfactions of services. Because similar services may be belonged in same categories, important service attributes are no great difference. Thus, common service attributes are extracted from review data of similar services and analyzed to

derive information about customer satisfactions for the service. C. Construction of dictionaries of service attributes and sentimental words In this phase, two dictionaries are constructed with reference to the review data. Firstly, dictionary of service attributes are constructed. To do this, as an ex-ante work for analyzing customer satisfaction, general service attributes that customers usually consider are extracted based on the literature review. Then, service attributes are hierarchically structured with parent components and sub- components. Parent components consist of general service attributes derived from the literature review and sub- component is extracted from review data by using text mining. By extracting sub-component for making hierarchical structure of service attributes, dictionary of service attributes are constructed. Generally, service attributes in the review data are expressed as a noun phrase. Thus, text mining is conducted to extract service attributes apropos of noun phrase in review data by using Term Frequency– Inverse Document Frequency (TF-IDF). Lastly, derived service attributes are corresponded to the sub-component of dictionary of service attributes. Secondly, sentimental words in review data are also extracted to construct dictionary of sentimental words. Since sentimental words are typically expressed as a verb phase, adjective phase and adverbial phase, sentimental words are extracted from these phases with reference to level of TF-IDF. Then, polarity of derived sentiment words in the dictionary are classified into three groups: positive polarity, negative polarity, and contextual polarity. Sentimental word that has contextual polarity is classified into neutral position and afterwards, this word is manually classified into positive or negative position based on relationships of service attributes. D. Construction of keyword vectors of customers’ opinions After dictionaries of service attributes and sentimental words are constructed, review documents are transformed into keyword vectors of customers’ opinions. For each review document, if keyword of service attribute and keyword of sentimental word are appeared at the same time, it is considered as information of customers’ opinions. In this case, keyword vectors of customers’ opinions are constructed as (1).

(1) : Review of customer i : Overall rating of customer i : Polarities for m attributes of service

246

Keyword vectors of customers’ opinions consist of two components: Overall rating and polarity for service attribute. Overall rating can be expressed as 5 point likert scale. Polarity for service attribute is expressed as an integer number between -2 and 2. If polarity for a certain service attribute is strongly positive, value of has 2. If polarity for a certain service attribute is positive, value of has 1. Meanwhile, if polarity for a certain service attribute is strongly negative, value of has -2 and if polarity for a certain service attribute is negative, value of

has -1. Lastly, if there is no polarity of a certain service attribute, it is express as a value 0. For example, if a customer writes a review of service such as “Overall rating: 4, a is the worst but b is good”, keyword vectors of customers’ opinions is constructed as (4, (-2,1,0,…)). Fig.2 shows examples of construction of keyword vectors.

Fig. 2. Examples of construction of keyword vectors For each customer, keyword vector is constructed and total score is derived by summing up score of the polarity for a certain service attribute. E. Measurement of customer satisfaction with respect to each service attribute For applying aggregating function based MCDM approach, in this phase, weights of service attributes and level of customer satisfaction for each service attribute should be derived. To measure weights of service attributes, some assumptions should be established. This is because a polarity’s impact of the service attribute into overall rating can be different among customers even if customers express same polarity for the same service attribute. Thus, it should be assumed that positive (negative) polarity for the service attribute contribute to increase (decrease) overall rating of service. Using this assumption, weights of service attributes can be derived based on (2). In this step, multiple regression analysis with least square method is used to measure weights of service attributes

(2) : Customer i’s overall rating of service

w: Weights of service attributes : Customer i’s polarity for service attribute On the other hand, to measure level of customer satisfaction for each service attribute, summing up score for the polarity of service attribute is needed. After total score of the polarity of service is derived, it should be normalized into a number between 0 and 10 to compare level of customer satisfaction among different services as shown in Fig. 3.

Fig. 3. Examples of normalization of score for the polarity of service attribute

Fig. 4. Examples of decision matrix Using derived weights of service attributes and level of customer satisfaction for each service attribute (criterion of evaluation), decision matrix is established to use in the next stage as shown in Fig. 4. F. Evaluation of final level of customer satisfaction For evaluating final level of customer satisfaction in each service, derived decision matrix is used as a data source for the aggregating function based MCDM approach. In this study, as mentioned earlier, three aggregating function based MCDM approaches, TOPSIS,

247

VIKOR, and GRA, are used and compared with each other. First, in TOPSIS, derived level of customer satisfaction for each service attribute is used to choose service alternative that is closest to the positive ideal solution and farthest from the negative ideal solution. Thus, TOPSIS method provides information of customer satisfaction that focuses on distances from best performance and worst performance in each service. Second, in VIKOR, service alternative is evaluated by measuring only closeness to ideal solution. Also, ranking index in VIKOR is derived by considering both maximum group utility and minimum individual regret of the opponent. Third, GRA uses level of the correlations between the desired factors and other alternative factors of system by considering the degree of similarity, not the closeness to ideal solution. Since each MCDM approach focuses on different perspectives to measure preference of alternatives, results of evaluation can be different among three approaches. For this reason, in this study, different MCDM approaches are used together to provide reliable results. G. Validity test To know how reliable results are derived, validity test is conducted. Preference results of three MCDM approaches are compared with real preference results by using real overall rating of service. Hit ratio of three MCDM approaches are calculated based on the accuracy of prediction in evaluating preference. Since high hit ratio implies that preference of service alternatives are well predicted, we can conclude that the MCDM approach that has high hit ratio are more suitable for measuring customer satisfaction in certain category of service. According to categories of service, MCDM’s performance for evaluating customer satisfaction can be changed. Thus, validity test is needed to find appropriate approach.

V. CONCLUSION This study proposed a framework for measuring customer satisfaction of service based on user generated contents using sentiment analysis and aggregating function based MCDM approach. Specifically, first, review data is collected to construct dictionaries of service attribute and sentimental words. Then, keyword vectors of customers’ opinions are derived by using sentiment analysis. Based on keyword vectors, customer satisfaction with respect to each service attribute is measured. Lastly, evaluation of final level of customer satisfaction is conducted by using three aggregating function based MCDM methods and validity test is conducted to compare the performance of each method. This study has three major contributions. First, since this study proposes framework for analyzing customer satisfaction of service based on customer review, it can be the basis of other research that deal with review data to

derive information of customer satisfaction. Second, this study uses MCDM approach to evaluate customer satisfaction. Therefore, compared with other service alternatives, we can know which service attributes should be improved. Also, through validity test, we can find the most suitable MCDM approach for a certain category of service. Third, this framework for analyzing customer satisfaction can provide valuable information to the practitioner because developer of certain service can take results of evaluation into consideration at the stage of the launch of beta version. Future research of this study is as follows. First, more advanced sentiment analysis is needed to increase the accuracy of the analysis especially for issue of rating of polarity. Second, previous MCDM approaches can be modified to analyze customer satisfaction of service well. Lastly, appropriate case study should be conducted to apply the proposed framework.

ACKNOWLEDGMENT This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government (MEST) (No.2011-0030814)

REFERENCES

[1] I. M. Klopping and E. McKinney, “Extending the technology acceptance model and the task-technology fit model to consumer e-commerce”, Information technology, learning, and performance journal, vol. 22 no. 1 , pp.35 – 48 , 2004

[2] C.C. Sun and G.T.R. Lin, “Using fuzzy TOPSIS method for evaluating the competitive advantages of shopping websites, Expert Systems with Applications, vol. 36, no. 9, pp. 11764- 11771, 2009

[3] G. Buyukozkan and G. Cifci, “A combined fuzzy AHP and fuzzy TOPSIS based strategic analysis of electronic service quality in healthcare industry”, Expert systems with applications, vol.39, no.3, pp.2341 – 2354, 2012

[4] E. Constantinides and S. J. Fountain, “Web 2.0: Conceptual foundations and marketing issues”, Journal of Direct, Data and Digital Marketing Practice, vol.9, no.3, pp.231 - 244, 2008

[5] V. Hatzivassiloglou and K. R. McKeown, “Predicting the semantic orientation of adjectives” In Proceedings of the 8th conference on european chapter of the association for computational linguistics, pp. 174–181, Madrid, Spain, 1997

[6] P. D. Turney, “Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews”, , In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), pp. 417-424, 2002

[7] L-W. Ku and H-H. Chen, “Mining opinions from the web: beyond relevance retrieval”, Journal of American Society for Information Science and Technology, vol. 58, no. 12, pp.1838-1850, 2007

[8] S-M. Kim and E. Hovy, “Determining the sentiment of opinions”, In Proceedings of the 20th international

248

conference on computational linguistics (COLING 2004), August 23 – 27, pp. 1367–1373. Geneva, Switzerland, 2004

[9] T. Wilson, J. Wiebe, and P. Hoffmann, “Recognizing contextual polarity in phrase-level sentiment analysis”, In Proceeding of the conference on empirical methods in natural language processing (EMNLP 2005), October 6–8, pp. 347–354, Vancouver, B.C., Canada, 2005

[10] B. Pang and L. Lee, “A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts”, In Proceedings of the 42nd annual meeting of the Association for Computational Linguistics (ACL), July 21–26, pp. 271–278, Barcelona, Spain, 2004

[11] J. Yi, T. Nasukawa, W. Niblack, and R. Bunescu, “Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques”, In Proceedings of the 3rd IEEE international conference on data mining (ICDM 2003), November 19–22, pp. 427–434, Florida, USA, 2003

[12] K. Hiroshi, N. Tetsuya,W. Hideo, “Deeper sentiment analysis using machine translation”, In Proceedings of the 20th international conference on computational linguistics (COLING 2004), August 23 – 27, pp. 494–500, Geneva, Switzerland, 2004

[13] T. Nasukawa and J. Yi, “Sentiment analysis: capturing favorability using natural language processing”, In Proceedings of the 2nd international conference on Knowledge capture, pp. 494–500, 2003

[14] Chen and C.L. Hwang, “Fuzzy multiple attribute decision making: methods and applications, Springer-Verlag, Berlin, 1992

[15] C.L. Hwang and K. Yoon, “Multiple Attribute Decision Making”, Lecture Notes in Economics and Mathematical System 186, Springer-Verlag, Berlin, 1981

[16] S. Opricovic, “Multicriteria optimization of civil engineering systems”, Faculty of Civil Engineering, Belgrade, 1998

[17] L. Dong, “Control problems of grey systems”, Systems and Controls Letters, vol. 5, pp. 288–294, 1982

[18] M.S. Kuo, G.L. Liang, and W.C. Huang, “Extensions of the multicriteria analysis with pairwise comparison under a fuzzy environment”, International Journal of Approximate Reasoning, vol. 43, no. 3, pp. 268–285, 2006

249