help

An Investigation of Effectiveness of “Opinion” and “Fact” sentences for Sentiment Analysis of

Coustomer reviews

Yuya Sawakoshi Osaka Prefecture University

1-1 Sakai, Osaka, Japan [email protected]

Makoto Okada Osaka Prefecture University

1-1 Sakai, Osaka, Japan [email protected]

Kiyota Hashimoto Osaka Prefecture University

1-1 Sakai, Osaka, Japan [email protected]

Abstract Recently, a customer review of travel information website has been a big influence on users in accommodations by the spread of the internet. In our research, as preprocessing of picking out information from the reviews, we propose a method to classify sentences into “Opinion sentences” and “Fact sentences” using SVM. And we confirm effectiveness of “opinion sentences” to estimate values by classification reviews into Positive and Negative using SVM.

Keywords Support Vector Machine, Opinion sentence, Fact sentence, Customer review, Sentiment Analysis

I. INTRODUCTION In recent years, a customer review of a travel information

website has a big influence on users in accommodations by the spread of the internet. Users tend to decide whether they consult reviews to accommodations and stay. And this tendency will become stronger.

On the other hand, because there are amount of reviews from other users, it is difficult for users to confirm and process these all reviews. Therefore, Picking out information which is needed by users automatically is beneficial. And for accommodations, it is also very important information sources for to know detailed value and opinions from users

In our research, as preprocessing of picking out information from reviews of the travel information website, we propose a method to classify sentences into “Opinion sentences” which may be efficient to estimate values and “Fact sentences” which is not “Opinion sentences”. And we confirm “Opinion sentences” efficient to estimate values by classification reviews into Positive and Negative by experiments.

In our research, we use Support Vector Machine to classify automatically.

In this paper, section 2 explains about reviews, “Opinion sentences” and “Fact sentences”, section 3 explains about Support Vector Machine, section 4 describes about experiments, their results and discussions and in section 5, we will conclude and describe future works.

II. CUSTOMER REVIEW, OPINION SENTENCES AND FACT SENTENCES

A. Customer review In our research, we collected reviews from a travel

information site “Trip advisor”. The review consist of a title, value of a hotel, a review text, date of visit, purpose of visit and value of detail (price, room, location, cleanliness, feeling in bed and service). In writing a review, a title and a review text are described freely, value of detail and value of a hotel are 5 degree evaluation (5 is best and 1 is worst.), date of visit needs year and month of visit, and purpose of visit is chosen from 4 categories family, couple, single and work. But, the entry of value of detail is optional. Therefore some reviews have no value of detail. In our research, the review texts were used for classification.

Fig. 1. A review of Trip advisor

Title

Text

Value of detail

Value of a hotel

2015 International Conference on Computer Application Technologies

978-1-4673-8211-3/15 $31.00 © 2015 IEEE

DOI 10.1109/CCATS.2015.33

98

B. Opinion sentences and Fact sentences In our research, sentences are classified 2 classes, one

related to the value of the hotel is Opinion and another is Fact. Specifically, Opinion sentences show the user considers about correspondence, equipment, location and so on and Fact sentences show the user writes affair and fact about correspondence, equipment, location and so on.

Before an experiment, we prepared sentence data from 2,000 reviews. There were 10,482 Opinion sentences and 5,010 Fact sentences. Next, I explain the way to prepare sentence data.

C. To prepare Opinion sentences and Fact sentences When we prepared data, we settled representations of an

end of sentences as standard criteria to classify them. Specifically, if the representation of an end of sentences shows feeling or value of reviewer, the sentence was Opinion sentence, and others were Fact sentences. For example,

5 ( The location is 5 minutes on foot. ) is Fact. But, 5 ( The location is convenient in 5 minutes on foot. ) is Opinion because there is

( = convenient ) in the end of the sentence. Besides the representation of end of the sentence, the sentence was written by order of the value word and the target word, for example ( A good point is

. ), and the sentence shows feeling of reviewer, for example ( I ll come back someday. ) and ( I ll never come here. ) are Opinion sentences.

III. SVM Support Vector Machine (SVM) is the supervised machine

learning that proposed by Vapnik[1]. By giving study data that includes plus and minus, SVM constructs the hyperplane that divides plus and minus. And using the hyperplane, SVM classifies unknown data into plus or minus. SVM can classifies in high dimension vector space, so that has high versatility Because vectorization in NLP often becomes high dimension, SVM is used much in NLP. In our research, we use SVM with

kernel function. Kernel function gives the way to calculate scalar product directly from data. It’s possible to reduce computational complexity.

Fig. 2. Support Vector Machine

IV. EXPERIMENTS AND DISCUSSIONS In our research, 2 experiments were made. Experiment #1

was to classify sentences into Opinion sentences and Fact sentences by SVM. Experiment #2 was to classify reviews into Positive reviews and Negative reviews by SVM using #1, and we validate the usability of classification Opinions and Facts. In these experiments, we adopt 10 cross-validations. We divide the experimental data into 10 data sets, and one of the data sets was used for test data and others are used for learning data. And if same experiment was repeated by 10th, we obtained 10 results. And we obtained the average the result.

A. Experiment #1 Classification Opinions and Facts We prepared experimental data by dividing reviews from

travel information website “Trip advisor” sentences and appended a label of Opinion sentences and Fact sentences. And I calculated accuracy, recall and f-measure by classifying the data by SVM.

In experiments, both test data and learning data divided into morphemes with MeCab. And we prepared the support vector based on morpheme group of learning data. And using the support vector, we classified the test data. The result is shown by Fig.3 and 4.

In this experiment, we prepared 5 results based on number of data.

Hyperplane

Plus

Minus

Support vector

99

Fig. 3. Figure 3-1 A result of classification of Opinion sentences

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1000 2000 3000 4000 5000

accuracy

recall

f- measure

Fig. 4. A result of classification of Fact sentences

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1000 2000 3000 4000 5000

accuracy

recall

f- measure

B. Examination #1 As the number of data increase, results of both Opinions

and Facts became better. And at 1,000 data, the difference between the accuracy and the recall was large. But, as the number of data increase, the difference is smaller. Therefore we suppose that SVM learned the feature of Opinions and Facts with increasing the number of the learning data. But, as the number of data increase, the rise rate became smaller. Therefore, we suppose that 0.9 is the upper limit of the classification by only increasing the number of data.

C. Future works

We think the key to get better results is to consider a part of speech and to change parameters of SVM’s kernel function. And, we need to increase the number of data more.

D. Experiment #2 Classification Positive and Negative As the experiment #1, we prepared experimental data by

dividing reviews “Trip advisor” into Positive and Negative. And I calculated accuracy, recall and f-measure by

classification the data by SVM. In the experiment #1, dividing unit of classification of Opinions or Facts was sentence made the sentence the subject, however in experiment #2, dividing Positive or Negative made the review the subject. In this study, we focused on the value of the hotel. We decided the review has value is 5 or 4 it is Positive. The review has value is 2 or 1 it is Negative. In this experiment, I did not use the review of value 3.

In this experiment, using classification Opinions and Facts, we picked up Opinions from reviews, and we made 2 experiments.

� (1)Both learning data and test data were only Opinions.

� (2)Learning data was only Opinions and test data contained both of them.

We collected reviews about Yokohama, Hiroshima, Sapporo, Kobe, Sendai, Osaka, Tokyo, Fukuoka and Nagoya that are principal cities in Japan. The number of reviews ware shown in TABLE1.

TABLE I. THE NUMBER OF POSITIVE AND NEGATIVE REVIEW

City Positive Negative Yokohama 5032 357 Hiroshima 2730 213

Sapporo 9453 578 Kobe 4390 298

Sendai 3098 246 Osaka 12469 905 Tokyo 26696 1858 1755

Fukuoka 6462 264 Nagoya 4353 384

total 74683 5103 5000

There was large gap between the number of positive data and negative data. We picked out same numbers of reviews from both of them in each city. And, we experimented with 5,000 positive reviews and 5,000 negative reviews. In this study, we settled the number of Tokyo’s reviews 1,755 in

order to make the total number of reviews to 5,000. Such as the experiment #1, we adopt 10 cross-validations and I made the average of the experimental result.

E. Experiment #2-1 Both study data and test data are only Opinion sentences As preprocessing of the classifying into Positive and

Negative of reviews, using the result of the experiment #1, we picked out Opinion sentences from reviews. And we used them for the experiment #2. In this experiment, I prepared 6 results based on number of Opinion sentences and Fact sentences.

100

Fig. 5. A result of classification of Positive in experiment #1

Fig. 6. A result of classification of Negative in experiment #1

F. Examination #2-1 The results of 1,000 Opinion sentences and Fact sentences

fell big in every criterion from the results using original text. I think there was 2 causes of the result.

� (1) We didn’t picking out effective Opinion sentences for classifying reviews into Positive and Negative.

� (2) There was no Opinion sentence, because all sentences of the review were classified into Fact sentences.

These causes were brought from the low ability of the classification constructed by 1,000 sentences in the experiment #1.

In experiment #1, as the number of data increase, results of both Opinion sentences and Fact sentences became better. However, it was not same in this experiment. The results of 5,000 Opinion sentences and Fact sentences was the almost same with the result of the original text. Therefore, if the classification Opinions and Facts is better, there is not influence for classifying reviews into Positive and Negative by using only Opinions.

G. Experiment #2-2 learning data is only Opinions and test data is the original texts We experimented with test data is the original texts in order

to prevent from wrong classification of effective Opinion sentences into Fact sentences.

Fig. 7. A result of classification of Positive in experiment #2-2

0 0.2 0.4 0.6 0.8 1

accuracy

recall

f- measure

Fig. 8. A result of classification of Negative in experiment #2-2

0 0.2 0.4 0.6 0.8 1

accuracy

recall

f- measure

H. Examination #2-2 As the number of Opinion sentences and Fact sentences

data increase, recall of Negative became better and accuracy of Positive became worse. Therefore, as the number of Opinion sentences and Fact sentences data increase, many reviews are classified into Negative.

In the experiment #2-1, both learning data and test data are only Opinions and there are no changes for accuracy, recall and f-measure. However, in the experiment #2-2, test data is the original texts, and used Fact sentences. Therefore, Fact sentences in test data influenced recall of Negative better.

101

V. CONCLUSION AND FUTURE WORKS From experiment #2-1, we obtained the result that if the

classification Opinion sentences and Fact sentences was better, there was not influence for classification reviews into Positive and Negative by using only Opinion sentences. And from experiment #2-2, we obtained the result that facts of test data influenced recall of Negative better. As future issue, we need to construct a method to classify Positive and Negative more adequately. For example, to verify Opinion sentences and Fact sentences correctly method will be useful for tagging of part of speech, Naïve Bayes or Deep Learning.

REFERENCES [1] V. N. Vapnik , The Nature of Statistical Learning Theory, 2nd ed.,

Springer-Verlag, New York, (2000) [2] Toshihiro Kawaguchi,Tohgoroh Matsui,Hayato Ohwada: “Opnion

Extraction from Weblog using SVM and Newspaper Article”,The 20th Annual Conference of the Japanese Society for Artificial Intelligence 2006, 1A3-3, 4pages, (2006)

[3] Kohei Shimada,Makoto Okada,Kiyota Hashimoto : “Opinion and Fact Extraction from Short Note of Stock Price using SVM(In Japanese)” ,The 19th Accociation for Natural Language Proccesing (NLP2013),2013

[4] Hong Yu,Vasileios Hatzivassiloglou.: “Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences” Empirical methods in Natural Language Processing(EMNLP2003),2003

[5] Bo Pang,Lillian Lee, Shivakumar Vaithyanathan: “Thumbs up? Sentiment Classification using Machine Learning Techniques” Empirical methods in Natural Language Processing(EMNLP2002)pp.76-86,2002

[6] Muhammad Abulaish,Jahiruddin,Mohammad Najmud Doja,Tanvir Ahmad: “Feature and Opinion Mining for Customer Review Summarization” Pattern Recognition and Machine Intelligence Lecture Notes in Computer Science Volume 5909, 2009, pp 219-224

[7] Bo Pang,Lillian Lee: “Opinion Mining and Sentiment Analysis” Journal Foundations and Trends in Information Retrieval archive Volume 2 Issue 1-2, January 2008 pp.1-135

[8] Dongjoo Lee,Ok-Ran Jeonq,Sang-goo Lee: “Opinion mining of customer feedback data on the web” Proceedings of the 2nd international conference on Ubiquitous information management and communication 2008(ICUIMC ’08) pp.230-235

102