helpfn

1668 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 8, AUGUST 2017

Automatic Sentiment Detection in Naturalistic Audio Lakshmish Kaushik, Abhijeet Sangwan, Member, IEEE, and John H. L. Hansen, Fellow, IEEE

Abstract—Audio sentiment analysis using automatic speech recognition is an emerging research area where opinion or senti- ment exhibited by a speaker is detected from natural audio. It is relatively underexplored when compared to text based sentiment detection. Extracting speaker sentiment from natural audio sources is a challenging problem. Generic methods for sentiment extraction generally use transcripts from a speech recognition system, and process the transcript using text-based sentiment clas- sifiers. In this study, we show that this baseline system is suboptimal for audio sentiment extraction. Alternatively, new architecture using keyword spotting (KWS) is proposed for sentiment detection. In the new architecture, a text-based sentiment classifier is utilized to automatically determine the most useful and discriminative sentiment-bearing keyword terms, which are then used as a term list for KWS. In order to obtain a compact yet discriminative sentiment term list, iterative feature optimization for maximum entropy sentiment model is proposed to reduce model complexity while maintaining effective classification accuracy. A new hybrid ME-KWS joint scoring methodology is developed to model both text and audio based parameters in a single integrated formula- tion. For evaluation, two new databases are developed for audio based sentiment detection, namely, YouTube sentiment database and another newly developed corpus called UT-Opinion Opinion audio archive. These databases contain naturalistic opinionated audio collected in real-world conditions. The proposed solution is evaluated on audio obtained from videos in youtube.com and UT-Opinion corpus. Our experimental results show that the pro- posed KWS based system significantly outperforms the traditional ASR architecture in detecting sentiment for challenging practical tasks.

Index Terms—Amazon, keyword spotting, lvcsr, maximum entropy, opinion, sentiment analysis, UT-sentiment audio archive.

I. INTRODUCTION

T EXT based sentiment detection is an established field innatural language processing (NLP). Sentiment analysis / opinion mining, analyzes peoples opinions, sentiments, evalua- tions, appraisals, attitudes, and emotions towards entities such as products, services, organizations, individuals, issues, events,

Manuscript received August 2, 2016; revised December 8, 2016 and February 13, 2017; accepted February 16, 2017. Date of publication March 3, 2017; date of current version June 28, 2017. This work was supported in part by AFRL under Contract FA8750-12-1-0188, in part by NSF under Grant 1218159, and in part by the University of Texas at Dallas from the Distinguished University Chair in Telecommunications Engineering held by J. H. L. Hansen. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Zhen-Hua Ling.

The authors are with the Center for Robust Speech Systems (CRSS), Eric Jonsson School of Engineering and Computer Science, University of Texas at Dallas, Richardson, TX 75080 USA (e-mail: [email protected]; [email protected]; [email protected]).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TASLP.2017.2678164

topics, and their attributes [23]. There is an enormous amount of opinionated data in the social media and on the Web in the form of Twitter, Facebook, message boards, blogs, and user fo- rums [1], [2], [5], [6], [23], [44]. The decision making process of people is affected by the opinions formed by a wide range of thought leaders and ordinary people over the web. Amazon, Yahoo, Google and various other personalized websites are a sig- nificant resource for obtaining opinions concerning products of any kind. Many consumers form their decision to buy a product dependent on feedback from online reviews. This information not only helps ordinary people make decisions, but also provides indicators for companies about the reception of a product, or a political context, to understand the mood of people regarding an ongoing social/cultural/political/economic issue. Typically, a given text is classified to exhibit positive, negative or neutral sentiment [6], [9]–[11], [22]–[27], [29], [30], [36], [37]. This form of automatic classification has numerous applications such as measuring public opinion/sentiment using Twitter feed, an- alyzing online product reviews, understand mass social human behavior over a topic, product or an event [7], [8], [15]–[18].

Text based reviews form only one of the many ways peo- ple can express their sentiment/opinion about products or social issues. Audio/Video is also a prominent method to express opin- ions. Millions of videos on YouTube [65] are about products and movie reviews, product un-boxing, political, social issue analy- sis and opinions on them. There are many audio platforms on the Internet where individuals express their opinions. Also, the au- dio mode is more powerful than text for many situations because they provide richer cues of the speaker regarding their opinions. This vast resource is untapped and extracting sentiment/opinion of society about specific products or mass opinion regarding social or political situations will be very useful for information analysis. Detecting sentiment in audio is still an unexplored area.

Speech based sentiment extraction is an emerging and chal- lenging field. In this study, robust methods are presented to extract sentiment/opinion from natural audio sources. A hybrid system is developed which utilizes a robust Automatic Speech Recognition (ASR) system in tandem with NLP based senti- ment analysis techniques to detect sentiment of audio streams. Unlike text based sources, audio sources have a high degree of variability both in terms of expressing opinion as well as the mode of expression of the opinion. There are a range of chal- lenges for sentiment extraction in highly natural speech sources including:

1) Domain and Vocabulary: The speaker can express opin- ions about any topic, (e.g., products, movies, politics, social issues, games, etc.) Hence the ASR system should

2329-9290 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

KAUSHIK et al.: AUTOMATIC SENTIMENT DETECTION IN NATURALISTIC AUDIO 1669

be efficient to handle a wide range of domains and vocab- ulary. The language model should be comprehensive.

2) Speaker Variability and Speaker Accents: ASR system should be robust to speaker variability which includes a wide range of English accents from all over the world.

3) Noisy Audio and Channels: Inconsistent recording equip- ment and different mode/distance of recording, incon- sistent acoustic and background environment conditions make the sentiment detection problem challenging. Also, background music/talk, intentional music mixing, rever- beration issues make the problem harder.

4) Natural and Spontaneous: Detecting audio sentiment in natural and spontaneous speaker settings and various speaker interactive scenarios (i.e., 1-way, 2-way, public speech etc.) is challenging.

Given the explosive increase of online videos on product re- views, un-boxing, politics, sports, culture, etc. on websites such as YouTube.com, Vimeo, News broadcasting, Daily Motion, Twitch and Vine, automatic audio sentiment detection technol- ogy would be useful in collecting and summarizing information for users.

Fig. 1, shows all various aspects of the baseline and proposed audio based sentiment detection. There are three core subsys- tems. First one is the offline text based sentiment model gen- eration, the second is the ASR based sentient detection system forming our baseline system, and finally the third is a proposed system using audio Keyword Spotting (KWS) approach. Each block is explained in detail in subsequent sections. Robust senti- ment model development is explained in Section III. Next, many strategies for audio sentiment detection using speech recognition are presented. In Section IV, a continuous speech recognition system is developed for this purpose. Advantages and disad- vantages of the formulated preliminary sub-tasks based on the baseline ASR transcript have previously been explored in tan- dem speech sentiment system discussed in [12], [13]. In reality, accurate sentiment detection generally relies on a small fraction of the speech recognition transcript, because sentiment bearing vocabulary tends to be sparse in spoken opinions. For example, in a statement like “I watched Game of Thrones last night and it was wonderful”, only 1 word out of 11 conveys the some sort of sentiment. While this may not be true for every utterance se- quence, sparseness is generally prevalent in spoken comments. Given this nature of spoken comments, it would be reasonable to assume that sentiment detection is tolerant of moderate word error rates (WERs), and this is precisely what we observed in our previous study [13]. In other words, sentiment detection accuracy depends on being able to reliably detect and recog- nize a very focused vocabulary in the spoken comment audio stream. Therefore, keyword spotting (KWS) technology is ex- pected to be better suited for sentiment detection, as opposed to full-transcript ASR. In this study, a detailed analysis and KWS sentiment system development is presented in Section V. A new hybrid ME-KWS audio sentiment modeling method is also pro- posed. In this study, the proposed system is evaluated on two different corpora: UT-Opinion and videos from YouTube.com. The UT-Opinion (Section VII-B) is a new corpus specifically collected for the purpose of audio sentiment detection. The cor-

pus contains interview style data, where subjects give their opin- ion on various topics in natural settings around a university cam- pus. YouTube database (Section VII-A ) is a collection of videos from youtube.com which was carefully chosen for the purpose of sentiment research. Our experimental results show that the proposed KWS framework significantly outperforms the con- ventional ASR approach on both datasets. Results and analysis are presented in Section VIII.

II. BACKGROUND

Sentiment analysis can be classified mainly into the following four categories,

a) Document-level sentiment analysis: The sentiment is gen- erated on the overall document/review level. This is a global analysis. References include: [39], [42], [45], [46];

b) Sentence-level sentiment analysis: This gives a microlevel sentiment assessment for every sentence. This is effectively a local analysis. References include: [47]–[49]

c) Aspect-based sentiment analysis: This gives a sentiment variation in both a local and global level. Aspect-based sentiment analysis focuses on the recognition of all sentiment expressions within a given document, and the aspects/objects to which they refer [41], [51]. It is possible to have varying local sentiment with one overall document assessed value.

d) Comparative sentiment analysis: Extracting opinion in re- views where a product is compared with other product [60] (e.g.,: Google search is better than Yahoo).

The first step in development is to pre-process text using lin- guistic tools such as stemming, tokenization, parts-of-speech tagging, entity extraction, and relation extraction depending on classification technique [14], [51], [63]. Once we have the fea- tures generated, next comes classification. There are two main approaches to document level sentiment analysis: supervised learning and unsupervised learning. The supervised approach assumes there is a finite set of classes into which the document should be classified, and training data exists for each class in- cluding positive and negative, with neutral being an option (i.e., a 2-class or 3-class system ). Using various supervised meth- ods such as Maximum Entropy [21], SVM [40], Naive Bayes [40], Logistic Regression, or KNN, features can be learned to classify the sentiment exhibited. There are also unsupervised approaches to document-level sentiment analysis using seman- tic orientation of specific phrases within the document. Most methods use supervised approaches to classify the sentences into the two classes. The above techniques are used for doc- ument level classification. There are very new methods using Neural Networks [52], [54] and Recursive Neural Tensor Net- work (RNTN) [49] which analyze sentiment at the sentence level. Of all these methods, we use the ME based document level sentiment detection strategy in this research.

III. MAXIMUM ENTROPY TEXT SENTIMENT DETECTION

Maximum Entropy (ME) is a multinomial logistic regression method that predicts the probabilities of different possible out- comes of a categorically distributed dependent variable, given a set of independent variables. Maximum Entropy models offer a

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

1670 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 8, AUGUST 2017

Fig. 1. Baseline audio sentiment detection system uses Automatic Speech Recognition (ASR) to generate transcripts which are then processed by the sentiment classifier. Proposed sentiment detection system uses Keyword Spotting (KWS) to detect sentiment-bearing terms in the audio data. An iterative feature pruning technique is proposed to generate the sentiment-bearing terms for KWS. Offline development of customized text based sentiment models.

way to combine diverse pieces of contextual evidence in order to estimate the probability of a certain linguistic class occurring with a certain linguistic context. This is achieved by estimat- ing the probability of class a occurring with context b [21]. To develop the ME sentiment models, relevant features that reflect both positive or negative sentiment from the text are extracted and trained.

Maximum entropy based sentiment system is an effective ap- proach for discriminative learning of features. This can be used in huge database scenarios to develop probabilistic models that can predict sentiment effectively. Sentiment analysis typically predicts positive, negative or neutral sentiment given a sentence. In this study, we consider sentiment classification as a two-class problem, (i.e., Positive vs. Negative classification), and the Max- imum Entropy based approach is used to develop the sentiment classifier. Fig. 2 shows the methodology used to develop the text sentiment models. In what follows, each step is explained in detail.

A. Sentiment Feature Representation and Generation

Table I shows a sample product review from the website amazon.com, along with the corresponding star rating. The star rating is an objective measurement of sentiment on a 1-to-5 point scale, where greater and lesser number of stars correspond to positive and negative sentiment, respectively. It is common practice to use a star rating as ground truth for reviews. In this study, reviews with greater and lesser than 3 stars are assumed to convey positive and negative sentiment, respectively. Comments with 3 star ratings are discarded as they tend to be a mixed

Fig. 2. Steps for generating sentiment models using the proposed Itera- tive Maximum Entropy Optimization algorithm. Using a sentiment rich text database, parts-of-speech tags (i.e., Adjective, Verb, Adverb, Nouns combi- nations) are used to extract sentiment features. Iterative Maximum Entropy Optimization is used to learn sentiment classification models. In this study, we generate four variations of sentiment classification models.

collection of both positive and negative comments, and can be disruptive for training.

In the first step towards feature extraction, we extract words and word combinations which potentially depict sentiment. This is done by parsing the raw text using a part-of-speech (POS) tagger. As shown in Fig. 2, we use Stanford’s Log-linear POS tagger system [62], [63]. After tagging, words and word combi- nations formed by particular combinations of adjective (JJ), Verb (V*), Adverb (RB*) and Noun (N) are extracted as sentiment bearing features. In particular, the following regular expressions

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

KAUSHIK et al.: AUTOMATIC SENTIMENT DETECTION IN NATURALISTIC AUDIO 1671

TABLE I EXAMPLES OF RAW INPUT FROM AMAZON REVIEWS AND THEIR NORMALIZED EQUIVALENTS

Star rating is binarized depending on the star value (Range: 0–5) which acts as ground truth. Sentiment feature vector is extracted from each review using POS tagger followed by sentiment model training.

are used: (i) JJ, (ii) JJ-JJ, (iii) JJ-V*, (iv) V-JJ, (v) RB-JJ, (vi) RB-V, (vii) V*-RB, (viii) JJ-NN, (ix) NN-JJ, and (x) V*-NN. Table I shows this process of extracting sentiment bearing fea- tures for the sample text. The automatically identified features are shown in bold face. In this study, a large dataset derived from various online sources is used to extract the mentioned sentiment bearing features (more details about the dataset can be found in Section VI). In the next section, we explain how the ME classifier is trained using sentiment bearing features.

B. Maximum Entropy Classifier

We use Maximum Entropy to determine the comment sen- timent polarity given the textual sentiment features as input [19], [20], [29]. Let yj be the sentiment where yj ∈ Y and Y ≡ {positive, negative} is the set of sentiment polarities. In this case j has two values (positive and negative). Let xk be kth textual sentiment feature, then function fi is defined as:

fi (xk , yj ) = {

1 If xk is present in text comment, 0 Otherwise.

(1)

Equation (1) hypothesizes the relation between a feature present in the review text, and the corresponding review rat- ing. Sentiment features from each review with the correspond- ing sentiment truth (1 or 0) is applied as input to the classifier which learns over many instances of the feature. Applying an evidence based modeling technique such as ME the relationship can be estimated quantitatively. The ME technique can predict the rating of the review yj from features xk by using:

p(yj |xk ) = {

1 Z λ(x)

∑N i= 0 exp(λijfi (xk , yj )) (2)

where, Zλ(x) is a normalizing term, and λij are weights assigned to the function fi .

C. Feature Selection

All feature combinations noted in the previous section do not contribute to the sentiment accuracy equally. In order to better understand the relative strengths of feature combinations, we performed experiments where four different groups of features were used to build sentiment detection models. In particular, the following groups were used: (a) All POS tags combinations

are employed as features, (b) All POS tag combinations except features that use noun POS tags, (c) All POS tag combinations except features that use noun and verb POS tags, and (d) All POS tag combinations except features that use verb POS tags. Four different sentiment classification models were built using these four feature groups. By comparing the sentiment accuracy of these models using a common evaluation set, it was found that the system that uses feature group (b) provided the highest accuracy, and the system that uses feature group (a) provided the least accurate model. This suggests that noun-based features can potentially reduce sentiment detection accuracy, and therefore we use feature group (b) for the remainder of this study. To see further experimental details, interested readers can review our previous work [13].

D. Iterative Maximum Entropy Feature Optimization

Even after removing noun based features (as described in the previous section), the sentiment bearing feature set is rather large, (i.e., we obtain more than 322,000 sentiment features on our datasets). Such a large feature set can be somewhat redun- dant for audio based sentiment recognition, as automatic speech recognition solutions typically use relatively smaller vocabu- lary. In other words, the output transcript (even if we were to obtain 100% accurate transcripts) would never contain certain features due to restricted ASR vocabulary. Experimental tests on text based sentiment detection show that reducing the size of the feature set adversely impacts accuracy. Therefore, there is a need to reduce the size of the feature set in a manner such that the most discriminative and effective features are retained, thereby minimizing the impact of accuracy of the sentiment de- tection models. This motivates the development of an Iterative Maximum Entropy Feature Optimization, where we develop a new iterative method to reduce feature set size while retaining the most effective features for sentiment detection.

Features with lower frequency may give a good classification probability in MaxEnt, but being rare it doesn’t contribute sig- nificantly to the model sentiment like high frequency features. Pruning higher frequency features will be costlier than pruning features with lower frequency. Previously published Iterative Maximum Entropy Optimization method in [53], prunes the features by varying MaxEnt probability threshold margins in

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

1672 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 8, AUGUST 2017

Fig. 3. Steps for the proposed Iterative Maximum Entropy Feature Optimiza- tion. It generates the a highly optimized features and an optimal text sentiment model. This method generates a highly discriminative set of sentiment keywords.

each iteration. By doing so many high frequency features were relatively nearer to equi-probable point than rarely occurring features were pruned (they occurred in both positive and nega- tive reviews in large numbers). Hence, to avoid high frequency feature loss which could contribute to develop an efficient dis- criminative sentiment model a new iterative pruning method based on feature frequency is proposed.

First, all 322,000 sentiment features are rank-ordered by their frequency of occurrence using a dataset of 72 million reviews obtained by CRSS-UTDallas from all the sources mentioned in Section VI. Let Ni be the cutoff for the ith iteration, where Ni > 0. Features that have frequency of occurrence greater than or equal to Ni are chosen for training the sentiment model Mi . We increase the cutoff value by 25 for every iteration, (i.e., Ni+ 1 = Ni + 25, where N1 = 0). Therefore, all the features are used for training in the first iteration and the size of the sentiment detection model decreases with every iteration, as we discard more features from training. Since the original fea- ture is rank ordered by frequency of occurrence, we start with removing the most rarely occurring features first. Hence, we ex- pect to see minimum impact on accuracy along with significant reduction in overall feature set size in the initial iterations. Sub- sequently, with more iterations, we expect to see an increasing impact on accuracy as more low quality features are removed from training. For speech based sentiment detection, we want to choose the sentiment model that balances feature set size with accuracy.

Fig. 4 shows model accuracy vs. feature set size along with number of iterations (i) and corresponding feature cutoff (Ni ). The training starts with a minimum cutoff frequency of 0 and was increased up-to 1300 with a step size of 25 (as explained previously). It can be seen from the figure that the feature set size reduces significantly from about 322 K to 60 K (absolute reduction of 262 K) during the first few iterations (i ≤ 5), with accuracy decreasing only from 91.7% to 91.5% (absolute reduc- tion of 0.2%). In other words, the loss in accuracy is marginal but the reduction in feature set is significant. In subsequent iterations (i > 5), the accuracy falls much more sharply, and the feature set size reduction is relatively smaller. For example, for iteration i = 53, the model accuracy is 90.6% (about 1.1% absolute reduction), with a feature set size of about 10 K (an

Fig. 4. Feature set reduction using the proposed Iterative Maximum Entropy Feature Optimization algorithm. The algorithm prunes the least effective fea- tures in the initial iterations and provides significant reduction in feature set size with limited impact on sentiment detection accuracy. The models in the shaded area are practical for most automatic speech recognition systems.

absolute reduction of about 312 K). When compared to the pre- viously proposed probability margin based pruned sentiment model the new sentiment model developed using frequency based pruning saw a 1.4% relative increase in text based senti- ment classification accuracy.

In this manner, the proposed Iterative Maximum Entropy Feature Optimization algorithm provides smaller models that are more easily used with ASR with only a minimal impact on sentiment detection accuracy. In fact, models under 60 K as a feature set size are usable with speech recognizers as the feature set vocabulary can be absorbed into the ASR vocabulary while keeping the ASR vocabulary size manageable. In this study, we use feature sets with sizes below 64 K for audio sentiment recognition experiments.

IV. BASELINE SENTIMENT DETECTION FOR AUDIO

A simple way to perform sentiment detection for audio data is to process ASR output transcripts through a text-based sentiment detection system (such as the ME method described in previous sections). In this study, this system constitutes the baseline for audio based sentiment detection experiments.

Fig. 1 gives an overview of how ASR based sentiment extrac- tion works. The automatic speech recognition (ASR) system used in this study is the KALDI speech recognition system [72]. The acoustic models are trained using a mix-style approach, where acoustic data from multiple corpora are used. We have used Conversational Telephone Corpora consisting of Switch- board and Fisher corpora to train our acoustic models. The above corpora constitutes around 2000 hours of speech data. Part of the training data is corrupted with additive noise at var- ious SNRs (signal to noise ratio) to create a mixture of clean and noisy conditions (to mimic various kinds of noises like office, fan, cafeteria, car road which we expect to see in the evaluation corpora used in this study). The acoustic models use Mel frequency cepstral coefficients (MFCCs) along with deep neural networks (DNNs). The ASR vocabulary size is about 120 K words. The vocabulary contained in the 64 K sentiment feature set (described in previous sections) is part of the ASR vocabulary. Additionally, the language model for the ASR is

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

KAUSHIK et al.: AUTOMATIC SENTIMENT DETECTION IN NATURALISTIC AUDIO 1673

trained on data from two sources, namely, CTS (conversational telephone speech) corpora and reviews dataset (described in Section VI). Triphone language models are trained using vari- ous text sources like Switchboard [73], Fisher [74] and UW191 [75] (191 M) words collected from the web by the University of Washington) and entire six sentiment corpora explained in Section VI. Altogether, the combined text dataset for language model training contained more than 1 billion words. In this man- ner, the language context of the sentiment features is captured in the language model.

The ASR uses a 2-pass decoding strategy, where an fMLLR transform is estimated in the first pass and used to normalize the MFCCs. In the second pass, the normalized MFCCs are used with DNN models to obtain the output recognition lattice. Subsequently, one-best transcripts are obtained from the lattice. Finally, the one-best transcripts are analyzed by the Maximum Entropy sentiment detection model described in Section III-D to obtain the sentiment decision for the audio file.

V. PROPOSED KEYWORD SPOTTING SENTIMENT DETECTION SYSTEM FOR AUDIO

Sentiment bearing terms are relatively sparse in both speech and text. Therefore, a large majority of the ASR transcript, (as well as the errors in the transcript), have limited impact on sen- timent detection accuracy. When compared to ASR, Keyword Spotting (KWS) provides the ability to focus on relevant terms and phrases, and ignore irrelevant (from an application per- spective) background text. Therefore, it is suggested that KWS would be better suited for sentiment detection as opposed to employing full ASR transcripts directly.

In order to further explain the above mentioned argument, we present a simple experiment that demonstrates the impact of ASR WER (word error rate) on sentiment detection. In this experiment, we add controlled amounts of substitution er- rors (to simulate a required word error rate) in our text based evaluation dataset. While simulating substitution errors, words were randomly chosen and replaced by other words from the ASR vocabulary. This process generates various noisy versions of the evaluation dataset. Using our best sentiment detection model, we then compute sentiment detection accuracy for these noisy variants, and compare this accuracy to the original clean version.

Fig. 5 shows sentiment detection accuracy obtained on noisy variants of the evaluation dataset vs. the amount of controlled WER introduced in the transcript. This shows the tolerance of sentiment detection to ASR word errors. It can be seen that as simulated WER increases from 0% to 80%, there is a gradual, almost linear drop in average sentiment detection accuracy from 91.7% to 80%. When simulated WER increases beyond 80%, we see a sharp decrease in accuracy (and the system reaches ran- dom performance, (i.e., 50%) at 100% simulated WER). This simple experiment shows that automatic sentiment detection in audio can tolerate high amounts of WER. In other words, successful sentiment detection depends on finding the few effecting sentiment relaying terms/phrases. This finding is fur-

Fig. 5. A plot of the variation in sentiment detection accuracy vs. simulated word error rate (WER). The trend suggests that only a small fraction of words can effectively determine sentiment, and most words are irrelevant from a detection perspective. This implies that KWS may be a feasible and superior solution for sentiment detection over ASR.

ther strengthened by empirical observations that sentiment bear- ing terms are relatively rare in text and audio (for example, approximately 2-to-5% relative frequency of occurrence in re- views). Taken collectively, this implies that sentiment detection should be possible with KWS using a relatively small keyword list, and it is likely to outperform full transcript ASR. Though an ASR system might generate word errors differently than a ran- dom substitution word error simulation; however the sentiment distance between the correct ASR word errors vs Randomly se- lected word errors should be equally large/divergent. Therefore we feel the simulation accurately represents the system’s abil- ity to accurately access sentiment of the content. It is difficult to force a predetermined WER in ASR systems for Keyword Spotting.

A. Keyword Spotting Based Sentiment Detection System

In order to build an effective KWS system, we need a com- pact yet effective keyword list. The Iterative Maximum Entropy Feature Optimization algorithm described in Section III-D al- lows us to extract the most effective and compact feature sets. As previously shown in Fig. 3, each iteration provides an in- creasingly smaller feature set (which can be directly used as the keyword list). Using this approach, we create keyword lists of small (approximately 200, 500, 1 K, and 2 K), to medium (approximately 4K, 8K, and 16K), and large (approximately 32 K, 48 K, 64 K and 80 K) word count sizes.

B. Keyword Spotting System

The KWS system developed is an extension of our KALDI based large vocabulary ASR system from Section Section IV. Using the word lattice generated by the ASR previously in Section IV, a finite state transducer (FST) based method is used to search the word lattices for keywords [72]. In parallel, the word lattices are converted in to phone lattices, and the PCN-KWS (phone confusion network keyword spotting) method employed to search for keywords as well [70], [71]. Subsequently, the search results from the two methods are

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

1674 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 8, AUGUST 2017

combined (by simple likelihood combination) to yield the final keyword result list. The KWS system assigns log-likelihood values to keyword hypotheses in the audio. The higher the value of the log-likelihood, the more confident the hypothesis. Keywords with a likelihood below a certain pruning threshold are rejected, and not used for further computation. In our experiments, we have rejected keywords with log-likelihood below −3 (this threshold value is empirically determined and yields a good equal error rate across many KWS experiments).

C. KWS-ME Hybrid Evaluation System

Maximum entropy is used to compute the sentiment detection likelihood using (2). The function fi (xk , yj ) in this equation as- sumes binary values of 0 and 1 either the absence or presence of evidence (i.e., sentiment bearing feature), respectively. Unlike text, using KWS for detecting sentiment bearing terms in au- dio produces the likelihood of the sentiment bearing feature (a probability number that is between 0 and 1). In order to incorpo- rate keyword detection confidence into the sentiment detection likelihood computation, we propose to set fi (xk , yj ) equal to the exponent of the log-likelihood. In this manner, fi (xk , yj ) can now assume values between 0 and 1, and it represents the confidence of detecting the sentiment bearing feature. The ben- efit of computing fi (xk , yj ) is this manner is that features that are detected with higher confidence have a greater say in deter- mining the overall sentiment detection outcome. In other words, let ck be the confidence score of kth detected sentiment bearing feature. Now, fi (xk , yj ) in (2) is computed as,

fi (xk , yj ) = ck , where 0 ≤ ck ≤ 1 (3) Using the above equation, we compute a new value of the ME

sentiment detection likelihood which now incorporates keyword detection confidence into scoring.

VI. TEXT DATABASES

In this study, we have used the following text corpora for our experiments:

1) Amazon Product Reviews [55], [57], [58], 2) Pros & Cons database [59], 3) Comparative Sentence Set Database [60], 4) Opinion Lexicon Database [55], 5) Scale database [14], and 6) R-T polarity database [55]. The noted selection attempts to obtain data from various do-

mains (ensuring diversity). Data from such varied sources tend to have inconsistencies in the rating and data representation. For example, some reviews are graded on a rating scale of 1–10, and others on 1–5. In this study, the star ratings for all reviews are normalized by first transforming all ratings to a 5-point scale. Subsequently, ratings of 3 are discarded, ratings below and equal to 2 are labeled negative, while ratings above and equal to 4 are rated positive.

The total dataset contains approximately 53 million reviews, 43 million positive, and 10 million negative. This dataset is used to create balanced training and evaluation datasets for our experiments. In particular, 8 million reviews (4 million positive

and 4 million negative reviews) are used for training, 1.1 million reviews for evaluation (550 K positive and 550 K negative), and 2 million reviews for development (1 million positive and 1 million negative). All datasets are chosen randomly and have no overlap.

VII. AUDIO DATABASES

Unlike text sentiment detection which has numerous data sources, audio corpora resources dedicated specifically for sen- timent detection task are very limited [64]. In the proposed work we have developed databases in a very controlled manner keeping audio sentiment detection as the main focus. In this context, we have developed two new audio datasets to support our sentiment detection experiments.

A. YouTube Sentiment Audio Database

YouTube videos [65] are an ideal source to collect speech data where speakers follow a natural and spontaneous speaking style while sharing their opinion on a wide variety of topics. We have collected a set of 85 videos (44 positive and 41 negative) that contain people speaking spontaneously. These videos cover a wide range of topics including product reviews, movies, social issues and political opinions. The dataset contains a balanced speaker distribution. It has 55 male speakers, 30 female speak- ers, and 5 videos with multiple speakers. The average duration of the videos is about 5 minutes with individual video durations ranging from 2 to 15 minutes. The total duration of the evaluation dataset is about 7.5 hours. The audio quality, recording equip- ment, channel characteristics, and accents/dialects vary across videos. These videos can be accessed via a YouTube playlist: http://bit.ly/YAgoYU.

In order to establish the sentiment ground truth on the YouTube videos, three listeners evaluated the videos and rated them for positive and negative sentiment. The listeners were asked to judge the videos for positive or negative sentiment.

The evaluators were asked to watch the video and listen to the speaker while wearing headsets. They were asked to set volume to a comfortable level and the experiment was conducted in an office environment. Each evaluator judged the sentiment of the speaker in each video on a 5-point scale, where, 1, 2, 3, 4, and 5 meant extremely negative, negative, neutral, positive, and extremely positive, respectively. In order to determine the final ground truth, the ratings of the three evaluators were averaged to obtain one rating per video. Finally, average ratings of 1 and 2 were considered negative, 4 and 5 positive, and 3 neutral. These decisions were used as ground truth for the experiments presented in this study.

B. UT-Opinion Audio Archive

The YouTube dataset mentioned in the previous section has some drawbacks. It is hard to control factors such as age, gender and accent (and these are relevant variants towards measuring performance of a speech based system). Additionally, the sen- timent expressed in YouTube videos is less likely to be subtle (i.e., speakers tended to take extreme positions). Finally, it is

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

KAUSHIK et al.: AUTOMATIC SENTIMENT DETECTION IN NATURALISTIC AUDIO 1675

hard to get a diverse set of opinions on the same topic (i.e., we find it hard to control for topic as a variable in the study).

In order to overcome the above mentioned issues, we have de- veloped the UT-Dallas Sentiment (UT-Opinion) Audio Archive. In this corpus, each participant was asked 10 questions to illicit opinion. The questions are:

1) Are you aware of our college Mascot TEMOC? Do you like it?

2) Which is the last movie that you saw? Did you like it? 3) Do you remember your last vacation? How was your

experience? 4) Are you satisfied with the library and other campus fa-

cilities? 5) Are you satisfied with the university’s bus/transport fa-

cilities? 6) Do you like living in Dallas? 7) What is your opinion about Google Glass? 8) Do you remember the last time you went for a fancy

restaurant dinner? What was your experience? 9) Recall the last course you completed. Did you like the

class? 10) Would you like a dog as your pet? The questions were asked in an interview style, with the in-

terviewer and participant facing each other. The responses were recorded using a Samsung hand-held tablet. The distance be- tween the participant and the tablet microphone was between 1-to-2.5 ft. After that participant answered the question sub- jectively, they were also asked to rate their opinion on a scale of 1-to-5 (where, 1, 2, 3, 4, and 5: meant extremely negative, negative, neutral, positive, and extremely positive, respectively). This data was collected in various acoustic settings inside the campus, namely, classrooms, hallways, offices, library, gym and street. Altogether, 63 people participated in the study, 43 male and 20 female. Students, staff and faculty participated in the data collection. The ages varied from 19 to 62 years old. The corpus consists of a diverse speaker set including native speakers of American English as well as Arabic, Spanish, Farsi, Mandarin, Korean, Hindi, Tamil, Kannada, Marathi, Bengali, Telugu, Ital- ian and French.

Using the method described in Section VII-A, 5 evaluators listened to the audio data and generated the sentiment ground truth. From the tabulated results, the Inter-Rater agreement for YouTube database is 94.5% and 80.1% for UT-Opinion dataset. It can be observed that when compared to YouTube dataset UT-Opinion has a lower agreement rate. This shows that the sentiment detection for UT-Opinion database is relatively not as easy when compared to YouTube database for human an- notators. This is an indication that the sentiment detection for UT-Opinion dataset could also be a challenging task for machines.

VIII. RESULTS AND DISCUSSION

In this section, experiments on text and audio based sentiment detection are presented.

TABLE II EQUAL ERROR RATE FOR AUTOMATIC FEATURE GENERATED AND

HANDCRAFTED FEATURE SYSTEMS

Dataset Equal Error Rate (Error Reduction)

Hand Crafted Features 13% Unigram and Bigram Features 8% (38%)

TABLE III TEXT BASED SENTIMENT MODEL RESULTS ON COMPARATIVE DATASET

Datasets Precision Recall F-Score

Naive Max Naive Max Naive Max Baye’s Entropy Baye’s Entropy Baye’s Entropy

Reviews 84% 92% 80% 90% 82 91 Articles 75% 88% 80% 84% 77 86 Forums 73% 86% 83% 86% 78 86

A. Text Sentiment Results

In this experiment, we show the accuracy of the automati- cally generated feature set developed in Section III-C on the text evaluation corpus described in Section VI. The automati- cally generated feature set in compared to a handcrafted list of 6,800 sentiment features developed by researchers at the Uni- versity of Chicago for the purposes of text sentiment detection [59]. The handcrafted feature set has been popular and used for building many text based sentiment detection systems [28], [33], [34]. The datasets mentioned in Section VI are used for this experiment.

Table II shows the The equal error rate (EER) of the above mentioned systems. The EERs for automatically ex- tracted feature set and handcrafted feature set are 8% and 13%, respectively. As expected, the automatically extracted feature set outperforms the handcrafted feature set. There is a 38% re- duction in error when both Unigram and Bigram features are used. An interesting point to note is that the handcrafted feature set (7 K features) is only 5% lower in accuracy compared to the automatically extracted feature set (322 K). This shows that typ- ically a small highly discriminative feature set determines the majority of the outcome in sentiment detection. From an audio based sentiment detection perspective, this is an important point since smaller keyword lists are more practical than larger ones.

In another experiment, the proposed ME based sentiment system (with the proposed Iterative Maximum Entropy Feature Optimization) is compared to the Naive Baye’s method for sen- timent detection in Table III. The same feature set is used for both MaxEnt and Naive Bayes. In this experiment, both meth- ods are evaluated on the widely used comparative dataset corpus which has 3 subdivisions, namely, product reviews, forum re- views, and articles [60]. The results are shown in Table III. It can be seen that the F-Score of proposed ME sentiment system is significantly better than the Naive Baye’s sentiment system

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

1676 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 8, AUGUST 2017

Fig. 6. Accuracy distribution curve for proposed hybrid ME-KWS audio sen- timent system. This gives information regarding the most optimal keyword list to be used.

across all three categories. From a performance perspective, this shows that the proposed ME system is competitive.

B. Audio Sentiment Results

In the first audio experiment, we try to determine the best keyword list for audio based sentiment detection. By using all keyword lists extracted using the method described in Section V-A, we evaluate sentiment detection accuracy for both YouTube and UT-Opinion datasets against each keyword list. Fig. 6 shows sentiment accuracy vs. keyword list size. It shows the impact of choosing larger sized keyword sets in order to im- prove the overall system accuracy. The intention is to show the saturating behavior of accuracy as we choose more and more keywords. It is observed that accuracy for both YouTube and UT-Opinion first increases as we increase the keyword list size from 200 to 32 K. After 32 K, the accuracy seems to level off before eventually decreasing. Since the smallest keyword lists have the most effective sentiment bearing features, the accuracy initially rises quickly.

As the keyword size grows, the new keywords are relatively less effective, and therefore accuracy does not rise as rapidly. In KWS systems, it is commonly understood that increasing the size of the keyword list can potentially increase the number of false alarms (i.e., the system falsely detects a keyword that is not actually present in the audio). In other words, bigger lists can be less accurate leading to high false alarms. From the trend in Fig. 6, it appears that incorrectly detected features (due to errors in KWS) in the biggest keyword lists overwhelm the system resulting in a loss of accuracy. It is possible that a high number of false alarms contribute towards this outcome. Since the 48K keyword set gives the best performance, we choose this keyword set for all remaining experiments.

Table IV shows the EER of the baseline ASR transcript based sentiment system and the proposed Hybrid ME-KWS based sen- timent detection systems. It can be observed that the proposed method outperforms the baseline system on both YouTube and UT-Opinion corpora.

The proposed system yields a +4% absolute improvement on the YouTube dataset (19% relative reduction in error) and

TABLE IV RESULTS FOR THE PROPOSED HYBRID ME-KWS AND ASR

TRANSCRIPT+ME SYSTEM

Dataset Method

ASR Transcripts Proposed Hybrid ME+KWS with Max Entropy (Total Error Reduction)

YouTube 18.41% 14.9% (19%) UT-Opinion 31.8% 29.2% (8.2%)

TABLE V PERFORMANCE ACROSS VARIOUS PARAMETERS IN UT-OPINION DATABASE

Variable Type Person Count Sentiment Detection EER

ASR Proposed Transcript Hybrid ME-KWS

Accent Native 40 25.8% 23.5% Non Native 23 38.1% 34.9%

Environment Office 41 20.6% 19.1% Hallway + 12 43.0% 39.3%

Library + Gym

a +2.6% absolute improvement on UT-Opinion dataset (8.2% relative reduction in error). Additionally, the performance of both ASR and KWS systems are better on the YouTube dataset when compared to UT-Opinion. The results show that the pro- posed KWS method for sentiment detection outperforms the baseline ASR method for both corpora. The results also show that sentiment detection task on UT-Opinion is more difficult, perhaps owing to wider variations in accents, noise, and subtly in sentiment expression. Another main reason is that UT-Opinion audio files are shorter in duration and captured in field inter- view scenarios. Since YouTube files are longer, there is more evidence for consistent sentiment detection, making the task rel- atively easier. In YouTube, in general, people express stronger opinion than the UT-opinion corpus (we note that the topics in UT-opinion are generally less controversial than many ex- amples in YouTube), which also makes the problem relatively easier.

Table V gives a detailed distribution of performance across various parameters of the developed UT-Opinion database. The performance is discussed in terms of accent and environment. It can be observed that the performance is improved in both the cases by using the proposed method. There is a very good improvement in detecting non-native speaker’s sentiment using ME-KWS system. Hallway and Gym are very noisy places. The performance in such environments have also increased by the using the proposed method. Human inter-rater agreement being relatively low for UT-Opinion as shown in Section VII-B is further proof that the sentiment detection for this data is a challenge.

But, when compared to YouTube audio database and text based systems the performance is almost less by 15%. Perfor- mance on text corporas are high because the data is rich in

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

KAUSHIK et al.: AUTOMATIC SENTIMENT DETECTION IN NATURALISTIC AUDIO 1677

opinion. In text based reviews, a case where a reviewer doesn’t have a strong opinion are seldom less. Reviews with strong sen- timent based words are easier to classify. Similarly, in YouTube database the opinions of the expressed by particularly strong and detailed. Hence the performance of the YouTube is relatively comparable to text based systems given some issues introduced by either speech recognition or speech signal related issues. But in UT-opinion the question are more natural and things that people come across in their everyday life.

IX. CONCLUSION

In this study, a new method for recognizing sentiment in audio has been proposed. The analysis shows that overall sentiment (both in audio and text) is governed by few sentiment bearing terms. In order to exploit this fact, a new method that uses Key- word Spotting (KWS) to search for sentiment bearing terms in audio has been proposed. By focussing on the terms that impact decision and ignoring non-sentiment bearing words/phrases, the overall system is more immune to speech recognition errors. Additionally, a new method to create the sentiment bearing key- word list for KWS has also been proposed. The method uses an iterative methodology to automatically extract sentiment bear- ing keywords from text. Using this method, we are able to build more practical systems that utilize equal to or less than 48 K keywords. Additionally, a new method for sentiment scoring that combines keyword spotting likelihood (or confidence) into Maximum Entropy likelihood computation has also been pro- posed. Furthermore, a new corpus for audio sentiment evaluation has been collected and presented in this study. The new corpus is called UT-Opinion and to the best of our knowledge, is one of its kind for audio based sentiment detection. Finally, we have pre- sented the evaluation of the proposed system on YouTube and UT-Opinion corpora. The new method has been compared to a baseline system that uses raw transcripts from ASR and feeds it to text based sentiment classifier. Our experimental results show that the new method outperforms the baseline system by reduc- ing the error rate by 19% relative in YouTube, and 8% relative in UT-Opinion. While the new method improves upon audio based sentiment detection, there is room for further improvement. For example, addressing the traditional robustness problems of ASR (accent, noise, etc.) can have significant impact of performance. Another areas of work could focus on using pure speech features to augment lexical information drawn for speech recognition to do speech sentiment detection.

REFERENCES

[1] S. Johnson, “Internet changes everything: Revolutionizing public par- ticipation and access to government information through the Internet,” Administ. Law Rev., vol. 50, no. 2, pp. 277–337, Spring 1998.

[2] D. Chrysanthos, “Strategic manipulation of internet opinion forums: Implications for consumers and firms,” Manage. Sci., vol. 52, no. 10, pp. 1577–1593, 2006.

[3] M. Wollmer et al., “Youtube movie reviews: Sentiment analysis in an audio-visual context,” IEEE Intell. Syst., vol. 28, no. 3, pp. 46–53, May/Jun. 2013.

[4] J. Naughton, “The internet: Is it changing the way we think?,” The Gaur- dian, Aug. 14, 2010.

[5] G. Mishne and N. S. Glance, “Predicting movie sales from blogger sen- timent,” in Proc. AAAI 2006 Spring Symp. Comput. Approaches Anal. Weblogs, 2006, pp. 155–159.

[6] L. Barbosa and J. Feng, “Robust sentiment detection on twitter from biased and noisy data,” in Proc. Int. Conf. Comput. Linguistics (COLING-2010). 2010, pp. 36–44.

[7] E. Cambria, N. Howard, Y. Xia, and T. S. Chua, “Computational intelli- gence for big social data analysis,” IEEE Comput. Intell. Mag., vol. 11, no. 3, pp. 8–9, Aug. 2016.

[8] E. Cambria, B. Schuller, Y. Xia, and B. White, “New avenues in knowledge bases for natural language processing,” Knowl. Based Syst., vol. 108(C), pp. 1–4, 2016.

[9] M. Bautin, L. Vijayarenu, and S. Skiena, “International sentiment analysis for news and blogs,” in Proc. Int. AAAI Conf. Weblogs Social Media, 2008, pp. 19–26.

[10] I. Becker and V. Aharonson, “Last but definitely not least: On the role of the last sentence in automatic polarity-classification,” in Proc. ACL 2010 Conf. Short Papers, 2010, pp. 331–335.

[11] Z. Zhai, B. Liu, H. Xu, and P. Jia, “Clustering product features for opinion mining,” in Proc. 4th ACM Int. Conf. Web Search Data Mining, Hong Kong, China, Feb. 2011, pp. 347–354.

[12] L. Kaushik, A. Sangwan, and J. H. L. Hansen, “Sentiment extraction from natural audio streams,” in Proc. 2013 IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 8485–8489.

[13] L. Kaushik, A. Sangwan, and J. H. L. Hansen, “Automatic sentiment extraction from YouTube videos,” in Proc. 2013 IEEE Workshop Automat. Speech Recognit. Understanding, 2013, pp. 239–244.

[14] B. Pang and L. Lee, 2005, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” in Proc. 43rd Annu. Meeting Assoc. Comput. Linguistics, 2005, pp. 115–124.

[15] E. Kouloumpis, T. Wilson, and J. Moore, “Twitter sentiment analysis: The good the bad and the omg!,” Proc. 5th Int. AAAI Conf. Weblogs Social Media, 2011, vol. 11, pp. 538–541.

[16] A. Bifet and E. Frank, “Sentiment knowledge discovery in twitter streaming data,” in Discovery Science. Berlin, Germany: Springer, 2010, pp. 1–15.

[17] H. Saif, Y. He, and H. Alani, “Semantic sentiment analysis of Twitter,” Proc. 11th Int. Conf. Semantic Web 2012, pp. 508–524.

[18] A. Go, L. Huang, and R. Bhayani, “Twitter sentiment analysis,” Entropy, vol. 17, pp. 252–269, 2009.

[19] A. Berger, D. Pietra, and D. Pietra, “A maximum entropy approach to natural language processing,” Comput. Linguistics, vol. 22, pp. 39–71, 1996.

[20] X. Fei, H. Wang, and J. Zhu, “Sentiment word identification using the maximum entropy model,” in Proc. 2010 Int. Conf. Natural Lang. Process. Knowl. Eng., Aug. 2010, pp. 1–4.

[21] A. Ratnaparkhi, “A simple introduction to maximum entropy models for natural language processing,” University of Pennsylvania, Philadelphia, PA, USA, Tech. Rep. IRCS-97-08, 1997.

[22] C. Lin and Y. He, “Joint sentiment/topic model for sentiment analysis,” in Proc. ACM Int. Conf. Inform. Knowl. Manage., 2009, pp. 375–384.

[23] B. Liu, Sentiment Analysis and Opinion Mining. Williston, VT, USA: Morgan & Claypool publishers, 2012

[24] Q. Mei, X. Ling, M. Wondra, H. Su, and C. Zhai, “Topic sentiment mixture: Modeling facets and opinions in weblogs,” in Proc. Int. Conf. World Wide Web., 2007, pp. 171–180.

[25] Y. Lu and C. Zhai, “Opinion integration through semi-supervised topic modeling,” in Proc. Int. Conf. World Wide Web, 2008, pp. 121–130.

[26] S. Moghaddam and E. Martin, “ILDA: Interdependent LDA model for learning latent aspects and their ratings from online product reviews,” in Proc. 34th Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2011, pp. 665–674.

[27] A. Mukherjee and B. Liu, “Mining contentions from discussions and debates,” in Proc. 18th ACM SIGKDD Int. Conf. Knowl. Discovery and Data Mining, pp. 841–849, 2012.

[28] M. Hu and B. Liu, “Mining and summarizing customer reviews,” in Proc. 10th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2004, pp. 168–177. ACM.

[29] W. Zhao, J. Jing, Y. Hongfei, and L. Xiaoming, “Jointly modeling aspects and opinions with a MaxEnt-LDA hybrid,” in Proc. 2010 Conf. Empirical Methods Natural Lang. Process., 2010, pp. 56–65.

[30] B. Pang and L. Lillian, “Opinion mining and sentiment analysis,” Found. Trends Inf. Retrieval, vol. 2, no. 1/2, pp. 1–135, 2008.

[31] B. Snyder and R. Barzilay, “Multiple Aspect Ranking Using the Good Grief Algorithm,” in Proc. Human Lang. Technol. 2007, Conf. North Amer. Chapter Assoc. Comput. Linguistics, 2007, pp. 300–307.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

1678 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 8, AUGUST 2017

[32] M. Joshi and C. Penstein-Ros, “Generalizing dependency features for opinion mining,” in Proc. ACL-IJCNLP, 2009 Conf. Short Papers. Assoc. Comput. Linguistics, 2009, pp. 313–316.

[33] X. Ding, B. Liu, and P. S. Yu, “A holistic lexicon-based approach to opinion mining,” in Proc. 2008 Int. Conf. Web Search Data Mining, 2008, pp. 231–240.

[34] L. Qian, Z. Gao, B. Liu, and Y. Zhang, “Automated rule selection for aspect extraction in opinion mining,” in Proc. Int. Joint Conf. Artif. Intell.. 2015, pp. 1291–1297.

[35] B. Pang and L. Lee, “A sentimental Education: Sentiment analysis using subjectivity summarization based on minimum cuts,” in Proc. 42nd Ann. Meeting Assoc. Comput. Linguistics, Assoc. Comput. Linguistics, 2004, pp. 271–278.

[36] E. Cambria and A. Hussain, Sentic Computing: Techniques, Tools, and Applications. New York, NY, USA: Springer, 2012.

[37] W. Pedrycz and Shyi-Ming Chen, Sentiment Analysis and Ontology En- gineering. New York, NY, USA: Springer, 2016

[38] V. Hatzivassiloglou and K. R. McKeown, “Predicting the semantic ori- entation of adjectives,” in Proc. 35th Annu. Meeting Assoc. Comput. Linguistics 8th Conf. Eur. Chapter Assoc. Comput. Linguistics, 1997, pp. 174–181.

[39] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs Up? sentiment classifi- cation using machine learning techniques,” in Proc. Ann. Conf. Empirical Methods Natural Lang. Process., Assoc. Comput. Linguistics, 2002, paper 7986.

[40] X. Bai, “Predicting consumer sentiments from online text,” Decision Sup- port Syst., vol. 50, pp. 732–742, 2011.

[41] A. Popescu and E. Orena, “Extracting product features and opinions from reviews,” in Natural Language Processing and Text Mining. London, U.K.: Springer, 2007, pp. 9–28.

[42] P. Turney, “Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews,” in Proc. 40th Ann. Assoc. Comput. Linguistics, Assoc. Comput. Linguistics, 2002, pp. 417–424.

[43] J. Kamps, M. Marx, R. J. Mokken, M. De Rijke, “Using wordNet to measure semantic orientation of adjectives,” in Proc. 4th Ann. Int. Conf., 2004, pp. 1115–1118.

[44] R. Feldman, “Techniques and applications for sentiment analysis,” Com- mun. ACM, vol. 56, no. 4, pp. 82–89, 2013.

[45] J. Brooke, M. Tofiloski, and M. Taboada, “Cross-Linguistic sentiment Analysis: From english to spanish,” in Proc. Int. Conf. Recent Adv. Natural Lang. Process., 2009, pp. 50–54.

[46] X. Wan, “Using bilingual knowledge and ensemble techniques for unsu- pervised Chinese sentiment analysis,” in Proc. Conf. Empirical Methods Natural Lang. Process.. Assoc. Comput. Linguistics, 2008, pp. 553–561.

[47] H. Yu and V. Hatzivassiloglou, “Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences,” in Proc. 2003 Conf. Empirical Methods Natural Lang. Process. Assoc. Comput. Linguistics, 2003, pp. 129–136.

[48] E. Rilof and J. Wiebe, “Learning extraction patterns for subjective expres- sions,” in Proc. 2003 Conf. Empirical Methods Natural Lang. Process. Assoc. Comput. Linguistics, 2003, pp. 105–112.

[49] R. Socher et al., “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2013, vol. 1631, pp. 1631–1642.

[50] T. Wilson, J. Wiebe, and P. Hoffmann, “Recognizing contextual polarity in phrase-level sentiment analysis,” in Proc. Conf. Human Lang. Technol. Empirical Methods Natural Lang. Process., 2005, pp. 347–354.

[51] X. Ding, B. Liu, and L. Zhang, “Entity discovery and assignment for opinion mining applications,” In Proc. 15th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2009, pp. 1125–1134.

[52] M. Ruiz and P. Srinivasan “Hierarchical neural networks for text catego- rization (poster abstract),” in Proc. 22nd Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 1999, pp. 281–282.

[53] L. Kaushik, A. Sangwan, and J. H. L. Hansen, “Automatic audio senti- ment extraction using keyword spotting,” in Proc. INTERSPEECH, 2015, pp. 2709–2713.

[54] N. H. Tou, G. Wei, and L. Kok, “Feature selection, perceptron learning, and a usability case study for text categorization,” in Proc. ACM SIGIRConf, 1997, pp. 67–73.

[55] [Online]. Available: www.cs.uic.edu/∼liub/FBS/sentiment-analysis.html [56] (2004). [Online]. Available: http://www.cs.uic.edu/∼liub/FBS/opinion-

lexicon-english.rar [57] J. McAuley, C. Targett, Q. Shi, and A. D. Hengel, “Image-based recom-

mendations on styles and substitutes,” in Proc. 38th Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2015, pp. 43–52.

[58] J. McAuley, R. Pandey, and J. Leskovec, “Inferring networks of substi- tutable and complementary products,” in Proc. 21th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2015, pp. 785–794.

[59] M. Ganapathibhotla and B. Liu, “Mining opinions in compara- tive sentences,” in Proc. 22nd Int. Conf. Comput. Linguistics, 2008, pp. 241–248.

[60] N. Jindal and B. Liu, “Identifying comparative sentences in text docu- ments,” in Proc. 29th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2006, pp. 244–251.

[61] Amazon reviews downloader and parser, Feb. 12, (2015). [Online]. Avail- able: http://esuli.it

[62] K. Toutanova and C. D. Manning., “Enriching the knowledge sources used in a maximum entropy part-of-speech tagger” in Proc. Conf. Empirical Methods Natural Lang. Process. Very Large Corpora, 2000, pp. 63–70.

[63] K. Toutanova, D. Klein, C. Manning, and Y. Singer, “Feature-rich part-of- speech tagging with a cyclic dependency network,” in Proc. 2003 Conf. North Amer. Chapter Assoc. Comput. Linguistics Human Lang. Technol. 2003, pp. 252–259.

[64] G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, “The SEMAINE database: Annotated multi- modal records of emotionally coloured conversations between a person and a limited agent,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 5–17, Jan–Mar. 2011.

[65] [Online]. Available: www.youtube.com [66] L. Morency, R. Mihalcea, and P. Doshi, “Towards multimodal sentiment

analysis: Harvesting opinions from the web,” in Proc. 13th Int. Conf. Multimodal Interfaces, 2011, pp. 169–176.

[67] E. Cambria, B. Schuller, Y. Xia, and C. Havasi, “New avenues in opin- ion mining and sentiment analysis,” IEEE Intell. Syst., vol. 28, no. 2, pp. 15–21, Mar./Apr. 2013.

[68] N. Jindal and B. Liu, “Identifying comparative sentences in text docu- ments,” in Proc. 29th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2006, pp. 244–251.

[69] H. Cui, V. Mittal, and M. Datar, “Comparative experiments on sentiment classification for online product reviews,” in Proc. 21st Nat. Conf. Artif. Intell., 2006, vol. 6, pp. 1265–1270.

[70] A. Sangwan and J. H. L. Hansen, “Keyword recognition with phone con- fusion networks and phonological features based keyword threshold de- tection,” in Proc. 2010 Conf. Record 44th Asilomar Conf Signals, Syst. Comput., 2010, pp. 711–715.

[71] J. Keshet, D. Grangier, and S. Bengio, “Discriminative keyword spotting,” Speech Commun., vol. 51, no. 4, pp. 317–329, 2009.

[72] D. Povey et al., “The Kaldi speech recognition toolkit,” In IEEE 2011 workshop on automatic speech recognition and understanding, IEEE Sig- nal Processing Society, 2011.

[73] J. J. Godfrey, E. C. Holliman, and J. McDaniel, “SWITCHBOARD: Tele- phone speech corpus for research and development,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 1992, pp. 517–520.

[74] C. C. David, D. Miller, and K. Walker, “The fisher corpus: a resource for the next generations of speech-to-text,” in Proc. 4th Int. Conf. Lang. Resources Eval., 2004, pp. 69–71.

[75] (2011). [Online]. Available: http://ssli.ee.washington.edu/ssli/projects/ears/ WebData/web_data_collection.html

Lakshmish Kaushik was born in Hassan, India. He received the B.E. and M.A.Sc. degrees in elec- trical engineering from Visveswaraiah Technologi- cal University (VTU), Karnataka, India, INRS-EMT (University du Quebec), Montreal, QC, Canada, re- spectively, and is currently working toward the Ph.D. degree at the University of Texas at Dallas, Richard- son, TX, USA. He worked as Research Assistant at MILE Lab, Indian Institute of Science, developing Text-to-Speech systems in Indian Languages. During 2010–2012, he worked for Nuance Communications

where he focused on research and development of Natural Language Under- standing systems in Mobile environments. He interned with the Speech Recog- nition group at Sony Research, Foster city, in 2014. Since 2012, he has been a part of The Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, Richardson, TX, USA. His research interests include automatic speech recognition, Keyword Recognition, Automatic Sentiment analysis, nat- ural language understanding, and human behavior analysis using speech.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.

KAUSHIK et al.: AUTOMATIC SENTIMENT DETECTION IN NATURALISTIC AUDIO 1679

Abhijeet Sangwan (S’04–M’09) was born in Mysore, India. He received the B.E., M.A.Sc., and Ph.D. degrees in electrical engineering from Visveswaraiah Technological University (VTU), Karnataka, India, in 2002, Concordia University, Montreal, QC, Canada, in 2006, and the University of Texas at Dallas, Richardson, TX, USA, respectively. He is currently serving as the Chief Technology Of- ficer of Speetra Inc., a company he co-founded. His research interests include automatic speech recog- nition, keyword recognition, sentiment analysis and

accent assessment systems.

John H. L. Hansen (S’81–M’82–SM’93–F’07) re- ceived the Ph.D. and M.S. degrees in electrical engineering from Georgia Institute of Technology, Atlanta, Georgia, in 1988 and 1983, and the B.S.E.E. degree from Rutgers University, College of Engineer- ing, New Brunswick, NJ, USA, in 1982. He received the honorary degree Doctor Technices Honoris Causa from Aalborg University, Aalborg, Denmark, in April 2016, in recognition of his contributions to speech signal processing and speech/language/hearing sci- ence. He joined University of Texas at Dallas (UT-

Dallas), Erik Jonsson School of Engineering and Computer Science in 2005, where he currently serves as Jonsson School Associate Dean for Research, as well as a Professor of electrical engineering, the Distinguished University Chair in Telecommunications Engineering, and a joint appointment as a Professor in the School of Behavioral and Brain Sciences (Speech & Hearing). He previ- ously served as the Department Head of Electrical Engineering from August 2005 to December 2012, overseeing a +4x increase in research expenditures (4.5–22.3 M) with a 20% increase in enrollment along with hiring 18 additional T/TT faculty, growing UTDallas to the eighth largest EE program from ASEE rankings in terms of degrees awarded. At UTDallas, he established the Center for Robust Speech Systems (CRSS). Previously, he served as the Department Chair- man and a Professor of Department of Speech, Language and Hearing Sciences (SLHS), and a Professor in Department of Electrical and Computer Engineering, at the University of Colorado Boulder (1998–2005), where he co-founded and served as the Associate Director of the Center for Spoken Language Research. In 1988, he established the Robust Speech Processing Laboratory (RSPL) and continues to direct research activities in CRSS at UTDallas. He has been named IEEE Fellow (2007) for contributions in “Robust Speech Recognition in Stress and Noise,” International Speech Communication Association (ISCA) Fellow (2010) for contributions on research for speech processing of signals under adverse conditions, and received the Acoustical Society of Americas 25 Year Award (2010) in recognition of his service, contributions, and membership to the Acoustical Society of America. He is currently serving as the elected Vice- President of ISCA and a Member of the ISCA Board. He was also selected and is serving as the Vice-Chair on U.S. Office of Scientific Advisory Com- mittees (OSAC) for OSAC-Speaker in the voice forensics domain (2015-2017). Previously he served as a IEEE Technical Committee (TC) Chair and Member of the IEEE Signal Processing Society: Speech-Language Processing Techni- cal Committee (SLTC) (2005–2008; 2010–2014; elected IEEE SLTC Chairman for 2011–2013, Past-Chair for 2014), and elected ISCA Distinguished Lecturer (2011/2012). He has served as Member of IEEE Signal Processing Society Ed- ucational Technical Committee (2005–2008; 2008–2010); Technical Advisor to the U.S. Delegate for NATO (IST/TG-01); IEEE Signal Processing Society Distinguished Lecturer (2005/2006), Associate Editor for IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING (1992–1999), Associate Editor for IEEE SIGNAL PROCESSING LETTERS (1998–2000), Editorial Board Member for IEEE SIGNAL PROCESSING MAGAZINE (2001–2003); and Guest Editor (October 1994) for special issue on Robust Speech Recognition for IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING. He has served on Speech Communications Technical Committee for Acoustical Society of America (2000–2003), and pre- viously on ISCA Advisory Council. His research interests span the areas of digital speech processing, analysis and modeling of speech and speaker traits, speech enhancement, feature estimation in noise, robust speech recognition with emphasis on spoken document retrieval, and in-vehicle interactive systems for hands-free human-computer interaction. He has supervised 77 Ph.D./M.S. thesis candidates (40 Ph.D., 37 M.S./M.A.), received the 2005 University of Colorado Teacher Recognition Award as voted on by the student body, author/co-author of 640 journal and conference papers including 12 textbooks in the field of speech processing and language technology, coauthor of textbook Discrete- Time Processing of Speech Signals, (IEEE Press, 2000), co-editor of DSP for In-Vehicle and Mobile Systems (Springer, 2004), Advances for In-Vehicle and Mobile Systems: Challenges for International Standards (Springer, 2006), In- Vehicle Corpus and Signal Processing for Driver Behavior (Springer, 2008), and lead author of the report The Impact of Speech Under Stress on Military Speech Technology, (NATO RTO-TR-10, 2000). He also organized and served as the General Chair for ISCA Interspeech-2002, September 16–20, 2002, Co- Organizer and the Technical Program Chair for IEEE ICASSP-2010, Dallas, TX, USA, March 15–19, 2010, and the Co-Chair and Organizer for IEEE SLT- 2014, December 7–10, 2014 at Lake Tahoe, NV, USA.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:47:00 UTC from IEEE Xplore. Restrictions apply.