critical question

“YEAH RIGH SARCASM RECOGNITION FOR SPO

Joseph Tepperman1, David Traum2, a

1 Signal Analysis and Interpretation Laboratory 2 Institute for Creative Technologies, Uni

[email protected], [email protected]

ABSTRACT

The robust understanding of sarcasm in a spoken dialogue system requires a reformulation of the dialogue manager’s basic assumptions behind, for example, user behavior and grounding strategies. But automatically detecting a sarcastic tone of voice is not a simple matter. This paper presents some experiments toward sarcasm recognition using prosodic, spectral, and contextual cues. Our results demonstrate that spectral and contextual features can be used to detect sarcasm as well as a human annotator would, and confirm a long-held claim in the field of psychology – that prosody alone is not sufficient to discern whether a speaker is being sarcastic. Index Terms: dialogue, user modeling, sarcasm, speech acts

1. INTRODUCTION

Sarcasm, also called verbal irony, is the name given to speech bearing a semantic interpretation exactly opposite to its literal meaning. Often used to express humorous analysis or commentary, sarcasm differs from lying in that there is no intent to deceive on the speaker’s part [4]. Like a Grounding Act, sarcastic speech requires collaboration – its effect is dependent on the assumed intentions and mutual beliefs held by all dialogue participants [1].

In the realm of spoken dialogue systems, the use of sarcasm presents an interesting fly in the ointment. Because of its complex semantic nature, sarcasm demands that we rethink our fundamental assumptions behind user modeling, grounding strategies, and other methods of automatic spoken dialogue understanding [8]. Sarcastic speech violates at least one of Grice’s maxims for cooperative dialogue (“Do not say what you believe to be false”) and probably many other pragmatic constraints of any standard spoken dialogue model [2]. Sarcasm also occurs sufficiently frequently in casual conversation that some meeting recognition tasks have purposely chosen to account for it [3]. These reasons all suggest that sarcasm is an important area of inquiry to those working to formulate robust spoken dialogue models.

expr as it conv anno of seve whe [5] s has – an

reco eval class mos sarca dialo

2.1.

Insp from each four

Ack right the Ack A: O B: Y

Agre expl not u pres sarca A: A B: Y

1838

INTERSPEECH 2006 - ICSLP

T”: KEN DIALOGUE SYSTEMS

nd Shrikanth Narayanan1

, University of Southern California versity of Southern California .edu, [email protected]

Here we have restricted our investigation to the ession “yeah right” because of its succinctness as well s common usage (both sarcastically and otherwise) in ersational American English. Though they are not tated for sarcasm, the Switchboard and Fisher corpora

spontaneous two-party telephone dialogues contain ral hundred occurrences of this expression. Moreover, n used sarcastically “yeah right” is an example of what ays to be the most common type of sarcasm – one that a positive literal meaning but a negative semantic value d so may serve as an exemplar for other sarcastic forms. In this study we trained an automatic sarcasm gnizer using spectral, contextual, and prosodic cues. In uating the performance and approach to training such a ifier, we intended to learn what types of features are

t indicative of sarcasm’s presence, formulate how sm functions as a speech act, and recommend how a gue manager should respond to sarcasm, once detected.

2. CONTEXTUAL FEATURES

Sarcasm as a Speech Act

ired by work in the area of Grounding Acts [8], and careful analysis of the data, we chose to categorize

of our examples of “yeah right” as one of the following types of speech acts (with examples from Switchboard):

nowledgment – When a speaker uses a sincere “yeah ” by way of presenting evidence of understanding (in Grounding Acts sense), we regarded it as an

nowledgment. h, well that’s right near Piedmont. (1) eah right, right...

ement/Disagreement – This act is largely self- anatory and differs from Acknowledgment in that it’s sed to ground understanding of information previously

ented. Disagreement, of course, could only occur in the stic case. thorn in my side: bureaucratics. (2)

eah right, I agree.

September 17-21, Pittsburgh, Pennsylvania

Laughter Q/A Start End Pause Male

Sarcastic 0.73 0.10 0.57 0.43 0.07 0.23 Sincere 0.22 0.20 0.94 0.48 0.19 0.51

Table 1. Probability of each contextual feature given the class of speech.

Indirect Interpretation – This is for the type of speech act not directed at the dialogue partner but at a hearer not present, as in recounting a past use of a sarcastic “yeah right” by way of telling a story (Switchboard and Fisher actually had several examples of this). A: “…We have too many pets!” I thought, “Yeah right, come tell me about it!” You know? B: [laughter] (3)

Phrase-Internal – This is when the “yeah right” does not function as a singular speech unit unto itself, but is included as part of some larger speech act. A: Park Plaza, Park Suites? (4) B: Park Suites, yeah right across the street, yeah. This example may contain the adjacent words “yeah right” but the “yeah” and the “right” are disjunct, the “right” being more properly semantically paired with the word that follows it: “across.”

After hand-coding all Switchboard and Fisher occurrences, we noted that there were no sarcastic Acknowledgment or Phrase-Internal realizations, nor did we ever classify a sincere “yeah right” as Indirect Interpretation, so this built-in mutual exclusivity indicated that our chosen act taxonomy did lend itself well to the sarcasm task. But the inherent subjectivity of the coding scheme didn’t allow for necessarily clear-cut categorizations. Disambiguating an Agreement from an Acknowledgment could not always be easily or objectively done. Consider the following example:

A: They were never allowed in the house, you know… B: Uh-huh. Yeah, it would ruin them. [laughter] A: Yeah, right. [laughter] That’s what he said. (5) B: They say that it’ll ruin them.

Speaker A’s “Yeah, right” is used as sarcastic interpretation in agreement with B’s previous sarcastic comment, but A’s use of sarcasm also functions as evidence of understanding B’s commentary. We found the Switchboard and Fisher examples of sarcastic “yeah right” often functioned in this way: not only as humorous interpretation or commentary, but as a grounding act of sorts, a Request for Acknowledgment on the part of the hearer, or sometimes an Acknowledgment itself. This Acknowledgment usually came in the form of laughter or explanation of the joke (both of which we see in the above example) – at any rate, some demonstration of evidence of understanding the speaker’s sarcastic commentary.

2.2.

To o since the s

Laug hum eithe utter

Que Ack yet l to a least

Start the s serv sarca was expl mea

Paus a w with eithe (eve coul defin this

Gen than cont

“yea at th and conv

The char word utter tone and thos spee

1839

INTERSPEECH 2006 - ICSLP

Objective Cues

bjectively define intersecting marks of sarcasm’s (or rity’s) context and allow for a softer distinction among peech acts, we used the following feature set:

hter – An obvious choice, given sarcasm’s often orous but benevolent intentions. This included laughter r in the “yeah right” turn itself, or in the adjacent turns ed by either dialogue participant.

stion/Answer – Defining what constitutes an nowledgment might not be clear-cut, but what’s related ess subjective was the use of “yeah right” as the answer question, which seemed correlated with sincerity, at in many of the Switchboard and Fisher examples.

, End – Did the “yeah right” come at the start or end of peaker’s turn? Or was it perhaps an entire turn itself,

ing as both start and end? In the latter case, we found sm to be very unlikely: usually a sarcastic “yeah right” followed by some elaboration on the point, some

anation of the joke. That’s why these seemed like ningful features.

e – In these examples sarcasm was often used as part of itty repartee, a quick back-and-forth type of dialogue funny rejoinders again and again. If a “yeah right” was r preceded or followed by a pause in the conversation n a filled one), that seemed like a clear indication that it d not have been intended sarcastically. Here we usually ed a pause as a lull of longer than 0.5 seconds, though

also depended on the pace of the conversation.

der – [5] claims that sarcasm is used more often by men women. Therefore we included this among the binary

extual features.

In example (5), for instance, the speaker was female, the h right” did not come as the answer to a question, it fell e beginning (but not end) of the turn, it was preceded followed by laughter, and there was no pause in the ersation.

3. PROSODIC AND SPECTRAL FEATURES

following list of 19 prosodic features were chosen to acterize the relative “musical” qualities of each of the s “yeah” and “right” as a function of the whole ance, by way of parameterizing a perceived sarcastic of voice in terms of word pitch, energy, and duration, other features derived from them. They are similar to e features previously used for detection of non-native ch [6] and syllable stress [7].

1, 2: Average pitch in “yeah” and “right” individually, normalized by the average pitch over the whole utterance 3, 4: Duration of each word, normalized by the utterance’s duration 5, 6: Average energy in each word, normalized by the average energy of the utterance 7, 8, 9, 10: The number of rising and falling frames (positive and negative pitch slopes) in each word, normalized by the total frames in that word 11, 12: The number of inter-frame changes from rising to falling pitch slope within each word, also normalized by the number of frames in that word 13, 14: The overall pitch slope from the first to last frame of each word, normalized by the whole utterance’s overall pitch range 15: The overall pitch slope of the whole utterance, normalized by the whole utterance’s pitch range 16, 17: The pitch range for each word, normalized by the pitch range of the whole utterance 18, 19: The energy range for each word, normalized by the energy range of the whole utterance

Each context-independent recording of “yeah right” (just the “yeah right” without any other words) was automatically segmented on the word level using the “align” fuction of the University of Colorado’s Sonic Continuous Speech Recognizer. Then pitch and energy information was extracted using the Snack Sound Toolkit’s “pitch” function.

As for spectral features, the first 12 MFCC’s (plus Energy) and their delta and acceleration coefficients were extracted from the context-free recordings every 10 msec using a window length of 20 msec, and these were used to train two five-state Hidden Markov Models using embedded re-estimation in HTK. We also tried single-state GMMs, but the HMMs performed better. Because of the sparsity of data, this model training and decoding was done with a leave-one- out crossvalidation procedure. After decoding, we obtained log-likelihood scores representing the probability that our acoustic observations were drawn from each class, sarcastic and sincere. Lastly, these two likelihood scores and their ratio were then used as features (alongside the prosodic and contextual ones) in the final decision-tree-based classification algorithm.

4. DATA ANNOTATION

For purposes of annotation and feature extraction, two versions of the Switchboard and Fisher “yeah right” examples were generated. One was without the surrounding dialogue context, for purely prosody-based classification and annotation. Two human labelers annotated a statistically significant subset of these context-independent recordings for use as a baseline for comparison with an automatic classifier that uses only prosodic features. Inter-annotator agreement was very low: 52.73% (just above the chance agreement of 43.93% - not 50% because the coders did not use both classes in equal proportions), and our Kappa

stati [5] t alon sarca cont iden only spea this sarca sarca voic our with

dialo exam asce give wha surro the t a Ka disag these and of th “gro was beca desig

Our occu Swit sarca very find watc Swit betw com

class since spok 30 hypo the f sign Swit [5] t With

1840

INTERSPEECH 2006 - ICSLP

stic was also very low: 0.1569. But according to [4] and his is to be expected. Both sources argue that prosody e is not a very reliable indicator of sarcasm. Insofar as a stic tone of voice exists, a listener also relies heavily on

extual and, when available, visual information to tify sarcasm. [5] even goes so far as to claim that the sarcasm-dependent prosodic feature is energy (or, the

ker’s volume). One explanation that may account for is that perhaps the phrase “yeah right” is used stically so often that, even when it’s not meant stically, it might out of habit be spoken in that tone of

e, thus obscuring the true semantic content. At any rate, two labelers had a hard time telling the difference out any context. The other version of the data included the two or three gue turns before and after the “yeah right,” so that each ple’s “true” class (sarcastic or sincere) might be

rtained. [5] reports that well-trained human labelers, n visual, prosodic, and contextual cues, can agree on t constitutes sarcasm about 80% of the time. Given the unding context, our two annotators agreed 76.67% of

ime (above the new chance agreement of 66%) and with ppa statistic of 0.313, signifying fair correlation. Any reements between the two annotators were resolved and labels were taken as the true classes for the training

test set examples. Similarly, one annotator’s own coding e binary contextual features we took to be a sort of

und truth” for classifier training and evaluation. There no need to measure inter-annotator agreement on these use the features themselves were beyond dispute by n, and could be detected automatically fairly easily.

5. DATA STATISTICS

dataset comprised a total of 131 uninterrupted rrences of the phrase “yeah right” found in the chboard and Fisher corpora, 30 of which were used stically (about 23%). Though this might seem like a small percentage, consider that [5] was only able to

48 sarcastic utterances (not limited to “yeah right”) after hing a total of 128 half-hour talk shows. Also, the chboard and Fisher recordings are of conversations een strangers, and [5] claims that sarcasm is much more mon among close friends, understandably. The proportions of each contextual feature given the are enumerated in Table 1. So, for example, of the 101 re utterances of “yeah right,” only about 22% were en in the presence of laughter, compared to 73% of the sarcastic realizations, which seems to confirm the thesis behind using laughter as a contextual feature in irst place. This difference in proportions proved to be

ificant at the 95% confidence level. However, the chboard and Fisher data did not support the assertion in hat sarcasm is used more often by men than women.

95% confidence we found we could accept the

F-measure accuracy

prosodic 0.00 0.69

spectral 0.35 0.77

contextual 0.52 0.84

prosodic + spectral 0.43 0.76

prosodic + contextual 0.46 0.77

contextual + spectral 0.70 0.87

all 3 0.68 0.86

Table 2. Classification results, sorted by feature set.

alternative hypothesis that the proportion of sarcastic females exceeded that of males in our examples.

6. EXPERIMENTS AND RESULTS

Table 2 reports classification results based on a leave-one- out crossvalidation test procedure over various combinations of features. This decision tree classifier, trained on the C4.5 algorithm implemented in the Weka toolkit [9], was chosen because of its versatility in combining binary and continuous features.

Just as [5] predicted, and as expected by human agreement, using only prosodic features yielded the worst accuracy, though performing better than chance and better than the human annotators could when listening without context. Adding contextual and spectral features to the prosodics did improve accuracy, but leaving the prosody out and just using contextual and/or spectral features exceeded the inter-human agreement of 80% as reported by [5], and our own annotators’ agreement of 76.67%. So it seems that prosodic features are unnecessary, when paired with contextual or spectral cues.

Among the contextual features we found that laughter was the most important contribution. And among the prosodic features, it was the number of rising pitch frames in “yeah,” the average energy over “right,” and the number of rising pitch frames in “right” that most contributed to this classifier’s performance; this seems to support [5]’s result that energy contributes the most to the perception of a “sarcastic tone of voice,” but it does suggest that pitch is an important aspect of sarcastic prosody as well. This feature ranking is based on the information gain with respect to the given class, using leave-one-out crossvalidation again.

7. CONCLUSIONS

Designers of a dialogue manager or dialogue transcriber wishing to detect sarcasm would probably be best served to just ignore prosodic features entirely and concentrate on extracting the kinds of contextual features we used in this study. Most of them would be simple enough to detect automatically, either from acoustics (for laughter, gender, pause) or semantics and syntax (question/answer, start/end).

dialo ackn for m the j

these usin diffe for e pros acqu

Than labe

[1] D stati Artif [2] H Mor pp. 4 [3] D Agre Unla Edm [4] Inton Reco Littl sixth Dev [5] Dete Cues Nov [6] Sonm Usin Wor Proc [7] Stres Lear [8] Task Intel [9] mac Mor

1841

INTERSPEECH 2006 - ICSLP

As for handling the sarcasm once it’s detected, a gue agent ought to do what real humans do and owledge it. Either generate some synthetic laughter or, ore advanced agents, somehow point out that it “gets”

oke. Future work in this area could focus on generalizing

findings for other utterances besides “yeah right”; g a finer-grain taxonomy to accommodate subtle rences in the types of sarcasm (good-natured vs. biting, xample); the incorporation of other, more meaningful

odic features, as well as visual cues; and, of course, iring more data.

8. ACKNOWLEDGMENTS

ks to Abe Kazemzadeh and Francesca Giancoli for ling some of the data.

9. REFERENCES

. G. Field and A. Ramsay. “Sarcasm, deception and ng the obvious: Planning dialogue without speech acts.” icial Intelligence Review 22(2): 149-177. 2004. . P. Grice. “Logic and Conversation.” In P. Cole and J.

gan (eds.), Syntax and Semantics, Vol. 3: Speech Acts, 1-58, New York: Academic Press, 1975. . Hillard, M. Ostendorf, and E. Shriberg. “Detection of ement vs. Disagreement in Meetings: Training with beled Data.” Proc. HLT-NAACL Conference, onton, Canada, May 2003. C. Nakassis and C. Snedeker. “Beyond sarcasm: ation and Context as Relational Cues in Children's gnition of Irony.” In A. Greenhill, M. Hughs, H.

efield, & H. Walsh (eds.), Proceedings of the Twenty- Boston University Conference on Language

elopment. Somerville, MA: Cascadilla Press. 2002. P. Rockwell. “Sarcasm on Television Talk Shows: rmining Speaker Intent Through Verbal and Nonverbal .” In A. Clark (ed.), Psychology of Moods. New York:

a. 2005. C. Teixeira, H. Franco, E. Shriberg, K. Precoda, K.

ez, “Evaluation of Speaker’s Degree of Nativeness g Text-Independent Prosodic Features,” in Proc. of the kshop on Multilingual Speech and Language essing, Aalborg, Denmark, 2001.

J. Tepperman and S. Narayanan, “Automatic Syllable s Detection for Pronunciation Evaluation of Language ners,” Proc. ICASSP’05, Philadelphia, 2005. D. Traum and E. Hinkelman, “Conversation Acts in -Oriented Spoken Dialogue,” In Computational ligence, 8(3): 575-599, 1992. I. H. Witten and E. Frank, “Data Mining: Practical hine learning tools and techniques,” 2nd Edition, gan Kaufmann, San Francisco, 2005.