project Ahmed
Auditory, Visual, and Auditory–Visual Perception of Emotions
by Individuals With Cochlear Implants, Hearing Aids, and
Normal Hearing
Tova Most
Chen Aviner
Tel-Aviv University
This study evaluated the benefits of cochlear implant (CI)
with regard to emotion perception of participants differing
in their age of implantation, in comparison to hearing aid
users and adolescents with normal hearing (NH). Emotion
perception was examined by having the participants identify
happiness, anger, surprise, sadness, fear, and disgust. The
emotional content was placed upon the same neutral sen-
tence. The stimuli were presented in auditory, visual, and
combined auditory–visual modes. The results revealed bet-
ter auditory identification by the participants with NH in
comparison to all groups of participants with hearing loss
(HL). No differences were found among the groups with
HL in each of the 3 modes. Although auditory–visual per-
ception was better than visual-only perception for the par-
ticipants with NH, no such differentiation was found among
the participants with HL. The results question the efficiency
of some currently used CIs in providing the acoustic cues
required to identify the speaker’s emotional state.
Spoken language communication comprises linguistic
information such as lexical items and syntactic pat-
tern, as well as vocal nonverbal information, such as
the speaker’s emotional state with respect to the topic
or the listener (Mozziconacci, 2002). Both linguistic
and nonverbal information are crucial for understand-
ing social interaction (Rieffe & Terwogt, 2000). This
study focuses upon nonverbal information, specifically
upon the emotional state of the speaker.
Perception of the speaker’s emotional state is based
upon auditory as well as visual cues. Banse and
Scherer (1996) reported that adults with normal hear-
ing (NH) were able to reliably identify the speaker’s
emotional state based solely upon auditory cues. For
example, anger is characterized by a high average fun-
damental frequency, large range of fundamental fre-
quency, as well as a high average intensity and a high
rate of speech. In comparison, sadness is characterized
by a low average fundamental frequency, low average
intensity, long duration, and a slow rate of speech.
Thus, auditory perception of the emotion is based
primarily upon the following acoustic cues: fundamen-
tal frequency characteristics (such as the average, the
range, and the shape and the direction of the intona-
tion contour), the average intensity, and the intensity
changes along the utterance; the energy distribution in
the spectral range, specifically the ratio between the
energy in the high frequencies and the energy in
the low frequencies; the formant location; and finally,
the duration of the production or the rate of speech
(Banse & Scherer, 1996; Williams & Stevens, 1972).
Murray and Arnott (1993) indicated that the most
significant contribution to emotion perception was
the fundamental frequency changes along the utter-
ance, after which comes duration, and lastly, intensity.
Previous research reported great variations in ac-
curate auditory perception of the different emotions.
In general, anger and sadness were found to be easier
to identify, whereas surprise and disgust were very
difficult to perceive (Banse & Scherer, 1996; Most,
Wiesel, & Zaychik, 1993; Pereira, 2000).
Most of the research that investigated emotion
perception focused upon perception through the vi-
sual mode, or more specifically, upon facial expres-
sions. Based upon findings of studies conducted in
No conflicts of interest were reported. Correspondence should be sent to
Tova Most, School of Education, Tel Aviv University, Tel Aviv 69978,
Israel (e-mail: tovam@post.tau.ac.il)
� The Author 2009. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org
doi:10.1093/deafed/enp007
Advance Access publication on April 27, 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
different countries on the perception of emotions
based on facial expressions, Ekman (1999) indicated
that there are universal visual cues for the perception
of fear, sadness, happiness, anger, disgust, and sur-
prise. The main visual cues for identifying facial
expressions center upon the eyes and the mouth. Fear,
anger, and sadness are best identified through the eyes,
whereas happiness and disgust are best identified
through the mouth. Surprise is equally identified
through the eyes or the mouth (Calder, Young, Keane,
& Dean, 2000; Sullivan, Ruffman, & Hutton, 2007).
There is a great variability as well in the accurate
perception of emotions through the visual mode. Pre-
vious research reported that happiness could be per-
ceived most accurately. Emotions such as fear,
disgust, and surprise were difficult to identify,
whereas anger and sadness were in-between (Calder
et al., 2000; Hosie, Gray, Russell, Scott, & Hunter,
1998; Montagne, Kessels, Frigerio, De Haan, &
Perret, 2005; Most et al., 1993; Wagner, McDonald,
& Manstead, 1986).
In general, in most social interactions, emotions
are transmitted through the auditory–visual mode
(Scherer & Ellgring, 2007). When comparing the
perception of emotions through each of the sensory
modes separately, it was found that in comparison to
the perception level of the auditory mode, emotions
were better perceived through the visual mode
(Hess, Kappas, & Scherer, 1988; Most et al., 1993;
Rigo & Liberman, 1989; Wallbott & Scherer, 1986).
Nevertheless, auditory cues are very important be-
cause they provide information in situations where
facial information is not available (Shackman &
Pollak, 2005). Combined auditory–visual cues al-
ways resulted in better emotion perception when
compared to the presentation of solely auditory in-
formation (Most et al., 1993; Rigo & Liberman,
1989; Wallbott & Scherer, 1986); however, this per-
ception was not always better than solely visual per-
ception. For example, Most et al. (1993) reported
better emotion perception of adults with
NH through the auditory–visual mode, when com-
pared to any of the other modes used alone, whereas
others, such as Wallbott and Scherer (1986),
reported similar perception through the auditory–
visual mode and the visual mode alone.
Emotion Perception by Individuals With
Hearing Loss Using Hearing Aids
As a result of sensorineural hearing loss, many indi-
viduals may have difficulties perceiving the spoken
signal in general, as well as difficulties perceiving
auditory nonverbal cues of emotions. Difficulties
with perceiving information about the emotional
state of the speaker may lead to a lack of awareness
of the individual’s impact upon others, a lack of
empathy, and social skills that are not adapted to
the situation (Mellon, 2000). Much of the acoustic
information about emotions is located in the low-
frequency range, and many individuals with hearing
loss have residual hearing in this range. Nevertheless,
sensorineural hearing loss negatively affects psycho-
acoustic abilities such as frequency resolution, fre-
quency discrimination, or time resolution, all of
which are necessary in order to accurately perceive
emotional information (Moore, 1996). Studies on the
auditory perception of emotions by children, youth,
and adults with hearing loss who use hearing aids
(HAs) reported lower performance in comparison
to individuals with NH (Most et al., 1993; Oster &
Risberg, 1986; Rigo & Liberman, 1989; Shinall,
2005). Rigo and Liberman (1989) reported on nega-
tive correlation between the identification of emo-
tions and the degree of hearing loss in the lower
frequencies, whereas on the other hand Most et al.
(1993), did not report such correlation. All the per-
sons with hearing loss who participated in this study
received low auditory scores in emotion perception,
without any correlation to the degree of hearing loss.
It should be noted, however, that the participants in
this study had severe and profound hearing loss,
whereas those examined by Rigo and Liberman had
a wide range of hearing loss from mild and moderate
to severe. Most et al. reported a similar hierarchy
with regards to the perception of different emotions
through the auditory mode by youth with and with-
out hearing loss.
Previous studies on the perception of emotions
through the visual mode by people with hearing loss
provided different results. Although some reported on
performance similar to that of individuals with NH
(Dyck, Farrugia, Shochet, & Holmes-Brown, 2004;
450 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
Hosie et al., 1998; McCullough et al., 2005; Rollman
& Harrison, 1996; Weisel, 1985), others reported that
the individuals with hearing loss also exhibited lower
performance through the visual mode (Bachara,
Raphael, & Phelan, 1980; Dyck & Denver, 2003; Most
et al., 1993; Weisel & Bar-Lev, 1992). Most of the
research findings did not report any significant corre-
lation between the degree of hearing loss and the abil-
ity to accurately identify facial expressions (Dyck &
Denver, 2003; Most et al., 1993). The lower perfor-
mance of the individuals with hearing loss when per-
ceiving visual emotional information was explained by
the fact that the ability to perceive emotions develops
in the context of spoken language. When the infant
watches facial expression while hearing matching vo-
calization, he/she can more easily identify the resul-
tant emotion (Walker-Andrews & Lennon, 1991). The
development of social knowledge and social skills and
the acquisition of theory of mind in a child with hear-
ing loss who is in a hearing environment and who is not
exposed to a rich natural spoken language might be
delayed (Peterson & Siegal, 1995; Weisel & Bar-Lev,
1992). Another explanation might be the fact that
individuals who communicate in spoken language
might focus upon the mouth in order to lip-read,
and therefore, they might miss information around
the eyes that might be relevant to emotions (Rigo &
Liberman, 1989).
Just as with individuals with NH, those with
hearing loss exhibited better perception of emotions
through the visual mode and the combined auditory–
visual mode than the perception through the auditory
mode. However, in contrast to hearing individuals,
the level of emotion perceived by individuals with
hearing loss through the combined auditory–visual
mode did not exceed that of emotions perceived
through the visual mode (Most et al., 1993; Rigo &
Liberman, 1989). In other words, they did not benefit
from the addition of auditory information to the
visual mode.
Emotion Perception by Individuals With
Hearing Loss Using Cochlear Implants
The above section reports on emotion perception by
individuals with hearing loss who were using HAs.
Cochlear implant (CI) technology has opened up re-
habilitation options for the use of spoken language
among individuals with severe and profound hearing
loss. The use of the CI has shown that it increases
the audibility of the speech signal and consequently
enables better speech perception by children using
CI compared to those with a similar degree of
hearing loss but who use HAs (Blamey et al.,
2001; Calmels et al., 2004; Gestoettner, Hamzavi,
Egelierlier, & Baumgartner, 2000; Meyer, Svirski,
Kirk, & Miyamoto, 1998; Mildner, Sindija, &
Zrinski, 2006). The different studies report on a
great difference in an individual’s performance as a
result of diverse variables, such as age of implanta-
tion (Oh et al., 2003; Taitelbaum-Swead et al., 2005),
the duration of CI use (Kishon-Rabin al., 2002), and
the onset of deafness (Geier, Barker, Opie, & Fisher,
2000). Most such research, however, reported the
perception of segmental features of speech.
Much less attention has been given to research on
the perception of suprasegmental features, and the
results attained did not necessarily favor the
CI (Carney, Kienle, & Miyamoto, 1990; Most &
Peled, 2007; Peng, Tomblin, & Turner, 2008; Waltzman
& Hochberg, 1990; Wu & Yang, 2003). For example,
Waltzman and Hochberg (1990) found that children
with Nucleus 22-channel CI and children with
HAs performed well in perceiving word emphasis and
pitch changes. Boothroyd and Eran (1990) reported that
perception of syllable number by children using the
Nucleus CI did not significantly differ from that of
children with HAs, but that children with HAs did
perform better at perceiving intonation. In addition,
Most and Peled (2007) reported on poorer perception
of syllable stress and intonation by children using CI
compared to children with severe and profound hearing
loss using HA.
Researchers have explained the poor CI perfor-
mance in perceiving suprasegmental features by sug-
gesting that the implant may not provide sufficient
information on the changes in fundamental fre-
quency, which are an important acoustic cue for the
perception of suprasegmental features. For example,
O’Halpin, Falkoner, Rosen, and Viani (2006) sug-
gested that perception of word emphasis via the
CI Nucleus 24 (with Spectral Peak and Advanced
Perception of Emotions by Individuals With CIs, HA, and Normal Hearing 451
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
Combination Encoder) does not rely upon changes in
fundamental frequency but rather upon duration and
intensity cues. Their methodology consisted of using
synthetic speech stimuli with controlled changes for
each of the acoustic parameters (amplitude, duration,
and fundamental frequency).
Thus, in many current CI systems, the place and
the temporal pitch-encoding mechanisms are inade-
quate, resulting in difficulties when transferring infor-
mation salient for the perception of suprasegmental
features of speech (Carroll and Zeng, 2007).
Just as with the suprasegmental features, auditory
perception of the emotional state is cued through
changes in fundamental frequency along the utter-
ance, as well as duration and intensity cues (Banse
& Scherer, 1996). Despite the importance of emo-
tional content perception for successful communica-
tion, only a few research studies have examined this
performance by individuals with CI. These research
studies examined auditory perception in postlin-
gually deafened adults who use CI, when compared
to hearing individuals (Luo, Fu, & Galvin, 2007;
Pereira, 2000; Peters, 2006). They reported lower
performance in comparison to the hearing controls
and a great variation among participants with CI. In
addition, after the normalization of the stimuli (con-
trolling the intensity cues of the various emotions),
the performance of the participants with CI de-
creased. In other words, they probably relied upon
their perception of intensity cues, and once these
cues were controlled, their performance decreased.
It should be noted that the normalization did not
affect the performance of the control hearing partic-
ipants in the same manner (Luo et al., 2007; Pereira,
2000).
In summary, the ability to perceive the emotional
state of a speaker is very important in communication
interaction. Because individuals with severe and pro-
found hearing loss miss a great deal of the verbal
content, they rely more heavily on nonverbal cues
during interaction. Previous research reported diffi-
culties in the auditory perception of emotions by
individuals with hearing loss. Only a few studies were
performed with individuals using CI, and these
examined the auditory perception of postlingually
deafened adults. These studies compared their per-
formance to that of hearing individuals, but no com-
parison was carried out with individuals wearing
HAs. In addition, no comparison was performed with
regard to perception via the different sensory modes.
The purpose of this article was to evaluate emo-
tions perception by individuals using CI (implanted
at different ages) versus individuals with HAs and
individuals with NH. Emotion perception was eval-
uated through solely the auditory mode, the visual
mode, and the combined auditory–visual mode. We
hypothesized that the auditory emotion perception of
individuals with NH would be superior to that of all
participants with hearing loss. We wanted to examine
whether CI users would perform differently from
HA users in auditory emotion perception. We hy-
pothesized that the early-CI group would perform
better than the late-CI group through both the audi-
tory and the auditory–visual modes. We further hy-
pothesized that individuals with NH would perform
better through the auditory–visual mode in compar-
ison to the visual mode. We wanted to examine
whether individuals with hearing loss—both CI
and HA users—would perform better through the
auditory–visual mode, in comparison with each of
the sensory modes alone. Finally we wanted to exam-
ine the perception of specific emotions through the
different modes by all the participants.
Materials and Methods
Participants
The participants consisted of 40 individuals aged 10–
17 years. Thirty participants had prelingual binaural
sensorineural hearing loss with no other disabling
condition. All had been educated in the regular ed-
ucation system and used spoken language as their
means of communication. Ten participants had NH.
The 30 participants with hearing loss were divided
into three groups of 10 participants each:
1. Ten participants (four males and six females)
aged 10:10–15:7 (M 5 13:9, SD 5 1:7) were CI users
who had been implanted before the age of 6 (range 5
2:6–5:6 years, M 5 3:11, SD 5 1:3) (early-CI group).
The duration of CI usage ranged between 6:9 and 13:1
years (M 5 9:7, SD 5 2.0). The hearing loss of all the
452 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
participants had been detected early when they were
3–24 months old. All of them had used HAs prior to
the implantation.
2. Ten participants (four males and six females)
aged 10–17:6 (M 5 15:1, SD 5 2:7) were CI users
who had been implanted after the age of 6 (range 5
6:2–16:2 years, M 5 11:4, SD 5 3:87) (late-
CI group). The duration of CI usage ranged between
1 and 6:9 years (M 5 3:8, SD 5 2:10). The hearing
loss of all the participants had been detected when
they were 3–38 months old. All of them had used
HAs before implantation.
Age 6 years (age of implantation) was used as the
cutoff age between the early- and late-CI groups on
the basis of previous reports showing significant dif-
ferences on many variables between children who
were implanted before 5 years of age and those
implanted after the age of 7 (El-Hakim et al., 2001;
Harrison, Gordon, & Mount, 2005; Seung-Ha et al.,
2003).
1. Ten participants (five males and five females)
aged 10:2–15:9 (M 5 13:10, SD 5 1:11) had severe
to profound binaural hearing loss (range 5 73–110
dBHL, M 5 88, SD 5 9.6, in the better ear) and
were HA users (the HA group). All used digital HAs
on both ears. The hearing loss of the participants had
been detected when they were 3 months to 5:2 years
old.
Detailed information regarding the participants’
CI-type processors and HAs appear in the Appendix.
The fourth group comprised 10 participants (four
males and six females) aged 10:5–16:10 (M 5 14:10,
SD 5 1:11) with hearing within the normal range,
with no history of hearing problems or any other
disabling condition.
All the participants had received a score above
75% on a written emotion vocabulary test (Weisel
& Bar-Lev, 1992) in order to ensure that they were
all familiar with the emotions, both linguistically and
conceptually.
Instrument
This study used an Identification of Emotion Test
(IET) developed by Most and Weisel (Most et al.,
1993). A complete description of the test construc-
tion is available in Most et al. (1993) who examined
adolescents with hearing loss using HAs as well as
adolescents with NH. The test had previously been
used with adolescents with learning disorders as well
(Most & Greenbank, 2000). The test was found to be
valid and reliable in both studies. The test includes
a total of 36 video-recorded items—six presentations
of each of the following emotions: anger, fear, sad-
ness, happiness, disgust, and surprise. Each of the
emotions is expressed through the use of the same
neutral sentence ‘‘I am going out now and I’ll be back
later.’’ Thus, the sentence’s verbal content remains
the same throughout the test and differences in emo-
tional content are expressed by auditory and visual
nonverbal cues. The emotions are produced by a pro-
fessional actor using normal voice and speech artic-
ulation. Each item lasts 2–3 s, and there is a 10-s
break between each of the items. Praat software was
used to conduct an acoustic analysis on the test items.
Table 1 presents the average fundamental frequency
(F0), the SD, the F0 range, and the average duration
of the six emotions.
Procedure
The participants with hearing loss were recruited via
the SHEMA Organization for the Education and Re-
habilitation of Children and Youth with Hearing Im-
pairment. SHEMA is a nonprofit association that
serves school-age children (aged 7–18 years) with
hearing loss. SHEMA supports the children and
their teachers during school hours and in extracur-
ricular activities. The participants with NH were
recruited through personal relatives and their
Table 1 Average values of acoustic parameters of the
stimuli
Average (Hz)
Range (Hz)
SD (Hz)
Utterance time (s)
Happiness 131 121 33 2.145
Anger 144 132 40 2.784
Surprise 163 188 54 2.29
Sadness 113 60 16 3.362
Fear 143 110 30 2.75
Disgust 111 85 21 3.03
Perception of Emotions by Individuals With CIs, HA, and Normal Hearing 453
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
friends. Signed consent forms were obtained from all
the parents.
The examiner (C.A.) held three individual ses-
sions with each participant in a quiet room. Each
session lasted approximately 20 min. The sessions
were a few days apart from each other. At the begin-
ning of the first session, the examiner presented
a written emotion vocabulary test, which contained
36 items describing an event that triggers a specific
emotion. For example, ‘‘Ruthy had a birthday party
and many friends came to her party. Ruthy felt..’’
The participant was asked to select the appropriate
emotion from four alternatives: happy, sad, angry,
and fear. This test was administered in order to en-
sure that all the participants were familiar with the
concept of the emotions and served as a criterion for
participating in the study. Next, the examiner exam-
ined the sensory aids and performed the Ling 6-
sound test (Ling & Ling, 1978) to ensure that the
participants’ sensory aids were functioning and were
being used properly. Lastly, the examiner explained
the task and used a few items (differing from the test
items) for practice. The training items were pre-
sented through the audiovisual mode.
The IET was then presented. The test was pre-
sented through a different mode during each of the
sessions. There were three modes of presentation: an
auditory mode—during which the screen was dark-
ened and the participant listened to the test items at
a normal conversational level (70 dBSPL at the par-
ticipant’s seat); a visual mode—during which the par-
ticipant only watched the screen without sound; and
an auditory–visual mode—during which the partici-
pant watched the screen while listening. The order of
presentation modes was randomized. The test items
were presented through the use of a portable com-
puter (IBM T43) connected to two loudspeakers
(SONY). The participant was seated facing the
screen, positioned about 1 m away from the screen.
The two loudspeakers were located on both sides of
the computer at a 45-degree angle from the partic-
ipant’s seat. After each item of the test had been
presented, the participant was asked to mark his/
her answer on a response form on which the six
emotions were presented. The order of the items
was different for each presentation mode.
Results
Each of the participants in each of the four groups
received three emotion perception scores—one for each
presentation mode: auditory, visual, and auditory–
visual. Because closed-set responses were used with
the six alternatives, the scores were adjusted according
to the following formula by Boothroyd (1988) to take
guessing into account:
Corrected score
5 Uncorrected score 2% probability for correct answer
% probability for error
3100:
Group Results
Table 2 presents the corrected mean percent scores
and the standard deviations for each of the four
groups in each of the three modes.
One-way ANOVAs were conducted with repeated
measures. The independent variable was the group,
the dependent variable was the percent of correct
emotion identification, and the repeated measures
were the three modes. The analysis revealed a signif-
icant main effect of group, F(3, 36) 5 6.47, p , .01,
Table 2 Percent identification (after correction for guessing) (M and SD) of the four research groups in each mode of
presentation and the F values
Group Mode
Early implantees
Late implantees
Hearing aid users
Normal hearing
Variance analysis, F(3, 36)
Auditory—visual 74.66 (9.82) 79.33 (10.86) 79.99 (9.89) 86.33 (8.53) 1.66
Visual 76.66 (10.06) 81.33 (8.09) 77.66 (12.22) 79.99 (9.71) 0.31
Auditory 21.63 (9.28) 15.97 (8.86) 16.96 (9.57) 50.65 (7.61) 23.8***
***p , .001.
454 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
a significant main effect of mode, F(2, 35) 5 303.88,
p , .001, and significant interaction between group
and mode, F(6, 70)5 5.26, p , .001.
One-way ANOVAs were conducted for each of
the presentation modes. There were no significant
differences between the groups with regard to the
visual or the auditory–visual modes. Multiple con-
trasts (Einot & Gabriel, 1975) among the auditory
scores of the four groups revealed a significant dif-
ference. The hearing group received a significantly
better score than the other groups. None of the other
groups were significantly different from each other.
Nevertheless, an inspection of the mean scores
showed that the mean identification score of the
early-CI was better than that of the late-CI and the
HA participants.
Contrast analysis was conducted to compare
the percent of identification of the different modes
in each of the groups. Table 3 presents the F values
obtained.
As can be seen in Table 3, the analyses revealed
that all the groups showed significantly lower auditory
scores than visual and auditory–visual scores. The
auditory–visual score, however, was only better than
the visual score for the hearing group. The auditory–
visual and the visual scores were not significantly dif-
ferent for all groups of participants with hearing loss.
The level of significance was set at .017, due to the
three contrasts that were conducted (according to
Bonferroni correction).
Individual Results
Examination of the individual identification scores
of the participants in the four groups with regard
to each of the three modes (auditory, visual, and
auditory–visual) revealed great variations in the
identification performance of all the participants in
all modes. Nevertheless, although seven of the 10
participants in the NH group exhibited better audi-
tory–visual performance in comparison to the per-
formance of the visual mode, this was not the case
in the three groups of participants with hearing
loss. The three groups with hearing loss experi-
enced greater variations. Although some performed
better in the auditory–visual mode in comparison to
the visual mode, others showed better performance
in the visual mode in comparison to the auditory–
visual mode, and there were also others who
exhibited comparable performance in the two
modes.
Identification of Specific Emotions
One-way ANOVAs with repeated measures were con-
ducted for each of the six emotions. These analyses
revealed a significant mode effect for each of the
emotions: happiness, F(2, 35) 5 172.04, p , .001;
anger, F(2, 35) 5 92.19, p , .001; surprise, F(2, 35)
5 116.61, p , .001; sadness, F(2, 35) 5 29.72, p ,
.001; fear, F(2, 35) 5 24.73, p , .001; and disgust,
F(2, 35) 5 86.39, p , .001. There was a significant
group effect for happiness, F(3, 36) 5 5.31, p , .01;
sadness, F(3, 36) 5 3.62, p , .05; and disgust,
F(3, 36) 5 4.91, p , .01. Finally, there was a signif-
icant interaction between group and mode for hap-
piness, F(6, 70) 5 4.53, p , .001; anger, F(6, 70) 5
3.74, p , .01; and disgust, F(6, 70) 5 2.74, p , .05.
Table 4 presents the means and standard devia-
tions for each of the emotions in the auditory mode.
Table 3 F values and level of significance obtained in a comparison between the three modes of presentations in each of the
research groups
Auditory vs. visual, F(1, 9)
Auditory vs. auditory–visual, F(1, 9)
Visual vs. auditory–visual, F(1, 9)
Early implantees 114.22*** 113.26** 0.76
Late implantees 313.14** 172.8** 0.44
Hearing aid users 125.51** 346.17** 0.28
Normal hearing 47.25** 69.6** 13.04***
**p , .01; ***p , .001.
Perception of Emotions by Individuals With CIs, HA, and Normal Hearing 455
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
One-way ANOVAs were conducted in order to exam-
ine the differences between the groups with regard to
the auditory mode, followed by multiple-contrast anal-
yses for each emotion. The F values obtained are pre-
sented in the table. It can be seen that there were
significant differences in the auditory scores for each
of the emotions, with the exception of surprise. All
groups scored very low in the identification of sur-
prise. The NH group scored better than all other
groups in identifying happiness, sadness, and disgust.
There were no significant differences in the scores of
the other groups for these emotions. The NH group
scored significantly better than the late-CI and the HA
groups in identifying anger. However, there was no
significant difference between the NH and early-CI
groups in identifying this emotion. The early-CI
group scored significantly better than the HA group.
The NH group scored significantly better than the
early-CI and late-CI group in identifying fear. There
was no significant difference between the scores of the
NH and the HA groups.
As can be seen in Table 4, the results of the
auditory mode revealed a similar hierarchy in the
perception of the different emotions by the NH
and the two CI groups: Anger and sadness were iden-
tified the best and fear and surprise were perceived
the least successfully. The HA group showed a differ-
ent hierarchy than the other three groups.
One-way ANOVAs were conducted in order to
examine the differences between the groups with
regard to the visual mode, followed by multiple-
contrast analyses for each emotion. These analyses
revealed no significant differences between the
groups with regard to each of the emotions. In gen-
eral, emotion perception through the visual mode
was comparable in the different groups. All the par-
ticipants perceived happiness successfully (M 5
92.95%, SD 5 10.38), then anger (M 5 89.44%,
SD 5 11.9), sadness (M 5 82.95%, SD 5 15.76),
and disgust (M 5 85.38%, SD 5 11.89). The most
difficult to identify were surprise (M 5 68.46%, SD
5 18.39) and fear (M 5 54.02%, SD 5 24.41).
One-way ANOVAs were conducted in order to
examine the differences between the groups with
regard to the auditory–visual mode, followed by
multiple-contrast analyses for each emotion. These
analyses revealed a significant difference with regard
to the identification of surprise, F(3, 36) 5 3.49, p ,
.05. The NH scored better than the late- and early-CI
groups. The results of the HA group were not signif-
icantly different from those of any of the other groups.
There were no significant differences between the dif-
ferent groups with regard to the perception of the
other emotions. All the groups identified happiness
best, then anger and then disgust. All participants
with hearing loss had difficulties perceiving surprise.
Fear was the most difficult for all the participants to
identify.
A comparison between the correct identification
of the different emotions in the three modes of pre-
sentation revealed a similar hierarchy between the
visual and the auditory–visual modes. With regard
to the auditory mode, the hierarchy was different.
The principal difference was expressed in the per-
ception of happiness and sadness. Happiness was the
easiest to perceive when presented in the visual and
the auditory–visual modes. On the other hand, it was
more difficult to identify when presented in the
Table 4 Auditory percent identification (after correction for guessing; M and SD) of the different emotions by the four
groups and the obtained F values
Early implantees Late implantees Hearing aid users Normal hearing Group comparison, F(3, 36)
Happiness 19.93 (13.47) 16.09 (15.14) 24.01 (17.17) 61.82 (20.3) 12.21***
Anger 47.9 (17.95) 33.97 (24.91) 15.97 (23.12) 61.7 (15.57) 6.08**
Surprise 9.96 (17.9) 23.96 (17.19) 21.96 (16.55) 11.04 (22.84) 1.34
Sadness 30.13 (25.25) 28.09 (28.52) 36.01 (29.09) 61.7 (25.3) 3.04*
Fear 1.92 (21.39) 6 (15.68) 15.97 (17.11) 38.06 (29.26) 4.87*
Disgust 19.93 (22.25) 15.97 (12.96) 7.92 (19.48) 54.5 (22.34) 8.78**
*p , .05; **p , .01; ***p , .001.
456 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
auditory mode. Sadness was difficult to perceive
when presented in the visual and auditory–visual
modes but easier in the auditory mode.
Pearson correlations were conducted between the
scores in the three modes for each of the four groups.
There was a significant correlation between the vi-
sual and the auditory–visual scores for the early-CI
group (r 5 .82, p , .01) and late-CI group (r 5 .68,
p , .05) and in the NH group (r 5 .88, p , .001).
There were no significant correlations between these
modes for the HA group. There were no significant
correlations between the auditory mode and the vi-
sual and auditory–visual modes for any of the
groups.
Pearson correlations were also conducted be-
tween some demographic variables such as chrono-
logical age (for all participants), age of implantation
and duration of implant use (for the participants
with CI), degree of hearing loss (for the HA group),
and age at onset of hearing loss (for all participants
with hearing loss). A significant correlation was
found only between the chronological age and
the auditory–visual scores (r 5 .45, p , .01) and
the visual scores (r 5 .32, p , .05). There were no
significant correlations between the scores in the dif-
ferent modes and the other variables (p . .05).
Discussion
This study examined the ability of adolescents with
and without hearing loss to perceive nonverbal emo-
tional content solely through an auditory mode,
solely through a visual mode, and through a combined
auditory–visual mode. The performance of adoles-
cents using HA was compared to the performance
of two groups of adolescents using CI—those
implanted early and those implanted at a later age.
Our first hypothesis that the auditory emotion per-
ception of individuals with NH would be superior to
that of all participants with hearing loss was sup-
ported. The auditory identification of emotions by
the individuals with NH significantly surpassed that
of all three groups of participants with hearing loss.
This finding supported previous research on partici-
pants with NH and HA users (Most et al., 1993; Oster
& Risberg, 1986; Rigo & Liberman, 1989; Shinall,
2005) and on CI users (Luo et al., 2007; Pereira,
2000; Peters, 2006).
We wanted to examine whether CI users would
perform differently from HA users in auditory emo-
tion perception. A comparison of the emotion iden-
tification patterns for the HA and CI users revealed
that these groups performed comparably with regard
to each of the presentation modes. Thus, the CI users
did not demonstrate any advantage over the HA users
in identifying emotions through either the auditory
or the auditory–visual mode. It should be noted,
however, that most of the CI users had a poorer
unaided threshold of hearing than the HA users. It
is possible that a CI advantage would have emerged if
the two groups had comparable unaided thresholds;
however, this possibility is difficult to pursue in fu-
ture research because, currently, most implanted
individuals have profound hearing loss. At this point,
it is unknown whether the auditory or auditory–
visual perception of emotions by CI users would ex-
ceed that of HA users matched for severity of hearing
loss. Although the implant’s advantage in speech per-
ception was demonstrated by previous studies, this
relative benefit was mainly shown for the perception
of segmental features (Calmels et al., 2004; Gestoett-
ner et al., 2000).
The implant’s advantage in relation to the percep-
tion of suprasegmental features is not clear-cut. Some
research has shown that perception of suprasegmental
features improves after implantation (Huang, Wang, &
Liu, 1995; Waltzman & Hochberg, 1990). Other stud-
ies, however, did not show an advantage for CI users
compared to HA users (Boothroyd & Eran, 1994; Lee,
van Hasselt, Chiu, & Cheung, 2002; Most & Peled,
2007; O’Halpin et al., 2006). Moreover, some of these
studies even demonstrated poorer performance by CI
users in comparison to HA users in the perception of
intonation (Boothroyd & Eran, 1994; Most & Peled,
2007) and in the perception of syllable stress (Most &
Peled, 2007). The authors attributed this poorer
performance to many current implants’ inability to
provide information on changes in the low-frequency
range.
Inasmuch as the fundamental frequency is a very
salient cue to the auditory perception of emotions
(Murray & Arnott, 1993), it was expected that CI
Perception of Emotions by Individuals With CIs, HA, and Normal Hearing 457
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
users would experience difficulties in this perception
in comparison to HA users. In contrast to expect-
ations, despite the CI users’ lack of important sensory
information, they did not demonstrate lower perfor-
mance than the HA users. Perhaps the implant en-
abled enhanced auditory exposure to the verbal and
social aspects of emotions in everyday life, and these
improved skills may have compensated for the lack of
sensory information required for emotion identifica-
tion. In other words, the CI intervention process may
have improved the linguistic and cognitive skills rel-
evant to efficient emotion perception (De Sonneville
et al., 2002). The findings of Gray, Hosie, Russell,
Scott, and Hunter (2007) support this notion by
demonstrating that the lack of verbal input in some
deaf children slowed their development of emotional
understanding.
Our hypothesis that the early-CI group would
perform better than the late-CI group through the
auditory and the auditory–visual modes was not sup-
ported. No significant differences emerged between
the early and late implantees in any of the presenta-
tion modes, and no significant correlation emerged
between the perception of emotions and age at im-
plantation; however, the auditory perception results
of the early implantees were slightly better than those
of the late implantees. Moreover, the large standard
deviations in both of these small groups, along with
the very low perception levels suggesting a possible
‘‘floor effect,’’ may have precluded the emergence of
significant differences in performance among these
groups. Future research with larger samples in the
different groups is recommended to further analyze
this issue.
The results showing the implant’s lack of advan-
tage over the HA and the insignificant correlation
between age at implantation and auditory percep-
tion of emotions support the hypothesis that the
current CI technology does not successfully trans-
mit the acoustic information required for emotion
perception.
All four study groups, including the hearing
group, revealed comparable emotion perception
through the visual mode. These results support pre-
vious reports of similarity in visual emotion percep-
tion by individuals with and without hearing loss
(e.g., Dyck et al., 2004; Hosie et al., 1998; McCullough
et al., 2005; Rollman & Harrison, 1996; Weisel, 1985).
Other studies, however, reported lower visual identifi-
cation of emotions by individuals with hearing loss
(Bachara et al., 1980; Dyck & Denver, 2003; Most
et al., 1993; Weisel & Bar-Lev, 1992). Future studies
would do well to explore these inconsistencies in
more detail, perhaps by administering a more sensitive
measure such as response time, which may enable de-
tection of possible subtle differences in the ability
to perceive emotions through the visual mode
(De Sonneville et al., 2002). Because the communica-
tion process is very dynamic, the time it takes to iden-
tify a facial expression in a real-life everyday situation
might be revealing.
For participants with NH, auditory identification
of emotions was significantly lower than visual iden-
tification or combined auditory–visual identification,
as found in previous research (Burns & Beier, 1973;
Hess et al., 1988; Most & Greenbank, 2000; Most
et al., 1993; Rigo & Liberman, 1989; Wallbott &
Scherer, 1986). Also supporting prior studies, and
our hypothesis, participants with NH performance
through the combined mode significantly surpassed
that of each sensory mode alone (Most & Greenbank,
2000; Most et al., 1993). Although the difference
between the visual and the combined modes was
smaller than the difference between the auditory
and the combined modes, participants with NH were
able to utilize the auditory information and to benefit
from the additional auditory information received
about emotions in the combined mode.
In contrast to the participants with NH, who
performed significantly better in the combined
auditory–visual mode in comparison to either of the
modes alone and who showed significantly superior
perception of emotions through the visual mode in
comparison to the auditory mode, all the groups of
participants with hearing loss performed equally well
in the combined auditory–visual mode and in the
solely visual mode. In other words, the participants
with hearing loss, whether they used HA or CI, were
unable to benefit from the additional auditory infor-
mation provided in the combined mode. In general,
these groups were unable to use their residual hear-
ing successfully. These results supported previous
458 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
results on individuals with HA (Most et al., 1993;
Rigo & Liberman, 1989).
Examination of individual outcomes, however,
revealed great variations. Seven of the 10 partici-
pants in the group with NH (70%) exhibited better
auditory–visual scores in comparison to the solely
visual scores, and the other three participants
(30%) exhibited similar scores in those two modes.
However, only 11 of the 30 participants with hearing
loss (37%) benefited more from the auditory–visual
mode than the visual alone, and four participants
(13%) exhibited similar scores in those two modes.
A full half of participants (15) even demonstrated
better performance in the solely visual mode in com-
parison to the combined auditory–visual mode. This
great discrepancy in the ability of individuals with
hearing loss to integrate the information from the
two sensory modes in the perception of speech was
previously reported by Grant, Walden, and Seitz
(1998).
Earlier results on implantees reported their suc-
cessful integration of visual and auditory information
when attempting to perceive words and sentences
(Bergeson, Pisoni, & Davis, 2005; Lachs, Pisoni, &
Kirk, 2001). The current findings indicating CI
users’ inability to integrate different channels of in-
formation when perceiving emotions emphasizes the
implant’s limitations in providing the auditory infor-
mation necessary for emotion perception.
Age at implantation did not significantly affect
participants’ emotion identification; however, it should
be noted that, in this study, the earliest implanta-
tion age was 2.6 years. Schorr, Fox, Wassenhove, and
Knudsen (2005) reported better auditory–visual in-
tegration in the speech perception of children who
were implanted before age 2.6 years, in comparison
to those who were implanted after this age. It is
possible that participants who were implanted earlier
might show better auditory–visual integration in the
perception of emotions as well. Future studies
should continue to examine emotion identification
among participants with CI who were implanted at
a younger age.
In addition to comparing the different groups of
participants and the different modes of presentation,
this study also investigated the hierarchy in accuracy
of perceiving specific types of emotions. Results
revealed differences in the ability to accurately
perceive the various emotions through each of the
presentation modes.
The NH and CI groups found anger and sadness
the easiest to identify; fear and surprise were the most
difficult, and happiness and disgust were in-between.
A similar hierarchy was reported earlier (Most
et al., 1993; Pereira, 2000; Sincoff & Rosenthal,
1985; Wallbott & Scherer, 1986).
The HA group revealed a similar pattern of
results, with the exception of small differences in
the ratings of anger and fear. The acoustic analysis
performed on the stimuli was unable to account for
these findings. Perhaps information on the voice
quality of the speaker, in addition to the acoustic
parameters of the fundamental frequency, the inten-
sity, and the duration of the stimuli, is necessary in
order to perceive some emotions better than others
(Scherer, 1986).
A similar hierarchy of emotion types emerged for
the visual and the auditory–visual modes, among
all groups of participants, in line with previously
reported results (Burns & Beier, 1973; Calder et al.,
2000; Montagne et al., 2005; Most et al., 1993;
Wagner et al., 1986). Happiness and anger were the
easiest to perceive, then disgust and sadness, then
surprise, and, finally, fear. This similarity of hierar-
chy in the visual and auditory–visual modes may
suggest that visual information was more dominant
in the combined mode.
Emotion perception in the three modes did not
link significantly with implantation age, duration of
implant use, or degree of hearing loss (for the HA
users). However, participants’ chronological age did
correlate significantly with emotion perception in the
visual as well as the auditory–visual modes, with
older participants obtaining better perception scores.
This finding supports De Sonneville et al. (2002),
who indicated that mental knowledge, which includes
representation of the typical emotions, grows with
age and results in a more effective emotion identifi-
cation process. Also, Gray et al. (2007) reported that
understanding of emotions developed among deaf
children; those aged 9:5–13:2 years were better at
assigning emotions to story characters than younger
Perception of Emotions by Individuals With CIs, HA, and Normal Hearing 459
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
deaf children aged 5:5–8:7 years. The participants in
this study were between the ages of 10:0 and 17:6
years. Thus, the currently obtained correlation with
chronological age suggests that the ability to perceive
emotions continues to develop at a later age as well.
This correlation might not have been expressed in
relation to the auditory perception because of the
very poor scores (aforementioned floor effect) for
participants with hearing loss. Future research
should continue to examine older participants in or-
der to determine until what age this ability continues
to develop.
Understanding of emotions by deaf children
seems to improve with age (Gray et al., 2007); how-
ever, the present outcomes do not yet explicate pos-
sible sources of improvement over the years, such as
increased understanding of emotional vocabulary or
additional exposure to socioemotional situations. We
used the inclusion criterion of a 75% score on an
emotion vocabulary test for participation in the cur-
rent study; therefore, we could not investigate the
correlation between emotion vocabulary and emo-
tion identification. Such a link has been found by
other researchers like Nygaard and Queen (2008),
who reported that listeners were faster in naming
items when the tone of voice was congruent with
the semantic content of the word. They demon-
strated that the perception of spoken words was
not independent of perception of the emotional
prosody. Future research would do well to examine
this relationship and its implications for develop-
mental trajectories.
In light of the risks inherent in poor ability to
perceive emotions, such as inadequate social pat-
terns of behavior, poor social skills, and difficulties
in social communication (Rieffe & Terwogt, 2000),
further empirical inquiry should focus on whether
children’s exposure to more opportunities for ex-
periencing social-emotional situations and emo-
tional tones at school and at home, with peers and
adults, may possibly help them learn to recognize
emotions better, and whether these skills may also
be augmented through education. Dyck and Denver
(2003) reported on improvement in the visual per-
ception of emotion following intervention. Further
research in this direction has important implications
for intervention planning in order to help individu-
als with hearing loss to communicate better,
with ramifications for social understanding and
functioning.
In summary, based on the visual and the auditory–
visual results, it can be assumed that the speaker’s
mental state is intelligible to individuals with hearing
loss in many communication interactions where the
listener not only watches the speaker’s face but also
listens to him/her. Nevertheless, in situations with
inadequate or unavailable visual information, such as
communication on the telephone, in the car, or in the
dark, the listener needs to rely on the auditory infor-
mation provided.
The findings of this study question the benefits
of CI in transmitting the acoustic information re-
quired to identify the speaker’s emotional state.
The current outcomes highlight the importance of
incorporating nonverbal aspects of communication,
including emotion perception, into the intervention
process for individuals with hearing loss wearing HA
or CI. Specific attention should be paid to the au-
ditory perception of emotions when visual input is
missing.
In addition, research should continue to exam-
ine the effects of current CI coding strategies,
which claim to provide better information in the
low-frequency range with regard to the ability to
perceive emotions. For example, the fine structure
processing (FSP) strategy recently offered by
Med-El claims to provide periodicity information.
Mitterbacher, Zierohofer, Schatzer, and Kals (2005)
reported improved pitch discrimination with the
FSP strategy in comparison to the Continuous
Interleaved Sampling strategy. Future studies
should also continue to research participants who
use CI on one ear and HA on the other, in order
to determine whether this bimodal condition results
in better emotion perception when compared to the
use of CI alone. Examination of children with
bilateral CI is recommended as well. And finally,
future examination of emotion perception by
postlingually deafened individuals in comparison to
prelingually deafened individuals would allow
researchers to evaluate the effect of auditory and lin-
guistic exposures on the ability to perceive emotions.
460 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
Appendix A: Early implantees
Participant no. Gender
Age (years)
Age of implantation (years)
Implant usage duration (years)
Implant type
Speech processor type
Implanted ear
PTA Rt/Lt (dB)a
Cause of hearing loss
Age at discovery (years)
Age at rehabilitation (years)
Parents are HH
1 F 11 4:1 6:9 Nucleus24 Sprint Left 108/NR Genetic 0:1 0:3 Yes
2 M 14:4 3:1 11:3 Nucleus22 Sprint Right 110/102 UK 0:6 1 No
3 F 15:7 2:6 13:1 Nucleus22 Sprint Left 117/113 UK 1:5 1:8 No
4 M 13:10 5:2 8:8 Clarion S-Series Right 115/95 UK 1:8 2 No
5 F 10:10 2:6 8:4 Clarion S-Series Right NR/97 Meningitis 1:6 2 No
6 M 15 5:6 9:6 Clarion90k Auria Left 100/NR UK 1 1:6 No
7 F 13:3 6 7:3 Nucleus24 Sprint Right NR/88 Genetic 0:1 0:3 No
8 F 13 3:11 9:1 Nucleus24 Sprint Right 118/110 UK 0:8 1:5 No
9 F 13:7 2:6 11:1 Nucleus22 Sprint Left 115/NR Genetic 0:2 0:3 No
10 M 14:11 4 10:11 Nucleus22 Sprint Left 113/NR Genetic 0:7 1 No
Note: F 5 female; HH 5 hard of hearing; M 5 male; UK 5 unknown etiology. a Pure tone average—means at 500, 1,000, and 2,000 Hz.
Appendix B: Late implantees
Participant no. Gender Age (years)
Age of implantation (years)
Implant usage duration (years)
Implant type
Speech processor type
Implanted ear
PTA Rt/Lt (dB)
a
Cause of hearing loss
Age at discovery (years)
Age at rehabilitation (years)
Parents are HH
1 F 16:2 6:2 10 Clarion90k Auria Left 113/NR UK 3 3:6 No
2 M 15:6 12 3:6 Nucleus24 3G Left 105/102 UK 0:6 0:9 No
3 M 13:8 6:11 6:9 Nucleus24 Sprint Left 95/115 Genetic 0:1 0:3 Yes
4 M 17:6 15 2:6 Clarion90k Auria Right 105/110 Genetic 1:10 2:3 No
5 F 10 8:3 1:9 Pulsar Tempo1 Right 103/102 Genetic 3 3:3 No
6 F 14:8 12:6 2:2 Nucleus24 3G Right NR/122 UK 2:6 2:11 No
7 M 17:3 11:11 5:4 Nucleus24 3G Left 105/NR UK 0:8 1:6 No
8 F 17 16 1 Freedom Freedom Right 107/105 UK 2 2:6 No
9 F 17:5 16:2 1:3 Freedom Freedom Right 102/92 CMV 3:6 6 No
10 F 11:9 8:9 3 Nucleus24 3G Left 88/NR UK 2:6 3 No
Note: CMV 5 cytomegalovirus; F 5 female; HH 5 hard of hearing; M 5 male; UK 5 unknown etiology. aPure tone average—means at 500, 1,000, and 2,000 Hz.
P ercep
tio n o f E m o tio
n s b y In d iv id u als
W ith
C Is,
H A , an d N o rm
al H earin
g 4 6 1
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
References
Bachara, G. H., Raphael, J., & Phelan, W. J. (1980). Empathy
development in deaf preadolescents. American Annals of the
Deaf, 125, 38–41.
Banse, R., & Scherer, K. (1996). Acoustic profiles in emotion
expression. Journal of Personality and Social Psychology, 70,
614–636.
Bergeson, T. R., Pisoni, D. B., & Davis, R. A. O. (2005).
Development of audiovisual comprehension skills in pre-
lingually deaf children with cochlear implants. Ear and
Hearing, 26, 149–164.
Blamey, P. J., Sarant, J. Z., Paatsch, L. E., Barry, J. G., Bow, C. P.,
Wales, R. J., et al. (2001). Relationships among speech
perception, production, language, hearing loss, and age in
children with impaired hearing. Journal of Speech, Language
& Hearing Research, 44, 264–285.
Boothroyd, A. (1988). Perception of speech pattern contrasts
from auditory presentation of voice fundamental frequency.
Ear and Hearing, 9, 313–321.
Boothroyd, A., & Eran, O. (1994). Auditory speech perception
capacity of child implant users expressed as equivalent
hearing loss. Volta Review, 96, 151–168.
Burns, K. L., & Beier, E. G. (1973). Significance of vocal and
visual channels in the decoding of emotional meaning. Jour-
nal of Communication, 23, 118–130.
Calder, A. J., Young, A. W., Keane, J., & Dean, M. (2000).
Configural information in facial expression perception.
Journal of Experimental Psychology: Human Perception and
Performance, 26, 527–551.
Calmels, M. N., Saliba, I., Wanna, G., Cochard, N., Fillaux, J.,
Deguine, O., et al. (2004). Speech perception and speech
intelligibility in children after cochlear implantation. Inter-
national Journal of Pediatric Otorhinolaryngology, 68,
347–351.
Carney, A. R., Kienle, M., & Miyamoto, R. T. (1990). Speech
perception with a single-channel cochlear implant. Journal
of Speech and Hearing Research, 33, 229–237.
Carroll, J., & Zeng, F. G. (2007). Fundamental frequency
discrimination and speech perception in noise in cochlear
implant simulations. Hearing Research, 231, 42–53.
De Sonneville, L. M. J., Verschoor, C. A., Njioktjien, C., Op het
Veld, V., Toorenaar, N., & Vranken, M. (2002). Facial iden-
tity and facial emotions: Speed, accuracy, and processing
strategies in children and adults. Journal of Clinical and
Experimental Neuropsychology, 24, 200–213.
Dyck, M., & Denver, E. (2003). Can the emotion recognition
ability of deaf children be enhanced? A pilot study. Journal
of Deaf Studies and Deaf Education, 8, 348–356.
Dyck, M., Farrugia, C., Shochet, I., & Holmes-Brown, M.
(2004). Emotion recognition/understanding ability in hear-
ing or vision-impaired children: Do sounds, sights, or
words make the difference? Journal of Psychology and
Psychiatry, 45, 789–800.
Einot, I., & Gabriel, K. R. (1975). A study of the powers of
several methods of multiple comparisons. Journal of the
American Statistical Association, 70, 351.
El-Hakim, H., Papsin, B., Mount, R., Levasseur, J., Panesar, J.,
Stevens, D., et al. (2001). Vocabulary acquisition rate after
cochlear implantation and the impact of age at implantation.
International Journal of Pediatric Otorhinolaryngology, 59,
187–194.
Ekman, P. (1999). Facial expressions. In T. Dalgleish & M.
Power (Eds.), Handbook of cognition and emotion (pp.
45–60). New York: John Wiley & Sons.
Geier, L., Barker, M. J., Opie, J., & Fisher, L. (2000). Speech
perception in postlingually deafened adult Clarion users
with long-term deafness. In S. Waltzman & N. Cohen
(Eds.), Cochlear implants (pp. 336–337). New York: Theme
Medical Publishers.
Gestoettner, W. K., Hamzavi, J., Egelierlier, B., & Baumgartner,
W. D. (2000). Speech perception performance in prelin-
gually deaf children with cochlear implants. Acta Otolaryn-
gology, 120, 209–213.
Grant, K. W., Walden, E. W., & Seitz, P. F. (1998). Auditory-
visual speech recognition by hearing impaired subjects:
Appendix C: Hearing Aid users
Participant no. Gender
Age (years)
Hearing aid type
PTA Rt/Lt (dB)a
PTA with hearing aid (dB)a
Cause of hearing loss
Age at discovery (years)
Age at rehabilitation (years)
Parents are HH
1 F 14:9 Widex 88/97 30 UK 3:6 5:2 No
2 F 12:11 Siemens 97/98 50 UK 2 3:10 No
3 M 10:2 Siemens 80/85 33 CMV 1:8 2:4 No
4 F 15:3 Oticon 80/75 33 UK 3 3:7 No
5 F 12:11 Phonak 73/73 28 UK 1 1:6 No
6 M 15 Siemens 75/110 25 Genetic 0:1 0:3 Yes
7 M 15:9 Widex 107/95 32 Genetic 0:2 0:6 Yes
8 M 15 Widex 88/93 25 UK 2:10 3:5 No
9 F 11:4 Phonak 97/110 50 Genetic 0:1 0:3 Yes
10 M 15:4 Widex 93/98 45 Genetic 0:1 0:3 Yes
Note: F 5 female; HH 5 hard of hearing; M 5 male. a Pure tone average—means at 500, 1,000, and 2,000 Hz.
462 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
Consonant recognition, sentence recognition and auditory-
visual integration. Journal of the Acoustical Society of
America, 103, 2677–2690.
Gray, C., Hosie, J., Russell, P., Scott, C., & Hunter, N. (2007).
Attribution of emotions to story characters by severely and
profoundly deaf children. Journal of Developmental and
Physical Disabilities, 19, 145–159.
Harrison, R., Gordon, K., & Mount, R. (2005). Is there a critical
period for cochlear implantation in congenitally deaf children?
Analyses of hearing and speech perception performance after
implantation. Developmental Psychobiology, 46, 252–261.
Hess, U., Kappas, A., & Scherer, K. R. (1988). Multichannel
communication of emotion: Synthetic signal processing. In
K. R. Scherer (Ed.), Facets of emotion: Recent research (pp.
161–182). Hillsdale (NJ): Lawrence Erlbaum Associates.
Hosie, J. A., Gray, C. D., Russell, P. A., Scott, C., & Hunter, N.
(1998). The matching of facial expressions by deaf and
hearing children and their production and comprehension
of emotion labels. Motivation and Emotion, 22, 293–312.
Huang, T., Wang, S., & Liu, S. (1995). Tone perception of
Mandarin-speaking postlingually deaf implantees using
the Nucleus 22-channel cochlear mini system. Annals of
Otology, Rhinology and Laryngology, 166(Suppl), 294–298.
Kishon-Rabin, L., Taitelbaum, R., Muchnic, C., Ghetler, I.,
Kronenberg, J., & Hildesheimer, M. (2002). Development
of speech perception and production in children with co-
chlear implants. Analysis of Otology, Rhinology and Laryn-
gology, 111, 85–89.
Lachs, L., Pisoni, D. B., & Kirk, K. I. (2001). Use of audiovisual
information in speech perception by prelingually deaf chil-
dren with cochlear implants: A first report. Ear and Hear-
ing, 22, 236–251.
Lee, K., Van Hasselt, C., Chiu, S., & Cheung, D. (2002). Can-
tonese tone perception ability of cochlear implant children
in comparison with normal-hearing children. International
Journal of Pediatric Otorhinolaryngology, 63, 137–147.
Ling, D., & Ling, A. H. (1978). Aural habilitation: The founda-
tions of verbal learning in hearing-impaired children.
Washington, DC: The Alexander Graham Bell Association
for the Deaf.
Luo, X., Fu, Q., & Galvin, J. J. (2007). Vocal emotion recogni-
tion by normal hearing listeners and cochlear implant users.
Trends in Amplification, 11, 301–315.
McCullough, S., Emmorey, K., & Sereno, M. (2005). Neural
organization for recognition of grammatical and emotional
facial expressions in deaf ASL signers and hearing non-
signers. Cognitive Brain Research, 22, 193–203.
Mellon, N. K. (2000). Psychosocial development of children
in deafness. In J. K. Niparko (Ed.), Cochlear implants:
Principles and practices (pp. 319–321). Philadelphia, PA:
Lippincott Williams & Wilkins.
Meyer, T., Svirski, M., Kirk, K., & Miyamoto, R. (1998).
Improvements in speech perception by children with pro-
found prelingual hearing loss: Effects of device, communi-
cation mode, and chronological age. Journal of Speech,
Language, and Hearing Research, 41, 846–858.
Mildner, V., Sindija, B., & Zrinski, K. V. (2006). Speech per-
ception of children with cochlear implants and children
with traditional hearing aids. Clinical Linguistics and
Phonetics, 20, 219–229.
Mitterbach, A., Zierfofer, C., Schatzer, R., & Kals, M. (2005).
Encoding fine time structure with channels specific sampling
sequences. Conference on Implantable Auditory Prosthesis,
Pacific Grove, CA.
Montagne, B., Kessels, R. P. C., Frigerio, E., De Haan, E., &
Perret, D. I. (2005). Sex differences in the perception of
affective facial expressions: Do men really lack emotional
sensitivity? Cognitive Processing, 6, 136–141.
Moore, B. (1996). Perceptual consequences of cochlear hearing
loss and their implications for the design of hearing aids.
Ear and Hearing, 17, 136–141.
Most, T., & Greenbank, A. (2000). Auditory, visual, and
auditory-visual perception of emotions by adolescents
with and without learning disabilities and its relation
to their social skills. Learning Disabilities Research &
Practice, 15, 171–178.
Most, T., & Peled, M. M. (2007). Perception of suprasegmental
features of speech by children with cochlear implants and
children with hearing aids. Journal of Deaf Studies and Deaf
Education, 12, 350–361.
Most, T., Weisel, A., & Zaychik, A. (1993). Auditory, visual and
auditory-visual identification of emotions by hearing and
hearing-impaired adolescents. British Journal of Audiology,
27, 247–253.
Mozziconacci, S. (2002). Prosody and emotion. Proceedings of
Speech Prosody, Aix-en-Provence, France.
Murray, I. R., & Arnott, J. L. (1993). Toward the simulation of
emotion in synthetic speech: A review on the literature on
human vocal emotion. Journal of the Acoustical Society of
America, 93, 1097–1108.
Nygaard, L. C., & Queen, J. S. (2008). Communicating emo-
tions: Linking affective prosody and word meaning. Journal
of Experimental Psychology: Human Perception and Perfor-
mance, 34, 1017–1030.
Oh, S. H., Kim, C. S., Kang, E. J., Lee, D. S., Lee, H. J.,
Chang, S. O., et al. (2003). Speech perception after cochlear
implantation over a 4-year time period. Acta Otolaryngolog-
ica, 123, 148–153.
O’Halpin, R., Falkoner, A., Rosen, S., & Viani, L. (2006). The
perception of stress and intonation by children with cochlear
implants. Paper Presented at the 8th European Symposium
of Pediatric Cochlear Implantation, March 2006, Venice,
Italy.
Oster, A. M., & Risberg, A. (1986). The identification of the
mood of a speaker by hearing-impaired listeners.
SLT-Quarterly Progress Status Report, 4, 79–90.
Peng, S. C., Tomblin, J. B., & Turner, C. W. (2008). Production
and perception of speech intonation in pediatric cochlear
implant recipients and individuals with normal hearing. Ear
And Hearing, 29, 336–351.
Pereira, C. (2000). The perception of vocal affect by cochlear
implants. In S. B. Waltzman & N. L. Cohen (Eds.), Co-
chlear Implants (pp. 343–345). New York, NY: Thieme.
Peters, K. P. (2006). Emotion recognition in speech: Discrimination,
identification, and the effects of talker and sentence variability.
Unpublished doctoral dissertation, Washington University.
Perception of Emotions by Individuals With CIs, HA, and Normal Hearing 463
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019
Peterson, C., & Siegal, M. (1995). Deafness, conversation, and
theory of mind. Journal of Psychology and Psychiatry, 36,
459–474.
Rieffe, C., & Terwogt, M. M. (2000). Deaf children’s under-
standing of emotions: Desires take precedence. Journal of
Child Psychology and Psychiatry, 41, 601–618.
Rigo, T. G., & Liberman, D. A. (1989). Nonverbal sensitivity of
normal hearing and hearing-impaired older adults. Ear and
Hearing, 10, 184–189.
Rollman, S. A., & Harrison, R. D. (1996). A comparison of deaf
and hearing subjects in visual nonverbal sensitivity and in-
formation processing. American Annals of the Deaf, 141,
37–41.
Scherer, K. R. (1986). Vocal affect expression: A review and
a model for future research. Psychological Bulletin, 99,
143–165.
Scherer, K. R., & Ellgring, H. (2007). Multimodal expression of
emotion: Affect programs or componential appraisal pat-
terns? Emotion, 7, 158–171.
Shackman, J. A., & Pollak, S. D. (2005). Experiential influences
on multimodal perception of emotion. Child Development,
76(5), 1116–1126.
Shinall, A. R. (2005). Emotion perception in pre-kindergarten
school children at central institute for the deaf. Unpublished
master’s thesis, Washington University.
Schorr, E., Fox, N., Wassenhove, V., & Knudsen, E. (2005).
Auditory–visual fusion in speech perception in children
with cochlear implants. Proc Natl Acad Sci USA, 102,
18748–18750.
Seung-Ha, O., Chong-Sun, K., Eun, J. K., Dong, S. L., Hyo, J. L.,
Sun, C., et al. (2003). Speech perception after cochlear im-
plantation over a 4-year time period. Acta Otolaryngologica
Supplement, 552, 55–63.
Sincoff, J. B., & Rosenthal, R. (1985). Content-masking method
as determinants of results of studies of nonverbal commu-
nication. Journal of Nonverbal Behavior, 9(2), 121–129.
Sullivan, S., Ruffman, T., & Hutton, S. B. (2007). Age differ-
ences in emotion recognition skills and the visual scanning
of emotion faces. Journal of Gerontology, 62, 53–60.
Taitelbaum-Swead, R., Kishon-Rabin, L., Kaplan-Neeman, R.,
Muchnic, C., Kronenberg, J., & Hildesheimer, M. (2005).
Speech perception of children using Nucleus, Clarion or
Med-El cochlear implants. International Journal of Pediatric
Otorhinolaryngology, 69, 1675–1683.
Wagner, H. L., McDonald, C. J., & Manstead, A. S. R. (1986).
Communication of individual emotions by spontaneous
facial expressions. Journal of Personality and Social Psy-
chology, 50, 737–743.
Wallbott, H. G., & Scherer, K. R. (1986). Cues and channels
in emotion recognition. Journal of Personality and Social
Psychology, 51, 690–699.
Walker-Andrews, A. S., & Lennon, E. (1991). Infants’ discrimina-
tion of vocal expressions: Contributions of auditory and visual
information. Infant Behavior and Development, 14, 131–142.
Waltzman, S., & Hochberg, I. (1990). Perception of speech pat-
tern contrasts using a multichannel cochlear implant. Ear
and Hearing, 11, 50–55.
Weisel, A. (1985). Deafness and perception of nonverbal expres-
sion of emotion. Perception Motor Skills, 61, 515–522.
Weisel, A., & Bar-Lev, H. (1992). Role taking ability, nonverbal
sensitivity, language and social adjustment of deaf adoles-
cents. Educational Psychology, 12, 3–13.
Williams, C., & Stevens, K. (1972). Emotions and speech: Some
acoustical correlates. Journal of the Acoustical Society of
America, 52, 1238–1250.
Wu, J. L., & Yang, H. M. (2003). Speech perception of Man-
darin Chinese speaking young children after cochlear im-
plant use: Effect of age at implantation. International
Journal of Pediatric Otorhinolaryngology, 67, 247–253.
Received December 18, 2008; revisions received March 16,
2009; accepted March 20, 2009.
464 Journal of Deaf Studies and Deaf Education 14:4 Fall 2009
D ow
nloaded from https://academ
ic.oup.com /jdsde/article-abstract/14/4/449/488373 by guest on 05 D
ecem ber 2019