Caricaturing facial expressions

A.J. Calder et al. / Cognition 76 (2000) 105±146

105

COGNITION Cognition 76 (2000) 105±146

www.elsevier.com/locate/cognit

Caricaturing facial expressions Andrew J. Calder a,*, Duncan Rowland b, Andrew W. Young c, Ian Nimmo-Smith a, Jill Keane a, David I. Perrett b a

MRC Cognition and Brain Sciences Unit, 15 Chaucer Road, Cambridge CB2 2EF, UK b School of Psychology, University of St. Andrews, St. Andrews, Fife KY16 9JU, UK c Department of Psychology, University of York, Heslington, York YO1 5DD, UK Received 13 April 1999; accepted 4 March 2000

Abstract The physical differences between facial expressions (e.g. fear) and a reference norm (e.g. a neutral expression) were altered to produce photographic-quality caricatures. In Experiment 1, participants rated caricatures of fear, happiness and sadness for their intensity of these three emotions; a second group of participants rated how `face-like' the caricatures appeared. With increasing levels of exaggeration the caricatures were rated as more emotionally intense, but less `face-like'. Experiment 2 demonstrated a similar relationship between emotional intensity and level of caricature for six different facial expressions. Experiments 3 and 4 compared intensity ratings of facial expression caricatures prepared relative to a selection of reference norms ± a neutral expression, an average expression, or a different facial expression (e.g. anger caricatured relative to fear). Each norm produced a linear relationship between caricature and rated intensity of emotion; this ®nding is inconsistent with two-dimensional models of the perceptual representation of facial expression. An exemplar-based multidimensional model is proposed as an alternative account. q 2000 Elsevier Science B.V. All rights reserved. Keywords: Facial expressions; Caricatures; Circumplex model

1. Introduction In a recent study, Calder, Young, Rowland and Perrett (1997) demonstrated a RT advantage for the recognition of computer-generated (photographic quality) carica* Corresponding author. Tel.: 144-1223-355-294, ext. 741; fax: 144-1223-359-062. E-mail address: [email protected] (A.J. Calder). 0010-0277/00/$ - see front matter q 2000 Elsevier Science B.V. All rights reserved. PII: S 0010-027 7(00)00074-3

106

A.J. Calder et al. / Cognition 76 (2000) 105±146

tures of emotional facial expressions. They also showed that people are slower to categorize the expressions when their features are made less distinctive (i.e. an anticaricatured representation). These results have been mirrored in recent brainimaging work using the same caricatured expressions. Here, caricatures of fear and disgust were shown to engage different brain regions, with changes in neural activity being positively related to level of caricature (Morris et al., 1996, 1998; Phillips et al., 1997, 1998). The caricature procedure has also been used to investigate the perception of other facial characteristics, including identity, attractiveness and age (Benson & Perrett, 1991a; Burt & Perrett, 1995; Calder, Young, Benson & Perrett, 1996; Perrett, May & Yoshikawa, 1994; Rhodes, Brennan & Carey, 1987). All of these studies have used the same basic process which operates by exaggerating the positions of set anatomical feature points relative to the locations of corresponding points on a reference norm face. The particular advantage of this procedure is that it is highly objective. Hence, although the system requires a number of anatomical landmarks to be identi®ed on the to-be-caricatured face, these are of suf®cient quantity, and in a suf®cient variety of locations, to ensure that all aspects of the face's shape are exaggerated. In addition, the system exploits the fact that by changing a feature's position with respect to a reference norm, those features of the to-be-caricatured face that differ most from the norm (i.e. the distinctive features) are exaggerated the most, while features that differ minimally from the norm are exaggerated the least. This means that, to some extent, the choice of norm can govern which aspects of the face are exaggerated more than others. Consequently, investigations of identity caricaturing have generally used an average face norm (abstracted from a number of faces of the same sex and approximate age as the to-be-caricatured faces), because here the aim is to exaggerate the features that differentiate a face from the population average (e.g. big nose, thick eyebrows, etc.). For facial expression caricaturing, however, the aim is to exaggerate the distinctive features of the expression (e.g. wrinkled nose, raised eyebrows, etc.), while leaving the distinctive features of the person's face (e.g. big nose, thick eyebrows, etc.) relatively intact. Hence, Calder et al. (1997) exaggerated each facial expression relative to a picture of the same person posing a neutral expression (i.e. a neutral facial expression norm). In the experiments described in the latter half of this paper we explore the extent to which the choice of reference norm can in¯uence the caricature effect for facial expression. But, ®rst we investigate the psychological basis of this effect. One interpretation offered by Calder et al. (1997) is that facial expression caricaturing works by enhancing an expression's emotional intensity. In Experiment 1 we investigated this hypothesis by asking participants to rate caricatures of fear, happiness and sadness for their intensity of fear, happiness and sadness, respectively. Each set of expression caricatures was presented in a separate block along with caricatures of an expression that is occasionally confused with target images (e.g. disgust is on occasions confused with sadness), and a third set of facial expression caricatures that is not so readily confused with the target. All of the images were caricatured at seven levels of exaggeration (275%, 250%, 225%, 0%, 125%, 150% and 175%). By using images caricatured by as much as 175%, we ensured

A.J. Calder et al. / Cognition 76 (2000) 105±146

107

that some of the images fell outwith the range of facial con®gurations seen in natural facial expressions. Rating studies with facial identity caricatures have shown that caricatured faces are perceived as signi®cantly better likenesses of people than the original (undistorted) representations. However, the caricature advantage for goodness-of-likeness is small (typically in the range 14±16%), and caricaturing identity above these optimum levels produces faces that are judged as progressively worse likenesses of people. One possible outcome of Experiment 1, then, was that intensity ratings for expression caricatures would show a similar pattern. However, our own impression of the caricatured expressions instead concurs with the idea that caricaturing operates by enhancing emotional intensity; that is, emotional intensity appears to increase monotonically with increasing levels of caricature ± even when the images are exaggerated to the point that they no longer resemble natural-looking faces. For example, as can be seen in Fig. 2, at higher levels of caricature smiling mouths can become exaggerated to almost twice their original size (middle row, 175%), and eyebrows raised in fear can move close to the centre of the brow (top row, 175%). Yet despite these considerable distortions the 175% expressions appear more emotionally intense than the lower levels of caricature. If our intuition was correct, this suggested an interesting hypothesis: that caricaturing should degrade the veridicality of the face (i.e. its `face-likeness'), but not the veridicality of the facial expression. To investigate this we asked a second group of participants to rate how `face-like' the caricatures looked. We reasoned that if rated emotional intensity is found to increase as a function of level of caricature (even when the caricatures are so exaggerated that they are no longer regarded as `natural-looking' faces), then this would give an insight into how expressions are coded. For instance, it would support the idea that facial expressions can be represented as continua as well as belonging to discrete categories. In addition, it would suggest that our representation of facial expression is coded independently of our representation of what is face-like. Experiment 2 used a similar design to determine whether this caricature effect was evident for all of the six basic emotions (happiness, sadness, anger, fear, disgust and surprise) from the Ekman and Friesen (1976) series of facial expressions. Both experiments demonstrated that enhancing the perceptual (structural) salience of a facial expression increases the intensity of the emotion displayed. Elaborating on this result, Experiments 3 and 4 used the caricature procedure to examine the predictions of two-dimensional models of facial affect recognition. 1.1. The perceptual representation of facial expressions A continuing debate in the emotion literature concerns the psychological basis of facial expression recognition. There are currently two main theoretical positions. One proposes that facial expressions are identi®ed by registering their positions on two continuous underlying dimensions, and the second that they activate qualitatively discrete categories. Here we are particularly interested in the predictions of the two-dimensional model. The continuous-dimensions theory was originally put forward by Schlosberg

108

A.J. Calder et al. / Cognition 76 (2000) 105±146

(1941, 1952) and was based on the observation by Woodworth (1938) that errors in facial expression recognition show relatively consistent patterns; for example, expressions of anger are more likely to be mistaken for disgust than for happiness or surprise, whereas expressions of fear are more readily confused with surprise than disgust, etc. Schlosberg (1941, 1952) showed that these error patterns could be accommodated within a system comprising two continuous dimensions (pleasant± unpleasant and attention±rejection) with neutral at their cross-over point (origin); in a later paper a third dimension (sleepiness±arousal) was added (Schlosberg, 1954). Over the years a number of similar dimensional accounts of facial affect recognition have been proposed. The most widely cited modern variant is the Circumplex model (Russell, 1980; Russell & Bullock, 1985). This has a similar structure to the Schlosberg (1952) system, in that facial expressions are coded as values on two continuous dimensions, pleasantness and degree of arousal (Fig. 1). Russell has also shown that when the stimulus materials are emotional words, a similar Circumplex structure is found to that for facial expressions. This would suggest that these two-

Fig. 1. The Circumplex model of emotion representation; modi®ed from Bullock and Russell (1986). Vector 1 shows the anger expression caricatured relative to a neutral-expression norm. Vectors 2, 3 and 4 show the same anger prototype caricatured relative to happiness, fear and disgust expression norms, respectively. Note that the points depicting each emotion should be regarded as centroids of clusters; that is, each emotion does not have a precise set of co-ordinates.

A.J. Calder et al. / Cognition 76 (2000) 105±146

109

dimensional models have a strong conceptual basis. Nonetheless, some authors have presented evidence to suggest that similar dimensional systems can also accommodate the perceptual representation of facial expressions; that is, they have shown that dimensions describing the physical shape of facial expressions are correlated with dimensions such as pleasantness and arousal. Frijda (1969), for example, has shown that ratings of expressive facial features (upturned upper lip, corners of the mouth turned down, etc.) are correlated with the dimensions he identi®ed as underlying the representation of emotion. More recently, Yamada and colleagues (Yamada, 1993; Yamada, Matsuda, Watari & Suenaga, 1993) showed that factor analyses (and discriminant analyses) of the physical displacement of feature points (e.g. corner of the mouth, corner of the eye, etc.) in schematic or human facial expressions reveals two principal dimensions which they labelled `slantedness' and `curvedness/openness'. Furthermore, Yamada and Shibui (1998) have shown that values on these two structural scales are correlated with participants' ratings of the same stimuli for pleasantness and arousal, respectively. In short, there is a growing body of evidence to suggest that Russell's Circumplex model of facial affect recognition may constitute a valid description of both conceptual and perceptual (structural shape) representations of facial expression. But, if this type of two-dimensional model is to be accepted as a plausible `front-end' of facial expression processing, it is important that it should stand up to detailed empirical testing. The caricature procedure offers a unique way of investigating this issue. 1.2. Caricaturing relative to different reference norms The basic phenomenon of caricaturing, say, an anger expression relative to a neutral-expression norm, can be represented in the type of two-dimensional model discussed by extending the vector formed between the origin (neutral) and anger; this is illustrated by vector 1 in the Circumplex model shown in Fig. 1. Note that anger (undistorted) is associated with moderate arousal and low pleasantness, and that extending the vector between neutral and anger has the effect of slightly increasing the emotion's arousal component while decreasing its pleasantness component; this concurs with the ®nding that people see anger caricatures as `more angry' (Experiment 2). Thus, the two-dimensional model appears to provide an adequate perceptual account of caricaturing a facial expression's shape relative to a neutral reference norm. However, the computer-based caricature procedure enables one to use any face as the reference norm; it is not restricted to caricaturing relative to neutral. An anger expression, for example, can be caricatured relative to any other emotion by simply increasing the differences between the anger face and the expression in question. So what are the predictions of the two-dimensional model for the condition in which the reference norm is another canonical expression? Applying the principle used above, we can see that in a two-dimensional account, caricaturing an anger expression relative to a disgust-expression norm (vector 4) would have a different effect to caricaturing anger relative to the neutral-expression norm (vector 1). The disgust norm causes the anger expression's arousal value to

110

A.J. Calder et al. / Cognition 76 (2000) 105±146

increase while leaving its pleasantness value the same, and this has the effect of shifting the anger expression towards a region occupied by fear. When the norm is a fear expression (vector 3), however, anger is shifted in the opposite direction into the region occupied by disgust. A happiness-expression norm (vector 2), on the other hand, appears to have a similar effect to caricaturing relative to neutral. The important point conveyed by Fig. 1, then, is that a two-dimensional account predicts that caricaturing an expression relative to a series of `different-expression' reference norms should create a series of images that are emotionally different to one another. To test this prediction Experiment 3 examined participants' ratings of caricatures of three facial expressions (anger, fear and sadness), each caricatured relative to a series of different reference norms. Contrary to the prediction of the two-dimensional model, we found that all of the reference norms used showed a linear relationship between rated intensity and level of caricature: caricaturing relative to any other emotion served only to increase the perceived intensity of a given emotion ± it did not change that emotion. This ®nding is inconsistent with a two-dimensional model of perceptual representation of facial expression. In Experiment 4 we sought to replicate these ®ndings using caricatures exaggerated by a higher level. This was to determine whether our failure to con®rm the predictions of the two-dimensional model could be attributed to the maximum level of exaggeration used in Experiment 3 not being high enough. It is worth pointing out that the caricature procedure we used only manipulates a face's shape. Clearly, facial expressions contain more than just shape information, they also contain information relating to the face's texture (i.e. skin tone, whites of the eyes, etc.). However, as we discussed earlier, Yamada has shown that a factor analysis of facial measurements (i.e. shape information) alone generates a Circumplex structure similar to that observed by Russell and colleagues. This shows that emotion-relevant information suf®cient for a Circumplex structure is coded in the shape of the facial expressions, and consequently, we felt that `shape caricatures' provided an adequate test of the predictions of the two-dimensional model. 2. Experiment 1 In Experiment 1, one group of participants were asked to rate the emotional intensity of caricatured emotional facial expressions, and a second group were asked to rate a subset of these images for how `face-like' they looked. In this ®rst experiment, we restricted our investigation to three facial expressions in order to maximize the number of levels of caricature used. 2.1. Method 2.1.1. Participants Twenty-four members of the MRC Cognition and Brain Sciences Unit Subject Panel participated in the experiment for payment. All had normal or corrected-tonormal vision. Half of the participants (six male, six female; ages 17±27 years) took

A.J. Calder et al. / Cognition 76 (2000) 105±146

111

part in the emotional intensity ratings section, and the other half (®ve male, seven female; ages 17±29 years) in the face-like ratings section. 2.1.2. Materials Photographic-quality caricatures were prepared from colour photographs of six facial expressions (happiness, sadness, anger, fear, disgust and surprise) from the Matsumoto and Ekman (1988) pictures of Japanese and Caucasian facial expressions. For every emotion there were four different examples posed by one Caucasian male, one Caucasian female, one Japanese male, and one Japanese female. Different models were used for each of the six emotion categories, hence, each model was seen posing only one facial expression. Each facial expression was caricatured at seven different levels of exaggeration (275%, 250%, 225%, 0, 125%, 150% and 175%) relative to a picture of the same model posing a neutral facial expression (i.e. a neutral-expression reference norm). Preparation of photographic-quality caricatures involves three stages, and to illustrate, these are described below for a happy expression. For a more detailed description see Benson and Perrett (1991b). 2.1.3. Preparation of the caricatures 2.1.3.1. Stage 1: delineation Photographs of one happy expression and one neutral expression posed by the same model were frame-grabbed at a resolution of 512 £ 720 pixels. In our example, the happy expression is the `to-be-caricatured' expression, and the neutral expression is the `reference norm'. For each photograph, 195 points were manually positioned onto set anatomical landmarks on the model's face (e.g. the mouth was represented by 22 points, and each eyebrow by eight points). Hence, across the happy and neutral expressions there was conformity with respect to the anatomical positioning of the 195 points, but their exact spatial position could vary (e.g. the mouth had a different shape in the two expressions, but the shape of the hairline was the same). 2.1.3.2. Stage 2: caricaturing shape. The differences between the happy and neutral expressions were described by a set of 195 vectors joining the 195 points in the neutral face to the corresponding points in the happy face. The degree of exaggeration was calculated on a percentage scale; hence, to create a 150% caricature, each vector was extended beyond the point on the happy face by multiplying its length by a factor of 1.5. The new resultant endpoints of the vectors were then joined to produce a caricatured happiness face shape. In short, the feature points on the to-be-caricatured expression (happiness) were shifted by a speci®ed degree of exaggeration relative to the same anatomical points on the reference norm (neutral expression). 2.1.3.3. Stage 3: producing a continuous-tone image. A continuous-tone (photographic quality) image of the caricatured happy expression was produced by ®rst dividing the original photograph of the happy expression into mesh of

112

A.J. Calder et al. / Cognition 76 (2000) 105±146

triangular tessera. An example of a triangulation might comprise the inner-most points of the right and left eyebrows and the mid-point of the hairline. Next, exactly the same points were joined in the caricatured expression so that there was correspondence between the relative location of the triangles in both the original and caricatured faces. Finally, the greyscale pixel values of each triangle in the original happy face were mapped onto the corresponding triangle in the caricatured happy face. Where any of the caricatured triangles was larger than in the original image `stretching' of the spatial distribution occurred; similarly, `shrinking' of the spatial distribution occurred when one of the caricatured triangles was smaller. Exactly the same procedure was repeated for each of the other expressions. Examples of the caricatures are shown in Fig. 2. 2.1.4. Design and procedure 2.1.4.1. Emotional intensity ratings Participants were presented with three blocks of trials, each containing three types of caricatured expression which we will refer to as TARGET, SIMILAR, and OTHER. Participants were asked to rate the caricatures in each block for intensity of a particular emotion; the scales ranged from 0 (e.g. not happy) to 10 (e.g. very happy). The intensity rating scale corresponded to the emotion displayed in the TARGET images, and in what follows each block is identi®ed by its emotional rating scale. The SIMILAR expressions were a facial affect that is on occasions confused with the TARGET (e.g. surprise is sometimes misidenti®ed as happiness), and the OTHER expressions corresponded to a third emotion selected from one of the four remaining expressions in the stimulus set (see above). The TARGET, SIMILAR and OTHER expressions for the three blocks were as follows: `intensity of fear' block: TARGET, fear; SIMILAR, sadness; OTHER, anger; `intensity of happiness' block: TARGET, happiness; SIMILAR, surprise; OTHER, fear; `intensity of sadness' block: TARGET, sadness; SIMILAR, disgust; OTHER, happiness. The stimuli were arranged so that the target expressions in one block were also seen in a second block in the SIMILAR or OTHER conditions. Each block contained four different examples of each of the TARGET, SIMILAR and OTHER caricature sequences. These were displayed individually, and in random order on a colour monitor. On each trial the target remained in view until the participant responded by pressing one of 11 keys, marked 0±10. Participants were asked to base their ratings on their ®rst impression and not spend time studying the images. No feedback was given. Each block was preceded by 20 practice trials selected at random from the stimulus materials for the particular block. The order of block presentation was counterbalanced across participants. 2.1.4.2. Face-like ratings. A different group of participants were asked to rate the TARGET images used in the ®rst part of the experiment for how `face-like' they looked. They were told the images had been produced by a computer system, and that we were interested in determining how realistic each face looked. For each face they were instructed to concentrate on the overall face shape, and to consider whether the facial features (eyes, nose, mouth, etc.) were in the correct

Fig. 2. From Experiment 1 ± examples of the three TARGET facial expressions caricatured at seven levels of exaggeration (275%, 250%, 225%, 0%, 125%, 150% and 175%) relative to a neutral-expression norm; note, 0% is the veridical (undistorted) expression. The emotions shown are fear (top row), happiness (middle row) and sadness (bottom row).

A.J. Calder et al. / Cognition 76 (2000) 105±146 113

114

A.J. Calder et al. / Cognition 76 (2000) 105±146

proportions and positions, and whether a person could actually look like this. No mention was made of the images being caricatured. The rating scale ranged from 0 (implausible face) to 10 (very plausible face). Each image was presented once in random order, and remained on the screen until the participant responded. Participants were asked to base their ratings on their ®rst impression and not on their time spent studying the images. No feedback was given. 2.2. Results 2.2.1. Emotional intensity ratings Participants' mean ratings (with standard error bars) for `intensity of fear', `intensity of happiness', and `intensity of sadness' are graphed separately in Fig. 3. Each graph shows ratings of the TARGET, SIMILAR and OTHER expressions caricatured at the seven different levels (275%, 250%, 225%, 0%, 125%, 150% and 175%). For each rating scale, participants' ratings of the four examples of each expression type (TARGET, SIMILAR and OTHER) are pooled to give one mean rating for each level of caricature. Separate ANOVAs were carried out on the participants' ratings for `intensity of fear', `intensity of happiness', and `intensity of sadness'. For each, a linear trend analysis of the level of caricature factor was investigated in the context of a two-factor design. The factors investigated were expression type (TARGET, SIMILAR and OTHER; repeated measure) and level of caricature (275%, 250%, 225%, 0%, 125%, 150% and 175%; repeated measure). To facilitate interpretation of the results, the F values from the three analyses are summarized in Table 1. All three ANOVAs (`intensity of fear', `intensity of happiness', and `intensity of sadness') showed signi®cant main effects of level of caricature; each caricature Table 1 From Experiment 1 ± F values for the three two-factor ANOVAs examining participants' emotional intensity ratings of fear, happiness and sadness (rating scale) a,b d.f.

Rating scale Fear

Happiness

Sadness

Level of caricature effect Trend analysis Linear component Non-linear component

F(6,66)

6.26**

30.52**

4.63**

F(1,66) F(5,66)

28.70** 1.78 ns

163.96** 3.83*

7.69** 4.02*

Expression type

F(2,22)

80.90**

236.86**

102.25**

Interaction effect Trend analysis Linear component Non-linear component

F(12,132)

15.69**

17.79**

8.88**

F(2,132) F(10,132)

90.07** 0.61 ns

100.75** 1.19 ns

49.77** 0.70 ns

a Factors investigated were level of caricature (275%, 250%, 225%, 0%, 125%, 150% and 175%; repeated measure) and expression type (TARGET, SIMILAR, and OTHER; repeated measure). b *P , 0.01, **P , 0.001, and nsP . 0.1.

Fig. 3. From Experiment 1 ± mean ratings for `intensity of fear' (left), `intensity of happiness' (middle) and `intensity of sadness' (right) are graphed separately. Each graph shows participants' mean intensity ratings with standard error bars for the TARGET, SIMILAR and OTHER expressions at seven levels of exaggeration (275%, 250%, 225%, 0%, 125%, 150% and 175%). All caricatures were prepared relative to a neutral-expression norm.

A.J. Calder et al. / Cognition 76 (2000) 105±146 115

116

A.J. Calder et al. / Cognition 76 (2000) 105±146

effect showed a strong linear trend with no signi®cant deviation from the linear (Table 1). For each ANOVA, the main effect of caricature was quali®ed by a signi®cant interaction between level of caricature and expression type (TARGET, SIMILAR and OTHER). In all cases, the interaction can be accounted for by differences in the linear component (slope) of each expression type; any differences in the remaining (non-linear) components of the interaction were not signi®cant (Table 1). Table 2 shows the estimated slopes of the three expression types in units of emotional intensity rating per 100% increase in level of caricature. For all three analyses, the TARGET caricature sequences showed a signi®cant increase, and the slope was more marked than those observed for the SIMILAR or OTHER images. The slope of the SIMILAR caricature sequences showed a signi®cant increase for the `intensity of happiness' analysis, and signi®cant and non-signi®cant decreases for the `intensity of fear' and `intensity of sadness' analyses, respectively. The slope of the OTHER caricature sequences showed a non-signi®cant increase for the `intensity of fear' analysis, and signi®cant and non-signi®cant decreases for the `intensity of sadness' and `intensity of happiness' analyses, respectively. Finally, all three analyses showed a signi®cant main effect of expression type (Table 1). In all three cases, the TARGET caricatures were rated as more intense expressions of the target emotion than were the SIMILAR or OTHER caricatures. 2.2.2. Face-like ratings Participants' mean `face-like' ratings (with standard error bars) for the TARGET caricatures are shown in Fig. 4 (open triangles, left y-axis). Ratings are pooled across the three TARGET expressions (fear, happiness and sadness) to give one mean rating for each level of caricature. The mean emotional intensity ratings for the TARGET images are shown in the same graph (®lled squares, right y-axis) for comparison. The emotional intensity ratings and face-like ratings of the TARGET images were submitted to a two-factor ANOVA examining the variables rating scale (intensity rating and face-like rating; between subjects) and level of caricature (275%, 250%, 225%, 0%, 125%, 150% and 175%; repeated measure). The interaction between these two factors was highly signi®cant (F…6; 132† ˆ 42:47, P , 0:0001). Face-like ratings were also submitted to a separate ANOVA examinTable 2 From the linear trend analyses for Experiment 1 ± the estimated slopes of the three expression types (TARGET, SIMILAR and OTHER) are shown in units of emotional intensity rating per 100% increase in level of caricature for each of the three blocks of ratings (fear, happiness and sadness) a Expression type

Fear Happiness Sadness

TARGET

SIMILAR

OTHER

1.465 2.133 0.911

2 0.465 0.693 2 0.137 ns

0.205 ns 2 0.057 ns 2 0.306

Standard error of estimate ^ 0.113 ^ 0.116 ^ 0.095

a ns Indicates a slope not signi®cantly different from 0. Otherwise slopes are signi®cantly different from 0 with P , 0:05 or less.

A.J. Calder et al. / Cognition 76 (2000) 105±146

117

Fig. 4. From Experiment 1 ± participants' mean `face-like' ratings (open triangles, left-hand y-axis) with standard error bars for the TARGET expressions caricatured at seven levels of exaggeration (275%, 250%, 225%, 0%, 125%, 150% and 175%). Participants' mean intensity ratings (®lled squares, righthand y-axis) with standard errors are also shown for comparison. All caricatures were prepared relative to a neutral-expression norm.

ing one repeated-measure factor, level of caricature (275%, 250%, 225%, 0%, 125%, 150% and 175%). The results showed a signi®cant effect of level of caricature (F…6; 66† ˆ 23:10, P , 0:0001). Post-hoc t-tests showed that participants' ratings of the 250, 225 and 125% images did not signi®cantly differ from the veridical (0%) images; the 175% caricatures were rated as the least face-like, followed by the 150 and 275% which did not reliably differ. 2.3. Discussion The results demonstrate that the TARGET expressions in each block were consistently rated as increasingly more intense with increasing levels of caricature. In contrast, for the main part, the SIMILAR and OTHER expressions did not show

118

A.J. Calder et al. / Cognition 76 (2000) 105±146

this pattern. This indicates that participants were able to attend to the particular emotion they were asked to rate, and that their ratings re¯ect changes in the TARGET emotion's intensity rather than changes in a more general factor such as level of arousal or `distortedness'. The fact that the TARGET stimuli from each block did not show a caricature advantage when they were shown in the SIMILAR or OTHER conditions of a different block further emphasizes this conclusion. There was one exception to the pattern described above, and this is that the surprise (SIMILAR) images in the `intensity of happiness' block showed a small effect of caricature ([275 and 250%Š , ‰225%, 0%, 125%, 150% and 175%]). One interpretation of this ®nding is that the surprise expressions in the JACFEE series seem to signal a response to a pleasant (happy) rather than unpleasant unexpected event. Nonetheless, the pattern found for surprise is clearly different to the strikingly linear effect found for the TARGET happiness images from the same block. It is also worth noting that of the 12 TARGET caricature sequences used (four examples of each of fear, happiness and sadness) all but one sequence showed a signi®cant positive correlation with level of caricature (fear: r…5† ˆ 1:0, 0.85, 0.96, 0.86, P , 0:01 (one tail); happiness: r…5† ˆ 1:0, 0.96, 1.0, 1.0, P , 0:05 (one tail); sadness: r…5† ˆ 0:88, 1.0, 0.68, P , 0:5 (one tail); r…5† ˆ 0:63, 0:1 . P . 0:5 (one tail)). These correlations illustrate that the caricature effect was not restricted to one or two TARGET faces that happened to caricature well. 2.3.1. Face-like ratings Participants' `face-like' ratings show that although the 175% TARGET caricatures were perceived as the most emotionally intense facial expressions, they were also rated as the least `face-like' of the images employed. This implies that our representation of facial expression is coded separately from our representation of what is face-like. Moreover, it suggests that facial expressions may be coded at a level that does not incorporate a full representation of the facial image. The participants' face-like ratings also demonstrate that the caricature effect for facial expression works beyond the range of intensity experienced in natural facial expressions. This is because the maximal intensity ratings were attributed to facial images that the participants rated as the least plausible-looking faces. Overall, then, these results show that people are not only accomplished at recognizing certain facial emotions (Ekman, 1982), they are also highly sensitive to changes in their intensity. This would ®t with the idea that facial expressions of the basic emotions are not simply coded as discrete category representations, but rather as continua varying in emotional intensity. In short, these results are consistent with the hypothesis that caricaturing facial expressions works by enhancing emotional intensity. 3. Experiment 2 Experiment 1 used caricatures of facial expressions associated with three basic

A.J. Calder et al. / Cognition 76 (2000) 105±146

119

emotions (fear, happiness and sadness). In Experiment 2 we assessed whether the caricature effect generalized to all six of the so-called basic emotions in the Ekman and Friesen (1976) series. 3.1. Method 3.1.1. Participants Ten students (®ve male, ®ve female) from the University of Cambridge participated in the experiment. The participants were between the ages of 20 and 25 years and had normal or corrected-to-normal vision. None had taken part in Experiment 1. 3.1.2. Materials Photographic-quality images of model JJ from the Ekman and Friesen (1976) series, posing one example of each of six basic emotions (happiness, sadness, anger, fear, disgust and surprise) were prepared at three levels of caricature (250%, 0% and 150%). The norm was a picture of the same model (JJ) posing a neutral expression. The resultant sets of images are shown in Fig. 5. Each facial image was printed onto an A4 sheet (resolution of 300 dpi), and was approximately 12 £ 8.5 cm. 3.1.3. Design and procedure The 18 stimuli were presented individually in six separate blocks; each block contained one presentation of the images. The participants rated the images on a different emotion scale for each block. The scales were as follows: `intensity of happiness', `intensity of surprise', `intensity of fear', `intensity of sadness', `intensity of disgust' and `intensity of anger'. Each scale ranged from 0 (e.g. not happy) to 10 (e.g. very happy). The order of presentation was random within each block, and the order in which the emotions were rated was counterbalanced across participants. No response feedback was given. After a period of a few days the participants completed the same procedure again. Hence, each participant rated the 18 images twice on each of the six emotion scales. 3.2. Results and discussion The participants' mean ratings (with standard error bars) are summarized in Fig. 6; the data for the six rating scales (`intensity of anger', `intensity of fear', etc.) are graphed separately. The mean ratings for each emotion scale were submitted to separate two-factor ANOVAs. In each analysis, the factors investigated were target emotion (happiness, sadness, anger, fear, disgust and surprise; repeated-measure) and level of caricature (250%, 0% and 150%; repeated measure). To facilitate interpretation of the results, we have summarized the F values for the main effects (level of caricature and target emotion) and interaction effects in Table 3. All analyses showed a signi®cant interaction between level of caricature and emotion. A summary of the simple effects for the level of caricature factor along with a breakdown of the main effect of emotion is described separately for each ratings scale in the text that follows.

120

A.J. Calder et al. / Cognition 76 (2000) 105±146

A.J. Calder et al. / Cognition 76 (2000) 105±146

121

Table 3 From Experiment 2 ± F values for the six two-factor ANOVAs examining participants' emotional intensity ratings of each of happiness, surprise, fear, sadness, disgust and anger (rating scale) a,b Rating scale

Level of caricature F(2,18)

Target emotion F(5,45)

Interaction F(10,90)

Happiness Surprise Fear Sadness Disgust Anger

8.63** 8.10** 1.35 (NS) 5.47* 2.01 (NS) 5.04*

100.63*** 98.93*** 64.84*** 124.94*** 37.33*** 61.01***

18.50*** 6.06*** 6.86*** 6.28*** 2.85** 6.84***

a

Factors investigated were level of caricature (250%, 0% and 150%; repeated measure) and target emotion (happiness, surprise, fear, sadness, disgust and anger; repeated measure). b *P , 0.05, **P , 0.01 and ***P , 0.0001.

3.2.1. `Intensity of happiness' ratings Simple effects analyses showed an effect of caricature for the happiness images only (F…2; 18† ˆ 25:16, P , 0:0001); post-hoc t-tests (P , 0:05) showed that for happiness, the caricature (150%) was rated as signi®cantly more intense than the veridical (0%) image, and this more intense than the anti-caricature (250%) image (150 . 0 . 250%). A breakdown of the effect of target emotion with post-hoc ttests (P , 0:05) showed that happiness was rated as signi®cantly more happy than the other emotion images. 3.2.2. `Intensity of surprise' ratings Simple effects analyses showed a signi®cant effect of caricature for the surprise (F…2; 18† ˆ 42:01, P , 0:0001) and fear images (F…2; 18† ˆ 4:23, P , 0:05) only; post-hoc t-tests (P , 0:05) showed the same pattern for surprise and fear (150 . 0 . 250%). A breakdown of the effect of target emotion with post-hoc t-tests (P , 0:05) showed that surprise and fear images were rated signi®cantly higher than the other emotion images, and surprise signi®cantly higher than fear. 3.2.3. `Intensity of fear' ratings Simple effects analyses showed an effect of caricature for the fear (F…2; 18† ˆ 24:35, P , 0:0001) and surprise images (F…2; 18† ˆ 6:61, P , 0:01) only. Post-hoc t-tests (P , 0:05) showed similar patterns for fear (150 . 0 . 250%) and surprise (150 . 0 ˆ 250%). A breakdown of the effect of target emotion with post-hoc t-tests (P , 0:05) showed that fear and surprise images were rated signi®cantly higher than the other emotion images, and fear signi®cantly higher than surprise. Fig. 5. From Experiment 2 ± the neutral-expression reference norm and sets of caricatured expressions. Six facial expressions posed by a single male were caricatured at three levels of exaggeration (250%, 0% and 150%) relative to a neutral-expression norm (top). From top to bottom the six emotions shown are happiness (top row), surprise (second row), fear (third row), sadness (fourth row), disgust (®fth row), and anger (bottom row).

122

A.J. Calder et al. / Cognition 76 (2000) 105±146

Fig. 6. From Experiment 2 ± participants' mean ratings for `intensity of happiness', `intensity of surprise', `intensity of fear', `intensity of sadness', `intensity of disgust', and `intensity of anger'. Each graph shows participants' mean intensity ratings (with standard error bars) for each emotion at three levels of caricature (250%, 0% and 150%). The caricatures were prepared relative to a neutral-expression norm.

3.2.4. `Intensity of sadness' ratings Simple effects analyses showed signi®cant effects of level of caricature for the sadness images (F…2; 18† ˆ 10:64, P , 0:0001), happiness images (F…2; 18† ˆ 2:01, P , 0:05), surprise images (F…2; 18† ˆ 12:32, P , 0:0001), fear images (F…2; 18† ˆ 5:98, P , 0:01) and anger images (F…2; 18† ˆ 4:21, P , 0:05). Posthoc t-tests (P , 0:05) showed that whereas sadness images showed increasing

A.J. Calder et al. / Cognition 76 (2000) 105±146

123

intensity ratings with level of caricature (250 , 0 , 150%), the opposite was found for the other emotions (250 $ 0 $ 150%). There was no signi®cant effect of caricature for the disgust images. A breakdown of the effect of target emotion with post-hoc t-tests (P , 0:05) showed that overall, the sadness images were rated signi®cantly higher than the other emotion categories. 3.2.5. `Intensity of disgust' ratings Simple effects analyses showed a signi®cant effect of caricature for the disgust images only (F…2; 18† ˆ 18:82, P , 0:0001); post-hoc t-tests (P , 0:05) showed the pattern 150 . 0 . 250%. A breakdown of the effect of target emotion with post-hoc t-tests (P , 0:05) showed that the disgust images were rated as signi®cantly more intense than the other emotions. 3.2.6. `Intensity of anger' ratings Simple effects analyses showed a signi®cant effect of caricature for the anger images only (F…2; 18† ˆ 25:16, P , 0:0001); post-hoc t-tests (P , 0:05) showed the pattern 150 . 0 . 250%. A breakdown of the effect of target emotion with post-hoc t-tests (P , 0:05) showed that the anger images were rated signi®cantly higher than the other emotion categories. The results of these ANOVAs replicate the ®ndings of Experiment 1. But whereas Experiment 1 demonstrated a caricature effect for three basic emotions (fear, happiness and sadness), Experiment 2 shows that the effect is evident for all six emotions from the Ekman and Friesen (1976) series. Moreover, we can see that when the number of different expression categories is increased to six, participants remain able to attend selectively to the intensity of the emotion they are asked to rate. This is consistent with Ekman and colleagues' idea that facial expressions are to some extent represented as discrete categories. The possible exception to this observation is that fear and surprise caricatures (250%, 0% and 150%) were both rated as increasingly more afraid and more surprised. Notably, of the six emotional facial expressions we have used, fear and surprise are the most commonly confused. This pattern is also re¯ected in the multidimensional scaling analysis of the same data (see Section 3.2.7), and is consistent with previous suggestions that surprise may not constitute a separate emotional expression in its own right (Ekman, 1973; Etcoff & Magee, 1992). The results of Experiment 1 also support this conclusion, because here the surprise caricatures were rated as increasingly `more happy'. 3.2.7. Multidimensional scaling As a means of presenting the data in a more `short-hand' graphic form, the participants' mean ratings for the caricatured (150%), veridical (0%), and anticaricatured (250%) representations of the six facial expressions were submitted to a replicated non-metric multidimensional scaling (R-MDS) procedure (ALSCAL, from the Statistical Package for Social Sciences; SPSS, 1994). R-MDS applies a Euclidean distance model to each of the participant's ratings matrices simultaneously. This allows us to generate a geometric representation (in one or more

124

A.J. Calder et al. / Cognition 76 (2000) 105±146

dimensions) of the similarity (dissimilarity) amongst the caricatured, veridical and anti-caricatured representations of the six facial expressions. The extent to which the similarity between these images can be expressed within the models obtained, or `goodness of ®t', is expressed by two factors: the Kruskal (1964) stress measure, which ranges from 0 (perfect ®t) to 1 (no ®t), and RSQ, the squared simple correlation between the transformed data and their distances in the resultant model. The RSQ represents the proportion of variance that is accounted for by the distances amongst points in the MDS model. Two- and three-dimensional models were generated: the two-dimensional MDS model accounted for 89% of the variance (RSQ 0.889, stress 0.148), while the threedimensional model accounted for 99% of the variance (RSQ 0.985), and with lower stress (stress 0.047). Given that the ®rst two dimensions of the three-dimensional solution are very similar to those obtained in the two-dimensional model, we only present the data from the three-dimensional model in graphical form. The three possible two-dimensional plots of this model are shown in Fig. 7; hence, the dimension 1 versus 2 plot (Fig. 7A) corresponds to the two-dimensional model. It is striking that with the exception of sadness, each emotion in the two-dimensional model (i.e. the dimension 1 versus 2 plot) shows the same basic pattern: moving away from the origin, the three levels of caricature (250%, 0% and 150%) are positioned in the order anti-caricature, veridical and then caricature, as if they fall along a single vector (Fig. 7A). In fact, a similar pattern is found for the dimension 1 versus 3 (Fig. 7B) and dimension 2 versus 3 (Fig. 7C) plots. Sadness, however, is represented rather differently: the images (250%, 0% and 150%) show similar values on the ®rst two dimensions, and are only clearly differentiated on the third. The overall geometrical representation, then, re¯ects the ®ndings of the conventional analysis of Experiment 2 (and Experiment 1); that is, `intensity of emotion' is a monotonic function of level of caricature. We have used the MDS procedure here as a convenient way to summarize the data, and we do not wish to make any strong claims regarding the mental representation of facial expressions from the resultant model. However, it is worth noting a couple of interesting points. First, no single dimension seems to code emotional intensity. Rather, emotional intensity is an interaction of the three dimensions. Second, for dimensions 1 and 2 of the three-dimensional model (Fig. 7A, and similarly for the two-dimensional), the emotions have the same relative positions as in Russell's Circumplex model (Russell, 1980; Russell & Bullock, 1985) shown in Fig. 1. Furthermore, the anti-caricature, veridical and caricature representations in our MDS model are positioned progressively further away from the origin, which is also consistent with the Circumplex structure. Hence, on the basis of these results, a two-dimensional model would seem to present a reasonably good account of the effects of caricaturing facial expressions, or at least, caricaturing expressions relative to a neutral expression reference norm. But as we discussed in Section 1, the computer caricature procedure also enables us to exaggerate the differences between a particular expression and any comparison facial reference norm. We also discussed that a two-dimensional account of the perceptual representation of facial expression predicts that caricaturing an expres-

Fig. 7. From Experiment 2, the three-dimensional MDS model obtained from participants' ratings of the expressions happiness, sadness, anger, fear, disgust and surprise; all images were rated for their intensity of the six basic emotions. For each emotion category the caricature (150%) is represented by the largest shape, the anticaricature (250%) by the smallest, and the veridical (0%) by the medium-sized shape. (A) Dimensions 1 and 2 ± this effectively constitutes the two-dimensional model of the data which accounted for 89% of the variance. (B) Dimensions 1 and 3 of the three-dimensional MDS model. (C) Dimensions 2 and 3 of the three-dimensional MDS model.

A.J. Calder et al. / Cognition 76 (2000) 105±146 125

126

A.J. Calder et al. / Cognition 76 (2000) 105±146

sion (e.g. anger) relative to a series of different facial expressions should create a series of emotionally different caricatures (see Fig. 1). To further assess the validity of the two-dimensional model we investigated this prediction in Experiment 3. 4. Experiment 3 Experiment 3 used caricatures of three facial expressions (anger, fear and sadness) prepared relative to three types of reference norm. The norms used were as follows: (1) a neutral-expression norm; (2) an average-expression norm; and (3) a set of three different-expression norms. Note, an average-expression norm condition was included because we felt that the origin of the two-dimensional accounts might be better conceived as an average, rather than a neutral expression, in which case we would expect to ®nd a maximal caricature advantage for the average-expression norm images. If two-dimensional models provide an accurate description of the perceptual representation of facial expressions, then we would expect to ®nd two results. First, the three sets of different-expression norm caricatures for each emotion should generate statistically different patterns of ratings. Second, the different-expression norm caricatures should, at the very least, produce a less marked effect of caricature than the neutral-expression and average-expression norm conditions. 4.1. Method 4.1.1. Participants Twelve members of the MRC Cognition and Brain Sciences Unit Subject Panel (seven male, ®ve female) participated in the experiment for payment. All were aged between 18 and 40 years and had normal or corrected-to-normal vision. None had taken part in Experiments 1 and 2. 4.1.2. Materials Photographic-quality caricatures were prepared from monochrome photographs of model JJ from the Ekman and Friesen (1976) series of pictures of facial affect. One example of each of the expressions anger, fear and sadness was caricatured at four different levels (0%, 115%, 130% and 150%) and relative to three different types of norm. The norms were as follows: (1) a neutral-expression norm (which was a picture of model JJ posing a neutral facial expression); (2) an average-expression norm, namely the average position of feature points on model JJ posing one example of the six facial expressions happiness, sadness, anger, fear, disgust and surprise; and (3) a set of three different-expression norms. The different-expression norms were different for each emotion. For anger, they were disgust, fear and happiness; for fear, they were surprise, sadness and anger; for sadness, they were fear, disgust and happiness. For each to-be-caricatured expression, the ®rst two different-expression norms listed are the two expressions that are most often confused with the target (Ekman & Friesen, 1976); and the third expression norm was selected from the remaining three expressions.

A.J. Calder et al. / Cognition 76 (2000) 105±146

127

Caricaturing each of the target expressions relative to three different-expression norms meant that there were three times as many different-expression norm caricatures as average-expression or neutral-expression norm caricatures. To balance the design, three copies each of the neutral-expression and average-expression norm images were used; this gave 36 stimuli in each of the neutral-expression, averageexpression and different-expression norm conditions for each target expression. Preparation of photographic-quality caricatures followed the method outlined for Experiment 1; however, for the average-expression norm condition the expressions were caricatured relative to the feature points on the average expression, and for the different-expression norm images relative to the feature points on the appropriate different expression. Figs. 8A±E show examples of the neutral-expression norm, average-expression norm, and different-expression norm caricatures. In each ®gure the corresponding norm expression for each row is shown on the far left. Each of the face images was printed onto an A4 sheet using the method outlined for Experiment 2. 4.1.3. Design and procedure Presentation of the caricatures was blocked by target expression (anger, fear and sadness). The faces were presented individually, and the participants were asked to rate the intensity of each expression on a scale from 1 to 7. For the anger block the participants rated each face for `intensity of anger', where 1 was `not angry' and 7 was `very angry'. Similarly, for the fear and sadness blocks the faces were rated for `intensity of fear' and `intensity of sadness', respectively. Participants were asked to respond promptly to each face, basing their answers on their ®rst impression. No response feedback was given. Within each block the images were presented in a different random order for each participant. All participants were presented with all three blocks of caricatures; the order of block presentation was randomized across participants. 4.2. Results The participants' mean ratings (with standard error bars) to the anger, fear and sadness caricatures (0%, 115%, 130% and 150%) prepared relative to neutralexpression, average-expression and different-expression norms are summarized in Fig. 9. Note that for each target expression (anger, fear and sadness) the participants' data for the three different-expression norm images are pooled to give one mean rating for each level of caricature. The separate mean ratings for each of the different-expression norm caricatures are presented in Appendix A. A preliminary analysis examined participants' ratings of the three types of different-expression norm caricatures for each target expression (anger, fear and sadness) in three separate ANOVAs. Each ANOVA examined two factors, level of caricature (0%, 115%, 130% and 150%; repeated measure) and expression norm (the three different-expression norms differed for each of the three target expressions; see Section 4.1 for expression norms used). All three analyses showed a highly signi®cant effect of caricature (anger: F…3; 11† ˆ 48:62, P , 0:0001; fear: F…3; 11† ˆ 43:68,

128

A.J. Calder et al. / Cognition 76 (2000) 105±146

A.J. Calder et al. / Cognition 76 (2000) 105±146

129

P , 0:0001; sadness: F…3; 11† ˆ 18:17, P , 0:0001). None showed a signi®cant effect of norm or a signi®cant interaction effect between norm and level of caricature. This demonstrates that, contrary to the predictions of a two-dimensional model, caricaturing a face relative to a series of different-expression norms has no signi®cant effect on the pattern of intensity ratings observed. In view of these ®ndings, participants' responses to the three different-expression norm caricatures are pooled in the following analyses to give one mean rating for each of the four levels of caricature (0%, 115%, 130% and 150%) for each target expression (anger, fear and sadness). A linear trend analysis of the level of caricature factor was investigated in the context of a three-factor design. The factors investigated were type of norm (neutralexpression, average-expression or different-expression; repeated measure) target emotion (anger, fear and sadness; repeated measure), and level of caricature (0%, 115%, 130% and 150%; repeated measure). The results showed a signi®cant main effect of caricature (F…3; 33† ˆ 91:06, P , 0:001), which showed a strong linear trend (F…1; 33† ˆ 272:10, P , 0:001) with no deviation from the linear (F , 1). This was quali®ed by a signi®cant interaction between level of caricature and type of norm (F…6; 66† ˆ 3:92, P , 0:005); this interaction can be accounted for by differences in the linear function of each type of norm (neutral-expression, averageexpression or different-expression) and level of caricature (F…2; 66† ˆ 10:82, P , 0:001); the non-linear components of this interaction did not reach statistical signi®cance (F , 1). A breakdown of the interaction effect between type of norm and level of caricature (0%, 115%, 130% and 150%) showed signi®cant linear trends of level of caricature for the neutral-expression (F…1; 33† ˆ 129:85, P , 0:001), average-expression (F…1; 33† ˆ 96:20, P , 0:001), and differentexpression (F…1; 33† ˆ 310:62, P , 0:001) norm conditions, with no signi®cant deviation from linear in each case (all F , 1). A post-hoc contrast analysis of the difference in slopes among the three norm conditions showed that the slope of the average-expression norm caricatures is signi®cantly less marked than the neutralexpression and different-expression norm slopes (F…1; 66† ˆ 18:58, P , 0:001), which do not reliably differ. Again, this ®nding is inconsistent with the two-dimensional model which predicts that the different-expression norms should show the least marked effect of caricature. There was also a borderline interaction between level of caricature and target Fig. 8. From Experiment 3, three facial expressions (anger, fear, and sadness) posed by model JJ were caricatured at four levels of exaggeration (0%, 115%, 130% and 150%) relative to neutral-expression norm (A), and an average expression norm (B). From top to bottom the three emotions shown in (A) and (B) are anger (top row), fear (second row) and sadness (bottom row). (C) Caricatures of JJ's fear expression prepared relative to three different-expression norms posed by the same model. The norm expressions used were as follows: a surprise expression (top row), a sadness expression (middle row) and an anger expression (bottom row). (D) Caricatures of the JJ's anger expression prepared relative to three differentexpression norms posed by the same model. The norm expressions were as follows: a disgust expression (top row), a fear expression (middle row) and a happiness expression (bottom row). (E) Caricatures of JJ's sadness expression prepared relative to three different-expression norms posed by the same model. The norm expressions were as follows: a fear expression (top row), a disgust expression (middle row) and a happiness expression (bottom row) posed by the same model. In all ®ve sections of the ®gure, the norm expressions are shown on the left.

130

A.J. Calder et al. / Cognition 76 (2000) 105±146

Fig. 9. From Experiment 3 ± participants' mean intensity ratings with standard error bars are shown for facial expressions of anger (left), fear (middle) and sadness (right) caricatured at four levels of exaggeration (0%, 115%, 130% and 150%) and relative to three different types of expression norms: a neutralexpression norm (Neutral), an average-expression norm (Average), and different-expression norms (Different). For each graph the data from the three different-expression norm images used are pooled to give one value for each level of caricature.

emotion (F…6; 66† ˆ 2:09, 0:1 . P . 0:05); this can be accounted for by differences in the linear function of each target emotion (anger, fear and sadness) with level of caricature (F…2; 66† ˆ 4:69, P , 0:05); the non-linear components of the interaction were not signi®cant (F , 1). A breakdown of this interaction effect showed signi®cant linear trends of level of caricature for each of the anger (F…1; 33† ˆ 169:88, P , 0:001), fear (F…1; 33† ˆ 148:84, P , 0:001), and sadness images (F…1; 33† ˆ 51:04, P , 0:001), with no signi®cant deviation from linear in each case (all F , 1:3). A post-hoc contrast analysis of the difference in slopes among the three target emotions showed that the slope of the sadness images was signi®cantly less marked than the slopes for anger and fear images (F…1; 66† ˆ 7:97, P , 0:01), which did not reliably differ. There was a signi®cant effect of type of norm (F…2; 22† ˆ 9:61, P , 0:001). Posthoc t-tests (P , 0:05) showed that, overall, the neutral-expression norm caricatures were rated as more intense than the average-expression and different-expression norm caricatures, which did not reliably differ. Finally, there was a signi®cant effect of target emotion (F…2; 22† ˆ 6:83, P , 0:01). Post-hoc t-tests (P , 0:05) showed that the sadness images were rated as less intense than the fear and anger images, which did not reliably differ.

A.J. Calder et al. / Cognition 76 (2000) 105±146

131

4.3. Discussion We can draw two conclusions from the results of Experiment 3. First, they replicate the ®ndings of Experiments 1 and 2; caricaturing facial expressions has a signi®cant effect on their perceived intensity ± the more positive the degree of caricature, the higher the emotional intensity rating attributed to the expression. The second point is more interesting. To the extent that two-dimensional models provide an accurate account of the perceptual representation of facial expressions, then they predict that caricaturing expressions relative to different-expression norms should produce a series of perceptually, and hence, `emotionally' different images. Thus, the three sets of different-expression norm caricatures of each emotion should produce different patterns of intensity ratings. In addition, the different-expression norm condition should produce a less marked caricature effect than the neutralexpression and average-expression norm caricatures, or no caricature effect at all. These predictions were not con®rmed. In fact, it was the average-expression norm images that produced the least marked caricature effect, and the different-expression condition images did not reliably differ from the neutral-expression norm images. These results demonstrate that although the norm used to prepare the caricatures has some effect on intensity ratings, it does not affect the basic pattern of increasing emotional intensity with increasing levels of caricature. This in¯uence of the norm can be explained in terms of the computer-based procedure used to generate the caricature images. That is, the caricature generator operates by exaggerating the physical differences between the to-be-caricatured expression and the norm expression. The more similar these two starting images are, the smaller the observable differences between the resultant veridical (0%) and 150% caricatured images, and the smaller the caricature effect. Given that the average norm consists of the mean feature positions across six facial expressions, it contains traces of these six expressions (i.e. one-sixth of each). Consequently, it follows that the average-expression condition should produce the smallest difference between the veridical (0%) and 150% caricature images. This is exactly the pattern we observed. Similarly, the same interpretation applies to the interaction effect between target emotion and level of caricature. It follows that the sadness expression should produce a less marked linear effect of level of caricature than the anger and fear expressions because the sadness expression is the least physically distinctive of the three expressions. Again this is exactly the effect that we found. In short, we have found little support for the predictions of the two-dimensional model for caricaturing expressions relative to different-expression norms. This suggests that the twodimensional model does not provide an adequate account of the perceptual representation of facial expressions. A proponent of a two-dimensional account might argue that one reason why our results do not support the model's predictions is because our maximum level of caricature was simply not large enough. In other words, caricaturing the images more would have produced larger differences between the three different-expression norm caricatures of each emotion category (anger, fear and sadness), so increasing the likelihood that they would be pushed into adjacent regions of space occupied by

132

A.J. Calder et al. / Cognition 76 (2000) 105±146

other emotions. Given the marked linear relationship between level of caricature and intensity ratings for the different-expression norm images, this explanation seems unlikely. That is, if this interpretation was correct, then we might at least have expected the intensity ratings to level off slightly at the higher levels of caricature, but this was not found. Nevertheless, to maximize the chances of con®rming the predictions of the two-dimensional model, Experiment 4 investigated this hypothesis. Experiment 4 also addressed a second issue. An assumption implicit in Experiment 3 is that different-expression norm caricatures of the same emotion (e.g. fear caricatured relative to sadness, and fear caricatured relative to surprise) become less similar with increasing levels of caricature. We also felt that it was important to verify empirically that the similarity of these images is decreasing with level of caricature (i.e. to demonstrate that the caricature procedure is doing what we say it is doing), so we asked a second group of participants in Experiment 4 to rate the physical similarity of pairs of different-expression norm images caricatured at 130 and 175%. 5. Experiment 4 Experiment 4 was in two sections. In the ®rst section, one group of participants rated the similarity between pairs of different-expression norm caricatures. In the second section, a different group of participants were shown 175% caricatures, and asked to rate them on each of ®ve emotional intensity scales (anger, disgust, fear, sadness, and surprise). Our aim was to determine whether the increased differences between 175% different-expression caricatures would result in their being perceived as different emotions, as the two-dimensional account predicts. 5.1. Method 5.1.1. Participants Twenty-four members of the MRC Cognition and Brain Sciences Unit Subject Panel participated in the experiment for payment. Half of the participants (three male, nine female; ages 17±36 years) were assigned to the similarity ratings task, and the other half (three male, nine female; ages 17±40 years) to the emotional intensity ratings task. All had normal or corrected-to-normal vision, and none had taken part in Experiments 1±3. 5.1.2. Materials From the three different-expression norms used to caricature each target emotion in Experiment 3, we selected the two norms that are most often confused with the target for use in Experiment 4; for example, for anger, the disgust and fear norms were selected. Similarly, two different-expression norm caricatures for fear and sadness were selected using the same criteria. This selection method was used because norms that are least likely to be confused with the target are also more likely to be positioned on the opposite side of a two-dimensional model to the target. And according to the two-dimensional model, caricaturing relative to these `least

A.J. Calder et al. / Cognition 76 (2000) 105±146

133

confusable' norms could, in some cases, be almost identical to caricaturing relative to a neutral-expression norm. Clearly, this would not always be the case and some `least confusable' norms may also shift the expression into an adjacent emotion space. Nonetheless, it is important to remember that two-dimensional models provide us with relative locations (i.e. fear is between surprise and anger) rather than absolute co-ordinates. Hence, it would be dif®cult to interpret whether the results of caricaturing relative to facial expressions on the opposite side of space concurred with the model or not. Using confusable norms, then, provides a stronger test of the models' predictions. In addition, it is worth remembering that confusable and non-confusable norms produced statistically indistinguishable patterns of data in Experiment 3. The stimulus list, then, was as follows (for each, the emotion shown in the square brackets refers to the facial expression used as the norm, i.e. anger[disgust] is anger caricatured relative to a disgust expression norm): anger[disgust] and anger[fear]; fear[surprise] and fear[sadness]; sadness[disgust] and sadness[fear]. Each differentexpression norm image was caricatured at two levels of exaggeration (130 and 175%). The images are shown in Fig. 10. 5.1.3. Design and procedure 5.1.3.1. Similarity ratings Participants were presented with pairs of stimuli and asked to rate how similar each pair was on a scale from 1 (not similar) to 7 (very similar). Each stimulus pair always consisted of two, non-identical caricatures of the same emotion (i.e. two anger, two fear, or two sadness caricatures). For each emotion, their were four possible pairings: (1) two 130% caricatures of the same expression caricatured relative to different facial expression norms (norm A and norm B); (2) two 175% caricatures of the same expression caricatured relative to different facial expression norms (norm A and norm B); (3) an expression caricatured at 130 and 175% relative to facial expression norm A; and (4) an expression caricatured at 130 and 175% relative to facial expression norm B. Note, the 130/ 175% pairs were used as a point of comparison for the 130/130% and 175/175% pairs. The following list of pairs for the anger caricatures illustrates the format for the stimulus pairs: 130% anger[disgust]/130% anger[fear], 175% anger[disgust]/ 175% anger[fear], 130% anger[disgust]/175% anger[disgust] and 130% anger[fear]/175% anger[fear]. There were 12 pairs of stimuli in all (three emotions £ four pairings). These were presented in random order on a computer screen, and remained in view until the participant responded. Participants were asked to respond quickly, basing their answer on their ®rst impression. No response feedback was given. Each pair was presented once in each of three consecutive blocks, and the ®rst block was discounted as practice. 5.1.3.2. Emotional intensity ratings. The six 175% caricatures used in Part A were presented individually on a computer screen in each of ®ve separate blocks. In each block the participants rated these images on one of ®ve emotion scales ranging from

134

A.J. Calder et al. / Cognition 76 (2000) 105±146

A.J. Calder et al. / Cognition 76 (2000) 105±146

135

1 (e.g. not angry) to 7 (e.g. very angry). The scales used were `intensity of anger', `intensity of fear', `intensity of sadness', `intensity of disgust', and `intensity of surprise'. The images remained in view until a response was made, but participants were asked to respond promptly to the faces, basing their answers on their ®rst impression. No response feedback was given. Within each block each image was presented three times, once in each of three consecutive runs. In each run, the order was random. The ®rst run was discounted as practice. 5.2. Results 5.2.1. Similarity ratings For each emotion, participants' ratings of the two types of 130/175% pairs for each emotion were pooled. This gave ratings for three stimulus pairs (130/130, 175/175 and 130/175%) for each of the three emotions (anger, fear and sadness). Mean ratings with standard error bars are shown in Fig. 11. The ratings were submitted to a two-factor ANOVA. The factors analyzed were emotion (anger, fear and sadness; repeated measure) and caricature pair (130/130, 175/175 and 130/175%; repeated measure). The results showed a signi®cant effect of caricature pair (F…2; 22† ˆ 38:54, P , 0:0001). Post-hoc t-tests (P , 0:01) showed that, overall, the 130/130% pairs were rated as more similar than the 175/175% and 130/ 175% pairs, which did not reliably differ. Hence, the rated similarity of pairs of different-expression norm caricatures of the same emotion decreases with increasing degree of caricature. There was also a signi®cant interaction between emotion and caricature pair (F…4; 44† ˆ 4:76, P , 0:005). Post-hoc t-tests (P , 0:05) showed that for all three emotions the 130/130% pairs were rated as more similar than the 175/175%, however, the relationship between the 175/175% and 130/175% was different for each emotion (anger: ‰130= 1 75%Š , ‰175= 1 75%Š; fear: ‰130= 1 75%Š ˆ ‰175= 1 75%Š; sadness: ‰130= 1 75%Š . ‰175= 1 75%Š). 5.2.2. Emotional intensity ratings Participants' mean intensity ratings (with standard error bars) for the two 175% different-expression norm caricatures of each of anger, fear and sadness are shown in Fig. 12. Each image was rated for its intensity of the ®ve emotions shown on the xaxes. The data corresponding to each of the ®ve rating scales (`intensity of anger', `intensity of fear', `intensity of sadness', `intensity of disgust', and `intensity of surprise') were submitted to separate ANOVAs. Each analysis examined one repeated-measure factor of image type (anger[fear], anger[disgust], fear[surprise], fear[sadness], sadness[disgust], sadness[fear]; the emotions in square brackets indicate the expression used as the reference norm). All analyses except disgust showed Fig. 10. From Experiment 4 ± caricatures (130 and 175%) of anger (top), fear (middle) and sadness (bottom) each prepared relative to two different facial expression norms. The norm for each row is shown on the left: for anger the norms are disgust (®rst row) and fear (second row), for fear the norms are surprise (®rst row) and sadness (second row), and for sadness the norms are fear (®rst row) and disgust (second row).

136

A.J. Calder et al. / Cognition 76 (2000) 105±146

Fig. 11. From Experiment 4, participants' mean similarity ratings (with standard error bars) for pairs of caricatures of three prototype expressions (anger, fear and sadness). Each expression was prepared relative to two different facial expression norms (see legend for Fig. 10), and caricatured at two levels of exaggeration (130 and 175%). Participants rated pairs of caricatures of the same emotion. There were three possible pairings (130/130, 175/175 and 130/175%).

a signi®cant effect of image type (anger: F…5; 55† ˆ 51:09, P , 0:0001; fear: F…5; 55† ˆ 54:12, P , 0:0001; sadness: F…5; 55† ˆ 28:33, P , 0:0001; disgust: F , 1; surprise: F…5; 55† ˆ 23:14, P , 0:0001). Of most relevance, post-hoc ttests (P , 0:05) showed no difference between participants' ratings of the two 175% different-expression norm caricatures of each emotion in any of the ®ve analyses (i.e. anger[fear] ˆ anger[disgust], fear[surprise] ˆ fear[sadness], sadness[disgust] ˆ sadness[fear]; for each of the ®ve emotional intensity scales). This demonstrates that the emotions displayed in the two different-expression norm caricatures of the same emotion did not signi®cantly differ. Hence, caricaturing an expression relative to two different facial expression norms generates two emotionally indistinguishable caricatures, even when the exaggeration level used is 175%. In the analyses of the anger, fear and sadness ratings, maximal ratings were attributed to the anger, fear and sadness images, respectively. For the surprise ratings

A.J. Calder et al. / Cognition 76 (2000) 105±146

137

Fig. 12. From Experiment 4, participants' mean emotional intensity ratings (with standard error bars) for the 175% different-expression norm caricatures of three emotions (anger, top; fear, middle; and sadness, bottom). Each emotional expression was caricatured relative to two different facial expression norms.

analysis, the fear caricatures (fear[surprise], fear[sadness]) were attributed the highest ratings.

138

A.J. Calder et al. / Cognition 76 (2000) 105±146

5.3. Discussion The results of Experiment 4 demonstrate that two 175% different-expression norm caricatures of each emotion (anger, fear and sadness) were rated as less physically similar than the corresponding 130% caricatures of these images. The only way to represent this in the two-dimensional model discussed is as two sets of points (or vectors) moving in different directions. As such, this means that at least one set of different-expression norm caricatures should be moving away from the area of space occupied by the to-be-caricatured emotion, and therefore, into an adjacent region of space occupied by a different emotion. Consequently, the twodimensional model predicts that the two 175% different-expression norm caricatures of each same emotion (anger, fear and sadness) should produce different pro®les of ratings on the ®ve scales. This was not observed, so the results of Experiment 4 do not support a two-dimensional perceptual model for the representation of facial expressions. Instead, it seems that whilst an increased degree of caricaturing relative to different expression norms certainly increases the degree of difference between the resultant caricatures, they continue to be seen as higher intensity exemplars of the same (caricatured) emotion.

6. General discussion The results of these experiments can be summarized as follows. 1. Experiments 1±3 show a monotonic relationship between level of caricature and the participants' emotional intensity ratings; the higher the level of caricature, the higher the intensity rating. 2. Experiment 1 demonstrates that emotional intensity ratings are not positively related to how `face-like' or `natural' the participants felt the images looked. Consequently, although the 175% caricatures were rated as maximally intense, they were also rated as the least plausible looking faces. 3. Experiments 1 and 2 show that when participants are asked to rate caricatures of a number of different expressions (e.g. happiness, fear, sadness, etc.) for their intensity of one particular target emotion, then, for the main part, only the caricatures relating to the target emotion are rated as increasingly more intense. This demonstrates that participants can attend selectively to one emotion, and that their ratings re¯ect changes in this emotion rather than a more general factor such as emotional arousal, or level of distortion. 4. Experiments 3 and 4 showed that the caricature advantage operates irrespective of the reference norm used in the caricature procedure. This result is inconsistent with a perceptual account of facial expressions based on two continuous underlying dimensions. Our study had two aims. First, to determine the psychological basis of caricature effects with facial expressions. Second, to use the caricature procedure to investigate whether the perceptual (structural) representation of facial expressions can be

A.J. Calder et al. / Cognition 76 (2000) 105±146

139

accommodated within a two-dimensional model. In relation to the ®rst aim, our results clearly demonstrate that caricaturing an expression enhances its emotional intensity. Moreover, this observation is consistent with a recent independent study by Benson, Campbell, Harris, Frank and ToveÂe (1999) investigating the effects of caricaturing `composite' facial expressions (i.e. blends of exemplars of the same facial expression). Together, these ®ndings present a plausible psychological interpretation of previous work showing that caricaturing facilitates facial expression recognition (Calder et al., 1997), and produces higher levels of neural activity than veridical (undistorted) expressions (Morris et al., 1996; Phillips et al., 1997). In other words, a facial expression's perceived emotional intensity is related to the salience of the expression's characteristic features (e.g. how raised the eyebrows are, how upturned the corners of the mouth are, etc.). It is interesting to compare our results with rating studies of facial identity caricaturing. As we discussed in Section 1, a number of studies have shown that facial identities caricatured by a small amount (in the range 14±16%) are perceived as signi®cantly better likenesses of people than their original (0%) representations (Benson & Perrett, 1991a; Rhodes et al., 1987; Rhodes, Byatt, Tremewan & Kennedy, 1997). However, all of these studies have also found that caricaturing above this `optimum' level produces increasingly worse likenesses of the faces. For identity, then, there is a cost linked to caricaturing too much. In view of these results for facial identity caricatures, it is interesting that this does not seem to apply to expressions. Even when caricatured by as much as 175%, caricaturing of expressions elicited the maximal intensity ratings. This pattern is all the more interesting when we consider that the most caricatured expressions from Experiment 1 were also rated as the least `face-like' of the levels of exaggeration used (275%, 250%, 225%, 0%, 125%, 150% and 175%). A clear implication of this latter result is that our perceptual representation of facial expression is coded independently of our perceptual representation of what is `face-like'. 6.1. Two-dimensional models of facial expression recognition The second aim of this study was to use the caricature procedure to investigate the predictions of two-dimensional models of the perceptual representation of facial expressions. As we discussed in Section 1, facial expression recognition has generally been discussed in terms of two types of theoretical models. For one model, facial expressions are recognized by activating discrete category representations (Ekman, 1992); for the other, they are recognized by registering their values on two continuous underlying dimensions (Russell & Bullock, 1985; Schlosberg, 1952). In relation to the latter model, some authors have suggested that the structural representation of facial expressions can also be accommodated within a similar type of two-dimensional system (Yamada, 1993). This type of two-dimensional model predicts that caricaturing one expression (e.g. anger) relative to a different-expression norm (e.g. disgust) should not generate a more intense exemplar of the original expression, but rather a different emotional expression. However, Experiment 3 found no evidence to support this prediction. Moreover, Experiment 4 showed

140

A.J. Calder et al. / Cognition 76 (2000) 105±146

that our ®ndings cannot be dismissed for the reason that we did not use large enough levels of exaggeration, because exaggerating the different-expression norm caricatures by as much as 175% did not produce results consistent with the two-dimensional model either. As we discussed earlier, the caricature procedure we have used is designed to manipulate facial shape, although texture information may be affected indirectly (e.g. as a result of a smiling mouth, or staring eyes becoming bigger). More recent caricature procedures developed by Perrett and colleagues (Lee & Perrett, 1997) enable the user to caricature both texture and shape. It would be interesting, then, to discover whether our results would also follow if one were to use texture and shape caricatures. However, as we have already pointed out, Yamada and colleagues have shown that a Circumplex structure is obtained from a factor analysis of facial measurements (i.e. shape information). To this extent, we feel that the shape caricatures used in Experiments 3 and 4 address the predictions of the two-dimensional models. That said, it is possible that expression-relevant information may also be represented in the texture component of facial expressions and this issue is worthy of further investigation. For the present, however, we ®nd that the predictions of a twodimensional model (derived from facial shape) are not supported by caricaturing shape alone. 6.2. A multidimensional model of facial expression coding In relation to the two-dimensional and category-based models discussed, the results of Experiments 3 and 4 seem more consistent with the idea that facial expressions are coded as discrete category representations. This interpretation would follow because Ekman and his colleagues have shown that each of the facial expressions we have used are associated with distinct patterns of facial muscle positions (Ekman & Friesen, 1978). Hence, when one of these expressions (e.g. anger) is caricatured relative to another (e.g. fear), the majority of features in the to-be-caricatured expression will be enhanced because these features are not also present in the reference norm (fear in our example). That said, although we ®nd no support for a two-dimensional model, there are aspects of our data that suggest that we should not reject the idea of continuous dimensional coding outright. In particular, the present study has demonstrated that people are highly sensitive to changes in an expression's intensity. This must mean that the perceptual representation of facial expression codes not only the presence or absence of particular features (wide open eyes, wrinkled nose, etc.) but also the feature's saliency (how wide the eyes are open, how wrinkled the nose is, etc.). To this extent, the idea of continuous perceptual coding at some level does not seem wholly inappropriate. One possibility is that emotions are recognized from the interaction of constellations of features, with each individual feature being coded along a single continuous dimension. A second solution is that con®gurations of features that frequently occur together (e.g. raised eyebrows, wide-open eyes and wrinkled brow) are coded on the same dimension. In either case, however, our results suggest that the dimensionality of the underlying space would need to be greater than two, because the principal

A.J. Calder et al. / Cognition 76 (2000) 105±146

141

problem the two-dimensional model has in accounting for our data is its limited number of dimensions. This is perhaps best illustrated by the results of Experiment 4. In this experiment, two 175% different-expression norm caricatures of the same emotion were rated as less physically similar than their corresponding 130% caricatures. This means that the two 175% caricatures must have different positions in perceptual space. In the two-dimensional model outlined, it follows that there is a high probability that these two images will be located in regions corresponding to different emotional expressions. But contrary to this prediction, Experiment 4 showed that the two 175% images were attributed statistically indistinguishable pro®les of ratings on ®ve different emotion scales. It is worth considering, however, that in a perceptual space built on more than two dimensions, a larger region could be devoted to each emotion. This would mean that even at higher levels of caricature there would be scope for these different-expression norm caricatures to have different positions in space (re¯ecting our ®nding that they are physically distinguishable), but still remain within a region dedicated to the same emotion (re¯ecting our ®nding that they are emotionally equivalent). Hence, a suitable solution may be that facial expressions are coded in a multidimensional space. If we are willing to accept this idea, then it is worth remembering that two types of multidimensional architecture have been investigated in the facial identity literature: a norm-based system and an exemplar-based system. In the norm-based model, faces are coded in relation to a perceptual norm (or average) face, while in the exemplar-based system there is no norm, and faces are coded as values on the individual dimensions (Rhodes et al., 1987; Valentine, 1991, 1995). Our present results suggest that facial expressions are unlikely to be represented in a norm-based system, because this type of model predicts that the probability of ®nding a caricature advantage should be some function of the similarity between the reference norm and the perceptual (or psychological) norm; in other words, caricaturing should not work if the reference norm is a poor match for the perceptual norm. Given that we have found a similar caricature advantage with every norm we have tested, it seems reasonable to conclude that our results lend little support to the idea of norm-based coding of facial expressions. With all of these factors in mind, perhaps the most appropriate system for the perceptual coding of facial expressions is an exemplar-similarity model (Estes, 1994; Nosofsky, 1984, 1986) based on a continuous multidimensional space. Here, each dimension would code either local or global features, and each newly encountered facial expression would be categorized in terms of its overall similarity to all stored exemplars. For example, an angry expression would be categorized as `anger' because it is more similar to stored anger exemplars than stored happiness, sadness, disgust, exemplars, etc. Furthermore, the fact that certain facial expression con®gurations are encountered more than others would mean that clusters of particular categories would form. Instead of the space being evenly distributed, then, it would contain a number of high density regions corresponding to the more frequently-encountered (basic) emotions (happiness, sadness, anger, fear, disgust, surprise, etc.), with each of these regions being separated by areas of low exemplar density. This would mean that these high density regions would have a `special

142

A.J. Calder et al. / Cognition 76 (2000) 105±146

status', one that is not inconsistent with the existence of basic emotional categories. Smith and Scott (1997) discuss that this is one way in which some researchers have attempted to reconcile dimensional and category accounts of emotional experience. 6.3. Caricature effects and categorical perception It is important to remember that any model of the perceptual representation of facial expression would have to account for more than just caricature effects. Foremost amongst other ®ndings are the results of recent studies examining categorical perception of morphed facial expression continua (Calder, Young, Perrett, Etcoff & Rowland, 1996; Etcoff & Magee, 1992; Young, Rowland, Calder, Etcoff, Seth & Perrett, 1997). In one such study, Young et al. (1997) asked participants to categorize the images in morphed (interpolated) continua ranging between all possible pairwise combinations of seven facial expressions (happiness, sadness, fear, anger, disgust, surprise and neutral). For this sort of task the two-dimensional model makes two predictions: (i) that transitions between expressions should be continuous (i.e. not abrupt step functions), and (ii) that at least some transitions between expressions should pass through a central neutral region or a region corresponding to a third emotion, meaning that images at the centre of some continua should be identi®ed as an expression other than the two endpoint expressions. Neither prediction was con®rmed. Instead, all morphed images were consistently identi®ed as the expression categories at one or other ends of the relevant continuum, with a sharp category boundary at the middle. Additional experiments have also shown that pairs of morphs that straddle the category boundary are better discriminated than pairs of equal physical magnitude from either side of the boundary (Calder, Young, Perrett, Etcoff & Rowland, 1996; Etcoff & Magee, 1992; Young et al., 1997). Again this ®nding has been interpreted as inconsistent with the two-dimensional model. To be fair, the classic categorical perception effect (abrupt category boundary and improved discrimination for morph pairs lying across the boundary) only poses a serious problem for a dimensional account if the underlying dimensional space is truly continuous. If we accept that expressions are coded in the sorts of high density `special status' regions discussed above, then a dimensional account could account for these results. In other words, the model would effectively act as a category-based system, because each morph would `gravitate' towards the nearest perceptual category in the space. Lack of support for the second prediction investigated by Young et al. (1997) (that some facial expression continua should pass through a region corresponding to a third emotion) is more dif®cult to explain in a two-dimensional system. But for a multidimensional model this is no longer a problem, because increasing the number of dimensions substantially reduces the probability of an expression continuum passing through a region corresponding to a third emotion (or neutral). In other words, the Young et al. (1997) results are inconsistent with a model based on two, continuous dimensions rather than dimensional coding per se. Hence, both caricature effects and categorical perception can adequately be

A.J. Calder et al. / Cognition 76 (2000) 105±146

143

accounted for by a multidimensional model incorporating `special status' regions corresponding to frequently encountered facial expression patterns. In summary, we have demonstrated that caricaturing facial expressions of the basic emotions increases their intensity; this effect is evidenced in the form of a monotonic relationship (often expressed as a linear trend) between intensity ratings and level of caricature. These results demonstrate that emotional intensity is an intrinsic component of facial expressions, and any model of their perceptual representation must take this into account. We have also shown that a caricature effect is observed regardless of the expression displayed in the reference norm. This observation is inconsistent with perceptual models of facial expression based on two continuous dimensions. In view of these ®ndings and the results of previous studies, we have suggested that the most appropriate system for the perceptual coding of facial expressions is a multidimensional exemplar-based system. Finally, on a more practical note, it is worth pointing out that until now, few studies of facial expression recognition have included intensity as an experimental factor. Undoubtedly, this has something to do with the fact that it is dif®cult to manipulate the intensity of natural (or posed) facial expressions in a controlled manner. Our results indicate that computer-generated caricatures offer a good alternative to varying intensity in natural facial expressions; in addition, the caricature procedure has the added advantage of being able to generate super-expressions beyond the scope of the human face. The caricature procedure, then, offers an extremely useful tool with which to explore the perceptual encoding of emotional facial expressions. Acknowledgements We are grateful to Professor P. Ekman for giving us permission to use pictures from the Ekman and Friesen (1976) pictures of facial affect and JACFEE (Matsumoto & Ekman, 1988) series. We would also like to thank Gary Jobe and Brian Cox for their assistance in preparing ®gures, and Brendan Hayes, Daniel Gibbenson and James Moriarty for collecting pilot data for Experiments 2 and 3. Appendix A A.1. Experiment 1 Experimental faces ± identi®er in Matsumoto and Ekman (1988) series, and percentage recognition as this emotion in their USA norms: happiness, HF-1C04 (97%), SA-1C35 (98%), LK-1C04 (99%), TA-1C04 (97%); surprise, ST-1C15 (98%), JG-1C17 (94%), YF-1C04 (98%), MM-1C17 (92%); fear, HU-1C35 (89%), SC-4C19 (66%), AM-2C25 (77%), KB-1C36 (77%); sadness, GO-1C31 (92%), SW-1C20 (94%), RK-1C31 (95%), NH1-1C31 (95%); disgust, EK-1C22 (81%), BC-1C15 (90%), ES2-1C22 (82%), GM-1C14 (70%); anger, AF-1C30 (90%), ES1-2C17 (92%), AL-1C21 (82%), KG-1C21 (86%).

144

A.J. Calder et al. / Cognition 76 (2000) 105±146

A.2. Experiment 2 Experimental faces ± identi®er in Ekman and Friesen (1976) series, and percentage recognition as this emotion in their norms: happiness, 34 JJ-4-07 (100%); surprise, 39 JJ-4-13 (97%); fear, 37 JJ-5-13 (96%); sadness, 36 JJ-5-05 (93%); disgust, 40 JJ-3-20 (88%); anger, 38 JJ-3-12 (76%). A.3. Experiments 3 and 4 Experimental faces ± identi®er in Ekman and Friesen (1976) series, and percentage recognition as this emotion in their norms: anger, 38 JJ-3-12 (76%); fear, 37 JJ5-13 (96%); sadness, 36 JJ-5-05 (93%). A.4. Experiment 3 Participants' mean ratings of anger, fear and sadness caricatures prepared relative to neutral-expression, average-expression and three different-expression norms. Standard errors are shown in brackets.

Emotion

Anger

Fear

Sadness

Expression norm

Level of caricature (%) 0

1 15

1 30

1 50

Neutral Average Different Disgust Fear Happiness

4.36 (0.34) 3.75 (0.42)

4.67 (0.41) 4.64 (0.42)

5.53 (0.30) 4.69 (0.42)

6.42 (0.18) 5.39 (0.34)

3.75 (0.35) 3.75 (0.33) 4.08 (0.23)

4.42 (3.36) 3.75 (0.28) 4.75 (0.39)

5.17 (0.39) 5.00 (0.48) 4.92 (0.38)

6.25 (0.22) 6.50 (0.19) 6.08 (0.50)

Neutral Average Different Surprise Sadness Anger

4.00 (0.32) 3.83 (0.42)

4.72 (0.31) 4.03 (0.37)

5.47 (0.36) 5.17 (0.33)

6.33 (0.20) 5.53 (0.31)

3.50 (0.42) 3.42 (0.34) 3.25 (0.37)

4.75 (0.30) 4.17 (0.34) 4.42 (0.40)

4.92 (0.36) 5.75 (0.22) 5.50 (0.31)

6.17 (0.30) 6.25 (0.21) 6.08 (0.31)

Neutral Average Different Fear Disgust Happiness

4.08 (0.43) 3.83 (0.39)

4.08 (0.45) 4.11 (0.47)

4.75 (0.32) 4.69 (0.44)

5.22 (0.39) 5.22 (0.37)

3.42 (0.48) 2.92 (0.36) 3.75 (0.35)

4.25 (0.41) 3.92 (0.47) 3.42 (0.29)

4.75 (0.39) 4.92 (0.48) 3.92 (0.42)

5.83 (0.34) 4.92 (0.53) 5.25 (0.40)

A.J. Calder et al. / Cognition 76 (2000) 105±146

145

References Benson, P. J., & Perrett, D. I. (1991a). Perception and recognition of photographic quality facial caricatures: implications for the recognition of natural images. European Journal of Cognitive Psychology, 3, 105±135. Benson, P. J., & Perrett, D. I. (1991b). Synthesising continuous-tone caricatures. Image and Vision Computing, 9, 123±129. Benson, P. J., Campbell, R., Harris, T., Frank, M. G., & ToveÂe, M. J. (1999). Enhancing images of facial expressions. Perception and Psychophysics, 61 (2), 259±274. Burt, D. M., & Perrett, D. I. (1995). Perception of age in adult Caucasian male faces: computer graphic manipulation of shape and colour information. Proceedings of the Royal Society, Series B, 259, 137± 143. Calder, A. J., Young, A. W., Benson, P. J., & Perrett, D. I. (1996). Self priming from distinctive and caricatured faces. British Journal of Psychology, 87, 141±162. Calder, A. J., Young, A. W., Perrett, D. I., Etcoff, N. L., & Rowland, D. (1996). Categorical perception of morphed facial expressions. Visual Cognition, 3 (2), 81±117. Calder, A. J., Young, A. W., Rowland, D., & Perrett, D. I. (1997). Computer-enhanced emotion in facial expressions. Proceedings of the Royal Society London, 264, 919±925. Ekman, P. (1973). Cross-cultural studies of facial expression. In P. Ekman (Ed.), Darwin and facial expression (pp. 169±229). New York: Academic Press. Ekman, P. (1982). Emotion and the human face. Cambridge: Cambridge University Press. Ekman, P. (1992). Facial expressions of emotion: an old controversy and new ®ndings. Philosophical Transactions of the Royal Society, London, B335, 63±69. Ekman, P., & Friesen, W. V. (1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists Press. Ekman, P., & Friesen, W. V. (1978). The facial action coding system: a technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press. Estes, W. K. (1994). Classi®cation and cognition. Oxford: Oxford University Press. Etcoff, N. L., & Magee, J. J. (1992). Categorical perception of facial expressions. Cognition, 44, 227±240. Frijda, N. J. (1969). Recognition of emotion. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 4, pp. 167±223). New York: Academic Press. Kruskal, J. B. (1964). Nonmetric multidimensional scaling. Psychometrika, 29, 1±27. Lee, K. J., & Perrett, D. (1997). Presentation-time measures of the effects of manipulations in colour space on discrimination of famous faces. Perception, 26 (6), 733±752. Matsumoto, D., & Ekman, P. (1988). Japanese and Caucasian facial expressions of emotion (JACFEE) and neutral faces (JACNeuF). Morris, J. S., Frith, C. D., Perrett, D. I., Rowland, D., Young, A. W., Calder, A. J., & Dolan, R. J. (1996). A differential neural response in the human amygdala to fearful and happy facial expressions. Nature, 383, 812±815. Morris, J. S., Friston, K. J., Beuchal, C., Frith, C. D., Young, A. W., Calder, A. J., & Dolan, R. J. (1998). A neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain, 121, 47±57. Nosofsky, R. M. (1984). Choice similarity and the context theory of classi®cation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 104±114. Nosofsky, R. M. (1986). Attention, similarity and the identi®cation-classi®cation relationship. Journal of Experimental Psychology: General, 1, 39±57. Perrett, D. I., May, K. A., & Yoshikawa, S. (1994). Facial shape and judgements of female attractiveness. Nature, 368, 239±242. Phillips, M. L., Young, A. W., Senior, C., Brammer, M., Andrew, C., Calder, A. J., Bullmore, E. T., Perrett, D. I., Rowland, D., Williams, S. C. R., Gray, J. A., & David, A. S. (1997). A speci®c neural substrate for perceiving facial expression of disgust. Nature, 389, 495±498. Phillips, M. L., Young, A. W., Scott, S. K., Calder, A. J., Andrew, C., Giampietro, V., Williams, S. C. R.,

146

A.J. Calder et al. / Cognition 76 (2000) 105±146

Bullmore, E. T., Brammer, M., & Gray, J. A. (1998). Neural responses to facial and vocal expressions of fear and disgust. Proceedings of the Royal Society London, 265, 1809±1817. Rhodes, G., Brennan, S. E., & Carey, S. (1987). Identi®cation and ratings of caricatures: implications for mental representations of faces. Cognitive Psychology, 19, 473±497. Rhodes, G., Byatt, G., Tremewan, T., & Kennedy, A. (1997). Facial distinctiveness and the power of caricatures. Perception, 26 (2), 207±223. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161±1178. Russell, J. A., & Bullock, M. (1985). Multidimensional scaling of emotional facial expressions: similarity from preschoolers to adults. Journal of Personality and Social Psychology, 48, 1290±1298. Schlosberg, H. (1941). A scale for the judgement of facial expressions. Journal of Experimental Psychology, 29, 497±510. Schlosberg, H. (1952). The description of facial expressions in terms of two dimensions. Journal of Experimental Psychology, 44 (4), 229±237. Schlosberg, H. (1954). Three dimensions of emotion. Psychological Review 61, 81±88. Smith, C. A., & Scott, H. S. (1997). A componential approach to the meaning of facial expressions. In J. A. Russell, & J. M. Fernandez-Dols (Eds.), The psychology of facial expressions. Cambridge: Cambridge University Press. SPSS (1994). SPSS Professional Statistics (Version 6.1), Chicago, IL: SPSS. Valentine, T. (1991). A uni®ed account of the effects of distinctiveness, inversion, and race in face recognition. Quarterly Journal of Experimental Psychology, 43A, 161±204. Valentine, T. (1995). Cognitive and computational aspects of face recognition: explorations in face space. London: Routledge. Woodworth, R. S. (1938). Experimental psychology. New York: Henry Holt. Yamada, H. (1993). Visual information for categorizing facial expressions of emotions. Applied Cognitive Psychology, 7, 257±270. Yamada, H., & Shibui, S. (1998). The relationship between visual information and affective meaning from facial expressions of emotion. Perception, 27, 133. Yamada, H., Matsuda, T., Watari, C., & Suenaga, T. (1993). Dimensions of visual information for categorizing facial expressions of emotion. Japanese Psychological Research, 35 (4), 172±181. Young, A. W., Rowland, D., Calder, A. J., Etcoff, N. L., Seth, A., & Perrett, D. I. (1997). Megamixing facial expressions. Cognition, 63, 271±313.