This article was downloaded by: [David Trafimow] On: 29 December 2014, At: 11:59 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
Is Consistency a DomainGeneral Individual Differences Characteristic? a
b
David Trafimow & Stephen Rice a
New Mexico State University
b
Florida Institute of Technology Published online: 24 Dec 2015.
Click for updates To cite this article: David Trafimow & Stephen Rice (2015) Is Consistency a DomainGeneral Individual Differences Characteristic?, The Journal of General Psychology, 142:1, 1-22, DOI: 10.1080/00221309.2014.961999 To link to this article: http://dx.doi.org/10.1080/00221309.2014.961999
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages,
and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [David Trafimow] at 11:59 29 December 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
The Journal of General Psychology, 2015, 142(1), 1–22 C 2015 Taylor & Francis Group, LLC Copyright
Downloaded by [David Trafimow] at 11:59 29 December 2014
Is Consistency a Domain-General Individual Differences Characteristic? DAVID TRAFIMOW New Mexico State University STEPHEN RICE Florida Institute of Technology
ABSTRACT. We explored randomness in responding in two ways across six experiments. First, we predicted that people would differ from each other in randomness in a stable way when tested in the same domain across two sessions; people who responded more randomly in a particular domain in one session also should respond more randomly in a second session whereas people who responded less randomly in one session also should respond less randomly in a second session. Second, we predicted that there would be some domain general randomness; people’s randomness in one domain should predict their randomness in another domain. We used consistency coefficients across blocks of a session as an inverse measure of randomness and found (a) consistency coefficients correlated across sessions within the same domain and (b) consistency coefficients in one domain correlated with consistency coefficients in other domains. Keywords: consistency
consistency coefficient, generalizability, randomness, within-persons
IT IS OBVIOUS TO ANYONE WHO HAS OBSERVED THE HUMAN CONDITION that randomness can play an important role in human behavior. For example, Durschmied (2012) has documented numerous occasions when chance events dramatically altered the course of human history. Our focus is less dramatic; we wish to study randomness in the context of quite mundane behaviors, such as responding to questions on academic topics, making moral judgments, and others. On the one hand, if we are to consider randomness in responding to be an individual differences characteristic, it follows that there must be some stability. A person’s behavior that is very random at one time, but not at another time, renders it difficult to assign him or her to a particular level of randomness. Our central issue Address correspondence to Prof. David Trafimow, New Mexico State University, Psychology, MSC 3452, New Mexico State University, P.O. Box 30001, Las Cruces, NM 88003-8001, USA;
[email protected] (e-mail). 1
Downloaded by [David Trafimow] at 11:59 29 December 2014
2
The Journal of General Psychology
pertained to whether a person’s level of randomness might be stable or unstable in the same domain across time or test taking occasions, or stable or unstable across different domains. If people’s levels of randomness are somewhat stable within the same domain but not across domains, then the best that can be claimed is that individual differences in behavioral randomness are domain-specific. In contrast, to demonstrate that randomness is, at least in part, a domain-general individual differences characteristic, it is necessary to demonstrate that randomness in one domain is related to randomness in other domains. As we will explain later, we consider consistency to be an inverse measure of randomness. Therefore, the foregoing reasoning suggests two basic predictions to be tested. The stability prediction is that the consistency with which people will respond to items in a particular domain will relate positively to their own consistency on the same items presented subsequently. The generality prediction is that consistency in one domain will relate positively to consistency in another domain. Six studies test these predictions. Previous Research on Intraindividual Variability Often, when the topic of intraindividual variability comes up, it is in the context of other kinds of variability. A prominent example is the scheme proposed by Cattell (1952), who suggested that there are three dimensions on which variability could arise. These are persons, variables, and occasions, and so Cattell suggested that variability should be considered in the context of a three-dimensional space (persons x variables x occasions). However, although Cattell’s view provides a broad way to think about the issue, many researchers have explored intraindividual variability within quite specific contexts to address particular research issues. Nesselroade and Ram (2004) provided a concise summary of this literature that we quote below (with a few updated citations): In the last 40 years, investigators have studied intraindividual variability in a wide variety of domains, including affect, emotion, and mood (Larsen, 1987; Lebo & Nesselroade, 1978; Wessman & Ricks, 1966; Zevon & Tellegen, 1982), personality (Hooker, 1991; Hooker, Nesselroade, Nesselroade, & Lerner, 1987; Roberts & Nesselroade, 1986; Shoda, Mischel, & Wright, 1994); human abilities (Hampson, 1990; Horn, 1972); cognitive performance (Baltes & Nesselroade, 1979; Hultsch & MacDonald, 2004; Jensen, 2006; May, Hasher, & Stoltzfus, 1993; Nesselroade & Salthouse, 2004; Siegler, 1994; Sontag, 1971); work values (Schulenberg, 1988); and teacher performance (Hundleby & Gluppe, 1974). The findings from these studies indicate that there are coherent, systematic patterns of intraindividual variability for many different kinds of psychological attributes, characteristics, and behaviors, some of which are traditionally considered to be highly stable within the individual. (pp. 11–12)
We wish to underscore that Nesselroade and Ram consider intraindividual variability to be coherent, systematic, and highly stable within the individual.
Downloaded by [David Trafimow] at 11:59 29 December 2014
Trafimow & Rice
3
We agree with Nesselroade and Ram (2004); researchers can learn much by considering the factors that cause coherence and stability in intraindividual variability. However, we also believe that it is useful to consider consistency from the point of view of indicating reduced randomness in human behavior. Fekken and Holden (1991) provided an admirable example of this approach. Their participants twice completed items from the Personality Research Form (Jackson, 1984) that measures twenty needs defined by Murray (1938). Fekken and Holden found that within-persons consistency within-sessions significantly predicted within-persons consistency across sessions. In addition, they ruled out social desirability as a plausible alternative explanation for their findings. Although there is much yet to do, the work performed by Fekken and Holden provides an excellent starting point to consider within-persons consistency. To move towards the experiments to be presented, it is worthwhile to risk some redundancy to be clear about what the main dependent variable is going to be. Like Fekken and Holden (1991), our participants completed all tests at least twice within a session (i.e., two blocks of trials), and we computed within-participants correlation coefficients. We term these consistency coefficients, though they also can be thought of as within-participants test-retest reliability coefficients. A participant who responds randomly should have a consistency coefficient of zero whereas a participant who performs exactly the same way on the same trials across the two blocks of trials (zero randomness) should have a consistency coefficient of unity. If consistency within a domain is a stable individual differences characteristic, then consistency coefficients obtained in one session of the performance of a task should correlate with consistency coefficients obtained in another session of the performance of that task. Thus, between-participants correlations of consistency coefficients across sessions is the first dependent variable of interest in Experiment 1. In the experiments to be presented, we used dichotomous items where participants responded to each item with a judgment of “true” or “false,” thereby rendering four possibilities: (1) participant responded “true” for the item on both blocks, (2) participant responded “true” on block 1 and “false” on block 2, (3) participant responded “false” on block 1 and “true” on block 2, (4) participant responded “false” on both blocks. Based on these four possibilities, the consistency coefficient for each participant, across the items on the two blocks of items, was computed using Equation 1 below. consistency coefficient = √
ad − bc (a + b) (c + d) (a + c) (b + d)
(1)
If consistency coefficients obtained via two blocks of trials in one session correlate with consistency coefficients obtained via two blocks of trials in another session, that would support the hypothesis of cross-session stability.
Downloaded by [David Trafimow] at 11:59 29 December 2014
4
The Journal of General Psychology
There will also be a second, and more important, dependent variable in all of the experiments to be presented. Specifically, we are interested in betweenparticipants correlations of consistency coefficients across domains. For example, for consistency to be a domain-general individual differences characteristic, we would expect people with large consistency coefficients in one domain to have large consistency coefficients in another domain, whereas people with small consistency coefficients in one domain should have small consistency coefficients in other domains. If consistency coefficients in one domain are not correlated with consistency coefficients in another domain, or if the correlations are negative, that would be contradict our hypothesis that consistency is a domain-general individual differences characteristic. The subsequent section presents findings regarding the transfer of skills across task domains. This research favors task-specific skills and so some initial pessimism might be warranted concerning the domain-generality of consistency coefficients.
Task Performance and Skill Transfer Eccles and Feltovich (2008; also see Carr & Baumann, 1996; Chi, Feltovich, & Glaser, 1981; Mayer, 1998) distinguished between skills that are domain-general versus domain-specific. Domain-specific skills are more efficient than domaingeneral skills provided that the task being performed is a good representative of the domain (Chi, Feltovich, & Glaser, 1981). However, when the task being performed is in a different domain, domain-specific skills are notorious for their lack of transfer to the new domain (e.g., Eccles & Feltovich, 2008; Logan, 1988; Palmeri, 1997; Rickard, 1997). In contrast, domain-general skills, though able to be used in a less efficient way than domain-specific skills, have the advantage of being more transferrable to novel domains (e.g. Mayer, 1998; Carr & Baumann, 1996). Given the distinction between domain-specific and domain-general skills, the issue before us is whether the ability to be consistent is domain-specific or domain-general. As most skills tend to be domain-specific, the most likely bet would be that being consistent also is domain-specific, and so although there would be some stability in consistency coefficients across sessions as Fekken and Holden (1991) found, consistency coefficients in one domain would fail to predict consistency coefficients in other domains. And yet, Eccles and Feltovich (2008) have put together a list of skills that have been shown to have at least some generality across domains, including some kinds of meta-cognition skills (e.g., Ertmer & Newby, 1996), self-talk (e.g., Baddeley, Chincotta, & Adlam, 2001; Emerson & Miyake, 2003; Goschke, 2000), relaxation skills (Alexander, Robinson, Orme-Johnson, Schneider, & Walton, 1994; Alexander et al., 1993; Berger, 1994; Feuerstein, Labbe, & Kuczmierczyk, 1986), and others. Possibly,
Downloaded by [David Trafimow] at 11:59 29 December 2014
Trafimow & Rice
5
cognitive ability plays a role in determining the domain-specificity or domaingenerality of some skills (Hambrick et al., 2014). Although we admit that there is no a priori reason to believe that the domain-general skills that Eccles and Feltovich considered have particular relevance for consistency, their article is at least mildly suggestive; the existence of a few domain-general skills suggests the possibility that consistency might, at least in part, be a domain-general individual differences characteristic. Going outside the skills domain, a recent article by Hepler and colleagues (2013) provides preliminary evidence that attitudes can be considered to be a function of people, as well as of attitude objects. People can be generally positively or negatively disposed towards objects, and these general tendencies are predictive across objects and across test-taking occasions. Like Fekken and Holden (1991), Hepler and colleagues provided evidence that their effects cannot plausibly be attributed to typical nuisance variables such as social desirability. Of course, researchers often have considered attitudes to be trait-like (e.g., Ajzen, 1988), but Hepler and colleagues’ data go further by showing an effect across attitude domains. Like much of the other literature we reviewed, this attitude research does not have direct implications for consistency. However, the fact that attitudes are a function of people, as well as of objects, is certainly suggestive of the possibility that other constructs also might be a function of people. Might consistency be one of these? Hypotheses All of the foregoing can be summarized in two empirical hypotheses pertaining to consistency coefficients. Before proceeding to the hypotheses, however, we wish to remind the reader that a person’s consistency coefficient is the correlation between his or her responses on one block of trials with his or her responses on another block of trials. According to Stability Hypothesis, consistency coefficients in one session should correlate with consistency coefficients in another session. In short, there should be some stability in people’s consistency coefficients across sessions. The Generality Hypothesis concerns the relations of consistency coefficients across domains and addresses the issue of whether consistency is a domain-specific or domain-general individual differences characteristic. Although we have little doubt that, consistent with much literature, consistency coefficients have some domain-specificity, our goal was to investigate whether consistency coefficients also would exhibit some domain-generality. Thus, the Generality Hypothesis states that consistency coefficients across domains will correlate and that the correlations will be positive. We report below the results from six experiments that test these hypotheses. Each experiment was performed following data analyses on the previous one.
6
The Journal of General Psychology
Experiment 1 Method Participants
Downloaded by [David Trafimow] at 11:59 29 December 2014
Fifty-four (35 females) undergraduate students from New Mexico State University participated in the experiment for partial course credit in an introductory psychology course. The mean age was 19.41 (SD = 2.27). Materials and Procedure Participants first signed a consent form and then sat comfortably in front of a computer screen in a laboratory. We used SurveyMonkey to present participants with 50 true-false items that we created from four different academic knowledge topics (biology, geography, grammar, and history), for a total of 200 items. An example history item was, In the Second World War, Italy fought with Germany. Appendix A contains some additional example items. The correct answer was true on half of the items and false on the other half. Participants then repeated this over three more blocks of trials, for a total of four blocks of trials per subject area. The first two blocks composed Session 1 whereas the latter two blocks composed Session 2. By arranging the four blocks into two sessions, we were able to test the stability of consistency coefficients across two sessions. There were two variables of interest. These were the between participants correlations of consistency coefficients across the two sessions and the between participants correlations of consistency coefficients across academic knowledge topics. Results We performed two categories of analyses on consistency coefficients that were computed via Equation 1. First, we performed analyses to test whether Session 1 consistency coefficients could predict Session 2 consistency coefficients within each of the academic knowledge topics. Second, we performed analyses within just Session 1 to test whether consistency coefficients in one academic knowledge topic would predict consistency coefficients in other topics. Stability Across Sessions We obtained evidence that consistency coefficients are stable across sessions (see Table 1). The average cross-session correlation was .56 (95% CI: .34–.72) and indicates that consistency can be considered to be an individual differences characteristic that has stability across sessions.
Trafimow & Rice
7
TABLE 1. The Entries Are Correlations Between Consistency Coefficients in Experiment 1 Biology
Downloaded by [David Trafimow] at 11:59 29 December 2014
Biology Geography Grammar History
.46 (.22–.65)
Geography
Grammar
History
.39 (.14–.60) .42 (.17–.62)
.60 (.40–.75) .27 (.002–.50) .61 (.41–.76)
.42 (.17–.62) .40 (.15–.60) .42 (.17–.62) .75 (.60–.85)
Note. These are within domains and across sessions (diagonals) and across domains. 95% confidence intervals for each correlation are included in parentheses.
Relations of Consistency Coefficients Across Topics Is consistency a domain-specific individual differences characteristic only or does it have domain-general aspects too? To investigate this issue, we correlated consistency coefficients in each domain with consistency coefficients in each other domain. If these correlations tend to be near zero, this would be evidence for a lack of domain generality. In contrast, if these correlations tend to be positive, this would provide preliminary evidence that there is domain generality. In fact, consistency coefficients were positively correlated (see Table 1) and the average correlation was .42 (95% CI: .17–.62). Thus, we have preliminary support that consistency, at least in part, is a general individual differences characteristic. Auxiliary Findings It is possible that participants’ answers on the second block depended on whether the answer on the first block was true or false. However, Table 2 shows that the data did not support this for any of the domains. The percentages of replications in Session 1 were 60% (95% CI: .47–.73) and 61% (95% CI: .48–.74) for true and false biology answers, respectively; 54% (95% CI: .41–.68) and 55% (95% CI: .42–.68) for geography; 60% (95% CI: .47–.73) and 63% (95% CI: .50–.76) for grammar, and 59% (95% CI: .46–.72) and 59% (95% CI: .46–.72) for history. In Session 2 these were 61% (95% CI: .48–.74) and 59% (95% CI: .46–.72) for biology, 54% (95% CI: .41–.67) and 55% (95% CI: .42–.68) for geography, 64% (95% CI: .51–.77) and 63% (95% CI: .50–.76) for grammar, and 58% (95% CI: .45–.71) and 59% (95% CI: .46–.72) for history. Thus, participants were equally likely to replicate true and false answers, and this was so in both sessions. Perhaps participants were more likely to answer true or false and this bias influenced the findings. In fact, there was no evidence that participants were biased towards answering true or false. They answered true 51% (95% CI: .38–.64) of the
8
The Journal of General Psychology
TABLE 2. Auxiliary Findings Pertaining to True or False Answers, Including Mean Percentage of True Versus False Answers, Mean Percentage of True Versus False Answers That Were Correct, and Mean Percentage of Replications for True Versus False Answers from Block 1 to Block 2 Participant Answers True
Downloaded by [David Trafimow] at 11:59 29 December 2014
Mean
Experiment 1 Session 1 Biology Geography Grammar History Session 2 Biology Geography Grammar History Experiment 2 History Biology Astronomy Algebra Experiment 3 Astronomy Morality Experiment 4 Intelligence Talent Experiment 5 Cities Models Experiment 6 Cities Models
Participant Answers False
Mean
Mean
Mean
Mean%
% Correct
% Rep
Mean%
% Correct
% Rep
51 50 50 50
70 63 73 69
60 54 60 59
49 50 50 50
72 63 73 69
61 55 63 59
50 51 49 50
70 61 75 69
61 54 64 58
50 49 51 50
70 64 72 69
59 55 63 59
62 62 60 50
79 80 76 86
89 90 90 89
38 38 40 50
56 56 56 86
79 83 79 87
47 51
69 NA
79 73
53 55
63 NA
73 65
60 62
NA NA
66 65
57 59
NA NA
65 64
45 44
42 NA
68 74
55 63
54 NA
72 64
55 59
NA NA
65 61
53 50
NA NA
49 55
time in biology, and 50% of the time in the other three areas in Session 1 (95% CI: .37–.63). In Session 2, there were 50% true answers in biology (95% CI: .37–.63), 51% in geography (95% CI: .38–.64), 49% in grammar (95% CI: .36–.62), and 50% in history (95% CI: .37–.63).
Downloaded by [David Trafimow] at 11:59 29 December 2014
Trafimow & Rice
9
Finally, participants were about as likely to be correct when they answered true as when they answered false in both sessions. Consider first Session 1, where participants were correct on 70% (95% CI: .58–.82) and 72% (95% CI: .60–.84) of the biology trials where they answered true and false, respectively. These values were 63% (95% CI: .50–.76) and 63% (95% CI: .50–.76) for geography, 73% (95% CI: .61–.85) and 73% (95% CI: .61–.85) for grammar, and 69% (95% CI: .57–.81) and 69% (95% CI: .57–.81) for history. In Session 2, these values are 70% (95% CI: .58–.82) and 70% (95% CI: .58– .82) for biology, 61% (95% CI: .48–.74) and 64% (95% CI: .51–.77) for geography, 75% (95% CI: .63–.87) and 72% (95% CI: .60–.84) for grammar, and 69% (95% CI: .57–.81) and 69% (95% CI: .57–.81) for history.
Experiment 2 Because the support for the Stability Hypothesis was solid, particularly when considered in combination with the work by Fekken and Holden (1991), there was no point in performing additional tests. In contrast, the support for the Generality Hypothesis was less solid. One potential criticism is that the number of topics was insufficient. Consequently, Experiment 2 was similar to Experiment 1, except that (1) we used different participants, (2) we used two different topics (astronomy and algebra replaced grammar and geography), and (3) we used only a single two-block session and did not perform any more tests of the Stability Hypothesis.
Method Participants Thirty-four (12 females) from an online survey participated in the experiment. The mean age was 35.88 (SD = 13.76). Unlike Experiment 1, these participants were not undergraduate students from an introductory psychology class but rather were recruited using MTurk. There were no exclusion criteria other than the obvious ones such as if the participant gave the same answer for every item. The advantage is that we were able to include practically all of the participants but at the cost of a lack of control over their academic qualifications.
Materials and Procedure The procedure of Experiment 2 was similar to that of Experiment 1, but with the exceptions noted earlier. The topics were history, biology, astronomy and algebra.
10
The Journal of General Psychology
TABLE 3. The Entries Are Correlations Between Consistency Coefficients in Experiment 2
Downloaded by [David Trafimow] at 11:59 29 December 2014
History Biology Astronomy
Biology
Astronomy
Algebra
.60 (.33–.78)
.52 (.32–.73) .38 (.05–.64)
.42 (.10–.66) .36 (.03–.62) .33 (−.01–.60)
Note. These are within domains and across sessions (diagonals) and across domains. 95% confidence intervals for each correlation are included in parentheses.
Results The correlations are presented in Table 3. The average correlation was .42 (95% CI: .10–.66). Therefore, Experiment 2 extends the evidence obtained in Experiment 1 that consistency, at least in part, is a domain-general individual differences characteristic. There were auxiliary findings. In contrast to Experiment 1, where we found no evidence for any kind of bias, there was a slight amount of evidence for bias in Experiment 2 (see Table 2). First, consider replications when the block 1 answer was true or false, respectively. These percentages were 89% (95% CI: .78–.99) and 79% (95% CI: .65–.93) for history, 90% (95% CI: .80–1.0) and 83% (95% CI: .70–.96) for biology, 90% (95% CI: .80–1.0) and 79% (95% CI: .65–.93) for astronomy, and 89% (95% CI: .78–.99) and 87% (95% CI: .76–.98) for algebra. Put more generally, with the exception of algebra, there was a slightly greater likelihood for people to replicate their answer on block 2 if the block 1 answer was true than if it was false. Considering the smallness of these effects, the overlapping confidence intervals, and the reduced sample size relative to Experiment 1 where there was no evidence for bias whatsoever, these effects fail to provide a convincing case for bias. In addition, the likelihood for answering true or false, respectively, was 62% (95% CI: .46–.78) and 38% (95% CI: .24–.52) for history. These values were 62% (95% CI: .46–.78) and 38% (95% CI: .24–.52) for biology, 60% (95% CI: .44–.76) and 40% (95% CI: .24–.56) for astronomy, and 50% (95% CI: .33–.67) and 50% (95% CI: .33–.67) for algebra. Finally, the mean percent correct was as follows for true and false answers, respectively, in the four domains. These were 79% (95% CI: .65–.93) and 56% (95% CI: .39–.73) for history, 80% (95% CI: .67–.93) and 56% (95% CI: .39–.73) for biology, 76% (95% CI: .62–.90) and 56% (95% CI: .39–.73) for astronomy, and 86% (95% CI: .74–.98) and 86% (95% CI: .74–.98) for algebra.
Trafimow & Rice
11
In summary, the support for the Generality Hypothesis replicated across both Experiments 1 and 2.
Downloaded by [David Trafimow] at 11:59 29 December 2014
Experiment 3 Although the findings thus far support that consistency is a domain-general individual differences characteristic, perhaps this conclusion is limited by the fact that all of the different domains concerned academic knowledge topics. What if we had used an academic knowledge topic and something else that is not one, such as making moral judgments? Moral judgments are particularly interesting because there is no objectively correct answer (see Trafimow & Rice, 2008, for a discussion). Will consistency coefficients in an academic knowledge topic, where there are objectively correct answers, predict consistency coefficients in the morality domain where there are not? Experiment 3 tested this issue. Method Participants Thirty-four (14 females) from an online survey participated in the experiment. The mean age was 27.06 (SD = 9.70). Materials and Procedure Experiment 3 was similar to Experiment 2 except that there were only two topics: astronomy and morality. An example morality scenario is as follows, and participants made true-false judgments as usual: Imagine that you are driving by an elderly nursing home and notice that there are several elderly people crossing. You realize that the speed limit is substantially higher than it should be and puts the elderly in the area at risk. You are morally obligated to drive at a speed that is substantially lower than the speed limit. The variable of interest was the between-participants correlation of consistency coefficients across astronomy and morality. Results The consistency coefficients correlated between astronomy and morality (r = .81, 95% CI: .65–.90). The finding extends support that there is a domain-general aspect of the individual differences characteristic of consistency. We also report auxiliary findings as in Experiments 1 and 2 (see Table 2). Replications for true and false answers on the first block, respectively, were 79% (95% CI: .65–.93) and 73% (95% CI: .58–.88) for astronomy and these were 73% (95% CI: .58 –.88) and 65% (95% CI: .49–.81) for morality. The proportion of true
12
The Journal of General Psychology
answers was 47% (95% CI: .30–.64) for astronomy and 51% (95% CI: .34–.68) for morality. Experiment 4
Downloaded by [David Trafimow] at 11:59 29 December 2014
After the success of Experiment 3, it seemed natural to ask, “How far can we go?” In Experiment 4, we investigated participants’ judgments of the intelligence of people of various nationalities and talent of famous artists. Would consistency coefficients in such subjective and unrelated domains correlate? Method Participants Thirty-five (11 females) from an online survey participated in the experiment. The mean age was 33.66 (SD = 12.39). Materials and Procedure Experiment 4 was similar to Experiment 3 except that participants were asked questions about the average intelligence of a person from a particular nationality (e.g., The average Colombian is more intelligent than the average person) and about the talent of different artists (e.g., Rubens is more talented than the average artist). The variable of interest was the between-participants correlation of consistency coefficients across the intelligence and talent domains. Results We obtained a correlation between the consistencies of judgments of the intelligence of members of different cultures and talent of artists (r = .66, 95% CI: .42–.81). The finding further supports that consistency has a domain-general component. There also were auxiliary findings. When the answer on the first block was true or false, respectively, replications were 66% (95% CI: .50–.82) and 65% (95% CI: .49–.81) for intelligence and 65% (95% CI: .49–.81) and 64% (95% CI: .48–.80) for talent. There was a slight tendency to answer true for both intelligence (60%, 95% CI: .44–.76) and talent (62%, 95% CI: .46–.78). Experiment 5 Lest the reader harbor concerns that perhaps the domains used in Experiment 4 were related, after all, we tried two new ones in Experiment 5.
Trafimow & Rice
13
Method Participants Twenty-eight (12 females) from an online survey participated in the experiment. The mean age was 31.43 (SD = 13.00).
Downloaded by [David Trafimow] at 11:59 29 December 2014
Materials and Procedure Experiment 5 was similar to Experiment 4 except that participants were asked questions about the size of random cities (e.g., Dakar is a city that is in the top ten largest cities in its country) and about the beauty of different models (e.g., Heidi Klum is prettier than the average female model). The dependent variable of interest was the between-participants correlation of consistency coefficients across city size and beauty items. Results As in the other experiments, consistency coefficients across the two areas were correlated (r = .75, 95% CI: .52–.88). There also were auxiliary findings. When the answer on the first block was true or false, respectively, replications were 68% (95% CI: .51–.85) and 72% (95% CI: .55–.89) for cities and these were 74% (95% CI: .58–.90) and 64% (95% CI: .46–.82) for models. There was a slight tendency against answering true for both city size (45%, 95% CI: .27–.63) and beauty (44%, 95% CI: .26–.64). Experiment 6 Experiment 6 was designed to provide an extreme test of the Generality Hypothesis. Experiment 6 was similar to Experiment 5 except that we used city and model names that were not real! Method Participants Thirty (14 females) from an online survey participated in the experiment. The mean age was 30.97 (SD = 10.65). Materials and Procedure Experiment 6 was similar to Experiment 5 except that none of the city or model names were real.
14
The Journal of General Psychology
Results
Downloaded by [David Trafimow] at 11:59 29 December 2014
Amazingly, we replicated the significant result from Experiment 5, but with the fake stimuli in Experiment 6 (r = .81, 95% CI: .64–.91). There also were auxiliary findings. When the answer on the first block was true or false, respectively, replications were 65% (95% CI: .48–.82) and 49% (95% CI: .31–.67) for cities and 61% (95% CI: .44–.78) and 55% (95% CI: .37–.73) for models. There was a slight bias in favor of answering true for both intelligence (55%, 95% CI: .37–.73) and talent (59%, 95% CI: .41–.77). General Discussion Both the Stability and Generality Hypotheses received strong support. Regarding the Stability Hypothesis, consistency coefficients correlated across sessions in Experiment 1, for all topic areas, thereby demonstrating cross-session stability. To the extent that people responded randomly versus consistently in Session 1, they exhibited similar tendencies in Session 2. Based on the preliminary work performed by Fekken and Holden, these findings were expected. Of greater interest was the research performed to test the Generality Hypothesis, that people’s consistency is a domain-general individual differences characteristic. We commenced with tests of the hypothesis involving generalization across academic knowledge topics in Experiments 1 and 2. We performed tests that were increasingly different in Experiments 3-6, even to the extent of using completely imaginary stimuli in Experiment 6. Nevertheless, consistency coefficients continued to generalize across topics even when put to the extreme test in Experiment 6. We performed a meta-analysis to find the weighted average cross-domain correlation across the six experiments (see Rosenthal & Rosnow, 2008). It was .62 (95% CI: .530–.696). As this procedure gave much weight to Experiment 1 due to the greater number of participants in that experiment, and as the correlations tended to be relatively small in that experiment relative to the later experiments (see below), it is possible that this weighted average cross-domain correlation is an underestimate. It is interesting to look closely at the correlations across the experiments. In Experiment 1, the average stability of the consistency coefficients (across sessions) was .56 whereas the average correlation of consistency coefficients across topics was .42. Because the stability of the consistency coefficients can be considered to be a type of reliability, and it is been known since Spearman’s (1904) famous work that reliability sets an upper limit on the correlations that can be expected to be obtained, the data make sense. Also, in Experiment 2, where we only measured the correlation of consistency coefficients across topics, the average correlation again was .42. But consider this correlation in the following experiments, where we purposely used topics that we felt would be less familiar, or lack objectively correct answers, or be as unrelated to each other as possible: Experiment 3 (r = .81), Experiment 4 (r = .66), Experiment 5 (r = .75), and Experiment 6
Downloaded by [David Trafimow] at 11:59 29 December 2014
Trafimow & Rice
15
(r = .81). To test more formally whether the weighted average correlation was larger in Experiments 3-6 than in Experiments 1–2, we performed a meta-analysis (Rosenthal & Rosnow, 2008). The weighted average correlation was .42 (95% CI: .23–.58) for Experiments 1 and 2 whereas it was .76 (95% CI: .68–.83) for Experiments 3-6. Although the data supported both Hypotheses 1 and 2, the relative magnitudes of the correlations provide a surprise. Why should greater magnitudes occur when the topics are unfamiliar, or lack objectively correct answers, or are unrelated to each other? There is a puzzle within the puzzle. Consider again that stability across sessions can be considered to be a type of reliability. Although the results in Experiment 1 were sufficient to support the Stability Hypothesis, they were not particularly impressive as reliability coefficients tend to go. From a reliability standpoint, they were in the range of what Shrout and Lane (2012) would have termed “fair,” and it is unlikely that a researcher would strongly depend upon such an instrument to attempt to solve a real world problem. However, as we pointed out in the foregoing paragraph, reliability sets an upper limit on the correlations a researcher can expect to obtain. We did not measure reliability of consistency coefficients across sessions in the experiments where we obtained impressive correlations across topic areas (Experiments 3–6), but it is obvious that reliability must have been excellent to allow for their occurrence. So the puzzle within the puzzle is as follows: Why should reliability have increased when consistency was assessed via unfamiliar topics? As the most extreme case, consider Experiment 6, where the stimuli consisted of fake cities and fake models, and the correlation across topics was .81. The reliabilities, had we measured them, would have had to be in excess of that number. One possibility is that the MTurk sample (Experiments 2–6) had greater reliability than the sample obtained from the participant pool (Experiment 1). However, this possibility does not explain the fact that the data obtained in Experiment 2, using similar content domains as in Experiment 1 but with MTurk participants, resulted in findings more similar to those obtained in Experiment 1 than those obtained in Experiments 3–6. A second possibility requires more explanation. Until this point, we have referred to consistency as an “individual differences characteristic” because we wanted to avoid using the word “trait,” fearing that the latter word was connotatively too strong. However, at this point, perhaps it is appropriate to use the stronger word. If we take a trait perspective, and assume that people really do have the trait of being more consistent or less consistent, it provides a nice solution to the puzzles brought about by the differences in the magnitudes of the correlations across the experiments. To see why this might be so, consider that for tasks where the participants have a high degree of familiarity, it might very well be the case that their consistencies are influenced partly by where they stand on the trait, but also by idiosyncratic aspects of the task, what their abilities happens to be on that task, and the fact of practice during the first session. Consequently, the obtained
Downloaded by [David Trafimow] at 11:59 29 December 2014
16
The Journal of General Psychology
consistencies are caused only partly by participants’ standings on the trait but with contamination due to other influences. In contrast, when people are unfamiliar with the task, and particularly if ability or practice effects are rendered less meaningful by a lack of an objectively correct answer, there is less contamination, and so obtained consistency coefficients really do provide a purer assessment of the trait. Thus, by taking seriously that consistency is a general trait, we have a solution to the puzzle within the puzzle. In turn, assuming that consistency is a trait and that unfamiliar tasks provide a better measure of it than do familiar tasks, a solution also is implied for the greater correlations of consistency coefficients across topics that were obtained in Experiments 3-6 relative to Experiments 1 and 2. That is, the consistency measures were more reliable and valid, thereby leading to increased correlations across topic areas. On the other hand, an argument also could be made that the task was more familiar in Experiment 5 (real names used) than in Experiment 6 (fake names used) and so the latter correlation should exceed the former one but the confidence intervals for the two correlations overlap. Limitations and Future Research The most obvious limitation of the present set of studies is that we did not directly measure cross-session stability in Experiments 2–6. The correlations across topic areas, particularly in Experiments 3-6, were extremely high, and so we are quite confident that cross-session stability also would have been high in those experiments, had we obtained the requisite measures. Nevertheless, it would be desirable to obtain those measures, and not just as a matter of form. To see where we are going with this, consider that although we characterized the items in Experiments 3–6 as differing from the items in Experiments 1-2 on a familiarity or experience dimension, we do not actually know that this is so. It could be a matter of having objectively correct answers or not, how much the participants cared about the items, or other dimensions. So a direction future research could take would be to obtain both cross-session and cross-topics data in a set of experiments that differ systematically on dimensions that plausibly could be used to explain the present data. It would be quite neat to find a mathematical function that relates correlations among consistency coefficients, both across sessions and topics, to whatever dimension the researcher favors as an explanation for the present findings. A second limitation is that the present data tell us nothing about the relation between consistency and task performance. Although it stands to reason (and from the phenomenon known as statistical regression) that people who are more consistent probably perform better on most tasks, we did not actually test this. At this point, it is useful to consider research by Trafimow and Rice (2008, 2009). Their equations specify precisely the extent to which consistency influences task performance. However, for Trafimow and Rice, consistency has to be measured on
Downloaded by [David Trafimow] at 11:59 29 December 2014
Trafimow & Rice
17
the task at hand. Our issue, and a direction that we believe would be informative from both the present point of view and from the point of view of the Trafimow and Rice equations, would be to investigate the extent to which generalized consistency influences task performance. For example, suppose the consistency of participants is measured using the fake cities method that we employed in Experiment 6, but performance on a visual search task is the main dependent variable. Will people who are more consistent on the fake cities measure perform better on the visual search task? Trafimow and Rice proved that consistency sets an upper limit on actual performance when both the consistency and actual performance measures concern the task under investigation. But to our knowledge, nobody has tested whether generalized consistency influences performance on specific tasks. A third limitation concerns the fact that we used relatively small sample sizes in the experiments, at least compared to most research on individual differences, and so our experiments could be criticized as being under-powered. Possibly mitigating this concern, the effect sizes (correlations) were sufficiently large for statistical significance, and so it seems problematic to argue that there was insufficient power for statistical significance when we repeatedly obtained statistical significance. In addition, the findings were quite consistent across all of the studies, despite the different methodologies, and we feel that this makes a stronger case than if we just had one or two experiments with many more participants. Conclusion We commenced our investigations by speculating that the tendency for people to be more consistent versus more random is not strictly domain-specific but is, at least in part, domain-general. As a minimum requirement to make this claim plausible, we must at least show cross-session stability, as we did in Experiment 1. An additional requirement involves generalization across domains. In both Experiments 1 and 2, we showed that consistency coefficients obtained in the context of academic knowledge topics predicted consistency coefficients obtained in the context of other academic knowledge topics. However, the obtained correlations were only moderate in magnitude. Although these findings were sufficient to meet minimal criteria to support our claim, could the claim meet stronger tests, with unrelated domains that are unfamiliar, subjective, or that involve fake stimuli? Surprisingly, the claim not only passed these strong tests, but the correlation between the consistency coefficients across domains increased from moderate to strong. Therefore, we believe that the data support more than the speculation with which we commenced, that consistency is partly a domain-general ability. The data support that it is a full-fledged general trait. Given that consistency is a trait, what makes it special? Many traits have been proposed in the personality literature so is the present research merely adding
Downloaded by [David Trafimow] at 11:59 29 December 2014
18
The Journal of General Psychology
another item to the list? We believe that consistency, or lack thereof, impacts almost all of human behavior. Consider job performance, where inconsistency has financial consequences that become obvious when particular tasks are performed incorrectly. Or consider the medical field, where inconsistency can prevent healing or even cause injury or death. Similar observations can be made about romantic relationships or even getting along with colleagues. It is difficult to avoid the consequences of statistical regression; as consistency decreases so that randomness increases, it is a mathematical fact that performance will regress towards the chance level. And performance that regresses towards the chance level tends to be unsatisfactory. Suppose future research shows that consistent people do better in almost all aspects of life than do inconsistent people. We believe that this is likely given that it is a straightforward implication of the combination of the present support that consistency is a general trait and the way that statistical regression works. Research such as this can hardly be characterized as merely adding an item to a long list of traits. AUTHOR NOTES Dr. David Trafimow is a Distinguished Achievement Professor of psychology at New Mexico State University, a Fellow of the Association for Psychological Science, Executive Editor of The Journal of General Psychology, and also for Basic and Applied Social Psychology. He received his PhD in psychology from the University of Illinois at Urbana-Champaign in 1993. His current research interests include attribution, attitudes, cross-cultural research, ethics, morality, methodology, and potential performance theory. Dr. Stephen Rice is an Associate Professor of Human Factors and Chair of the Graduate Programs in the College of Aeronautics at the Florida Institute of Technology. He received his PhD from the University of Illinois in 2006. His research interests include aviation psychology, trust, automation, and aviation consumer perceptions.
REFERENCES Ajzen, I. (1988). Attitudes, personality, and behavior. Chicago, IL: Dorsey. Alexander, C. N., Robinson, P., Orme-Johnson, D. W., Schneider, R. H. C., & Walton, K. G. (1994). The effects of transcendental meditation compared to other methods of relaxation in reducing risk factors, morbidity, and mortality. Homeostasis, 35, 243–263. Alexander, C. N., Swanson, G. C., Rainforth, M., Carlisle, T. W., Todd, C. C., & Oates, R. M. (1993). Effects of the transcendental meditation program on stress reduction, health, and employee development: A prospective study in two occupational settings. Anxiety, Stress and Coping: An International Journal, 6, 245–262. Baddeley, A., Chincotta, D., & Adlam, A. (2001). Working memory and the control of action: Evidence from task switching. Journal of Experimental Psychology: General, 130, 641–657.
Downloaded by [David Trafimow] at 11:59 29 December 2014
Trafimow & Rice
19
Baltes, P. B., & Nesselroade, J. R. (1979). History and rationale of longitudinal research. In J. R. Nesselroade & P. B. Baltes (Eds.), Longitudinal research in the study of behavior and development (pp. 1–39). New York, NY: Academic Press. Berger, B. G. (1994). Coping with stress: The effectiveness of exercise and other techniques. Quest, 46, 100–119. Carr, C., & Bauman, N. J. (1996). Life skills for collegiate student-athletes. In E. F. Etzel, A. P. Ferrante, & J. W. Pinkney (Eds.), Counseling college student-athletes: Issues and interventions (2nd ed., pp. 281–308). Morgantown, WV: Fitness Information Technology. Cattell, R. B. (1952). The three basic factor-analytic research designs-their interrelations and derivatives. Psychological Bulletin, 49, 499–520. Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121–152. Durschmied, E. (2012). How chance and stupidity have changed human history: The hinge factor. New York, NY: MJF Books. Eccles, D. W., & Feltovich, P. J. (2008). Implications of Domain-General “Psychological Support Skills” for Transfer of Skill and Acquisition of Expertise. Performance Improvement Quarterly, 21(1), 43–60. Emerson, M. J., & Miyake, A. (2003). The role of inner speech in task switching: A dual task investigation. Journal of Memory and Language, 48, 148–168. Ertmer, P. G., & Newby, T. J. (1996). The expert learner: Strategic, self-regulated, and reflective. Instructional Science, 24, 1–24. Fekken, G. C., & Holden, R. R. (1991). The construct validity of person reliability. Personality and Individual Differences, 12, 69–77. Feuerstein, M., Labbe, E. E., & Kuczmierczyk, A. R. (1986). Health psychology: A psychology perspective. New York, NY: Plenum. Goschke, T. (2000). Intentional reconfiguration and involuntary persistence in task switching. In S. Monsell & J. Driver (Eds.), Control of cognitive processes: Attention and performance XVIII (pp. 331–355). Cambridge, MA: MIT Press. Hambrick, D. Z., Oswald, F. L., Altmann, E. M., Meinz, E. J., Gobet, F., & Campitelli, G. (2014). Deliberate practice: Is that all it takes to become an expert? Intelligence, 45, 34–45. Hampson, E. (1990). Variations in sex related cognitive abilities across the menstrual cycle. Brain and Cognition, 14, 26–43. Hepler, J., Clore, G., Eagly, A., Roese, N., Schnall, S., & West, S. (2013). Attitudes without objects: Evidence for a dispositional attitude, its measurement, and its consequences. Journal of Personality and Social Psychology, 104, 1060–1076. Hooker, K. A. (1991). Change and stability in self during the transition to retirement: An intraindividual study using P-technique factor analysis. International Journal of Behavioral Development, 14, 209–233. Hooker, K. A., Nesselroade, D. W., Nesselroade, J. R., & Lerner, R. M. (1987). The structure of intraindividual temperament in the context of mother-child dyads: P technique factor analysis of short term change. Developmental Psychology, 23, 332–346. Horn, J. L. (1972). State, trait, and change dimensions of intelligence. British Journal of Educational Psychology, 42, 159–185. Hultsch, D. F., & MacDonald, S.W. (2004). Intraindividual variability in performance as a theoretical window onto cognitive aging. In R. A. Dixon, L. G. Nilsson, & L. Blackman (Eds.), New frontiers in cognitive aging (pp. 65–88). New York, NY: Oxford University Press. Hundleby, J. D., & Gluppe, M. R. (1974). Dimensions of change in instructor presentations. Journal of Educational Research, 74, 133–138.
Downloaded by [David Trafimow] at 11:59 29 December 2014
20
The Journal of General Psychology
Jackson, D. N. (1984). Personality Research Form Manual (3rd ed.). Port Huron, MI: Research Psychologists Press. Jensen, A. R. (2006). Clocking the mind: Mental chronometry and individual differences. Amsterdam, The Netherlands: Elsevier. Larsen, R. J. (1987). The stability of mood variability: A spectral analytic approach to daily mood assessments. Journal of Personality and Social Psychology, 52, 1195–1204. Lebo, M. A., & Nesselroade, J. R. (1978). Intraindividual differences dimensions of mood change during pregnancy identified in five P-technique factor analyses. Journal of Research in Personality, 12, 205–224. Logan, G. D. (1988). Toward an instance theory of automatization. Psychological Review, 95, 492–527. May, C. P., Hasher, L., & Stoltzfus, E. R. (1993). Optimal time of day and the magnitude of age differences in memory. Psychological Science, 4, 326–330. Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problemsolving. Instructional Science, 26, 49–63. Murray, H. A. (1938). Explorations in personality. New York, NY: Oxford University Press. Nesselroade, J. R., & Ram, N. (2004). Studying intraindividual variability: What we have learned that will help us understand lives in context. Research in Human Development, 1, 9–29. Nesselroade, J. R., & Salthouse, T. A. (2004). Methodological and theoretical implications of intraindividual variability in perceptual-motor performance. The Journals of Gerontology: Series B: Psychological Sciences and Social Sciences, 59B, P49–P55. Palmeri, T. J. (1997). Exemplar similarity and the development of automaticity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 324–354. Rickard, T. C. (1997). Bending the power law: A CMPL theory of strategy shifts and the automatization of cognitive skill. Journal of Experimental Psychology: General, 126, 288–311. Roberts, M. L., & Nesselroade, J. R. (1986). Intraindividual variability in perceived locus of control in adults: P-technique factor analyses of short-term change. Journal of Research in Personality, 20, 529–545. Rosenthal, R., & Rosnow, R. L. (2008). Essentials of behavioral research: Methods and data analysis (3rd ed). New York, NY: McGraw Hill. Schulenberg, J. E. (1988). Factorial invariance of career indecision dimensions across junior high and high school males. Journal of Vocational Behavior, 33, 63–81. Shoda, Y., Mischel, W., & Wright, J. C. (1994). Intraindividual stability in the organization and patterning of behavior: Incorporating psychological situations into the idiographic analysis of behavior. Journal of Personality and Social Psychology, 67, 674–687. Shrout, P. E., & Lane, S. P. (2012). Reliability. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol 1: Foundations, planning, measures, and psychometrics (pp. 643–660). Washington, DC: American Psychological Association. Siegler, R. S. (1994). Cognitive variability: A key to understanding cognitive development. Current Directions in Psychological Science, 3, 1–5. Sontag, L. W. (1971). The history of longitudinal research: Implications for the future. Child Development, 42, 987–1002. Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15, 72–101. Trafimow, D., & Rice, S. (2008). Potential Performance Theory: A general theory of task performance applied to morality. Psychological Review, 115(2), 447–462.
Trafimow & Rice
21
Trafimow, D., & Rice, S. (2009). Potential Performance Theory (PPT): Describing a methodology for analyzing task performance. Behavior Research Methods, 41(2), 359–371. Wessman, A. E.,& Ricks, D. F. (1966). Mood and personality. NewYork, NY: Holt, Rinehart, and Winston. Zevon, M., & Tellegen, A. (1982). The structure of mood change: Idiographic/nomothetic analysis. Journal of Personality and Social Psychology, 43, 111–122.
Downloaded by [David Trafimow] at 11:59 29 December 2014
Original manuscript received March 26, 2014 Final version accepted September 2, 2014
Appendix A
Domains History Geography Biology Grammar Algebra Astronomy Morality
Intelligence
Talent Cities
Example Items Julius Caesar conquered Gaul in the First Century BC. Written history goes back about 160,000 years. Cameroon borders the Republic of the Congo. Mount Aconcagua is the highest mountain in the Americas. The human skeleton is made up of less than 100 bones. Like humans, whales breathe air. A preposition is usually followed by a “noun.” The shortest possible sentence contains a subject, a verb and an object. 5x + 8x = 40; therefore x = 3.7 5x + 10 = −20; therefore x = −6 Stars produce energy through fusion of oxygen molecules. There are two known moons that orbit Pluto. Imagine that someone just punched you in the face during an argument. It is morally acceptable for you hit that person back. Imagine that you are in a nightclub and accidentally bump someone. He flips you off. It is morally acceptable for you to flip him off in return. The average Canadian is more intelligent than the average person. The average Italian is more intelligent than the average person. Van Gogh is more talented than the average artist. Hemingway is more talented than the average artist. Guadalajara is a city that is in the top ten largest cities in its country. Jaipur is a city that is in the top ten largest cities in its country. (Continued on next page)
22
The Journal of General Psychology
(Continued) Domains Models
Downloaded by [David Trafimow] at 11:59 29 December 2014
Cities (Made Up)
Models (Made Up)
Example Items Bar Refaeli is prettier than the average female model. Christy Turlington is prettier than the average female model. Tantingiani is a city that is in the top ten largest cities in its country. Qaitoe is a city that is in the top ten largest cities in its country. Marcie Discher is prettier than the average female model. Saundra Holl is prettier than the average female model.
