Developmental differences in scientific discovery processes

BTI~FILE COPY Lfl

00 00 DEVELOPMENTAL DIFFERENCES IN SCIENTIFIC DISCOVERY PROCESSES

N1

ITechnical

Report ALP - 41 Kevin

& David

Dunbar

Klahr

of Psychology

Department

Carnegie Mellon University Pittsburgh,

PA.

15213

The Artificial Intelligence and Psychology Project "" "-IC

ELECTE MAR1 1a S

Departments of Computer Science and Psychology

-

Carnegie Mellon University

Learning Research and Development Center University of Pittsburgh

Approved for public release; distribution unlimited.

9O 03

12

068

C

DEVELOPMENTAL DIFFERENCES IN SCIENTIFIC DISCOVERY PROCESSES Technical Report AIP - 41 Kevin

Dunbar

&

David

Klahr

Department of Psychology Carnegie Mellon University Pittsburgh, PA. 15213

30 June 1988

This research was supported by the Computer Sciences Division, Office of Naval Research and DARPA under Contract Number N00014-86-K-0678, and Personnel and Training Research Programs, Psychological Sciences Division, ONR Contract Number N-0014-86-K0349. To appear in Klahr, D., and Kotovsky, K. impact of Herbert A. Simon. Reprnduction Government.

(Eds.) Complex

information processing:

in whole or in part is permitted for purposes Approved for public release; distribution unlimited.

of

the

United

The

States

Unclassif ied SiCURI 'Y CLASSIFICATION OF rHIS PAGE

REPORT DOCUMENTATION PAGE Ia. REPORT SECURITY CLASSIFICATION Unclassified

1b RESTRICTIVE MARKINGS

2a SECURITY CLASSIFICATION AUTHORITY

3 DISTRIBuT ON AVAILABILITY OF REPORT Approved

2b OECLASSISCATIONi DOWNGRADING SCHEDULE

for

public

release;

Distribution unlimited

4 PERFORMING ORGANIZATION REPORT NUMBER(S)

5 MONITORING ORGANIZATION REPORT NUMBER(S)

AlP - 41 6& NAME OF PERFORMING ORGANIZATION

Carnegie-Mellon University i

6b OFFICE SYMBOL (If applicable) _ _ _

6C. ADDRESS (Oty, State and ZIPCode) Department of Psychology

7a1 NAME OF MONITORING ORGANIZATION Computer Division Ofc Office of Sciences ofNvlRsah Naval Research 7b ADDRESS(Cry, State, and ZIPCode) 800 N. Quincy Street

Pittsburgh, Pennsylvania 15213 8a. NAME OF FUjNOINGSPONSORING ORGANIZATION

Arlington, Virginia 22217-5000

8b OFFICE SYMBOL (Ifapplicable)

9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER

Same as Monitoring Organizatio

N00014-86-K-0678

8c. ADDRESS (City, State, and ZIP Code)

10 SOURCE OF FUNDING NUMBERS p4000ub201 7-4-86 PROGRAM PROjECT TASK WORK UNIT ELEMENT NO NO NO ACCESSION NO

N/A

N/A

N/A

N/A

11 TITLE (Include Security Classification)

Developmental diffe

, '- --

:n scientific discovery processes

12 PERSONAL AUTHOR(S)

Dunbar, Kevin 13a TYPE OF REPORT

Technical

i3b

and Klahr, David TIME COVERED

FROM

86

q,

Septl5TO9lSeptl,

ATE OF REPORT

1988

June

Year,Month,Day)

S PAGE COUNT

30

39

11 SUPPLEMENTARY NOTATION

To appear in Klahr, D., & Kotovskv, K. (Eds., Complex information processing: The impact of Herbert A. Simon, Hillsdale, N.J: Frlbaum, 1988. COSATI CODES

17

FIELD

GROUP

18 SiBjECT TERMS (Continue on reverse if necessar, and identify by block number)

SUB-GROUP

scientific reasoning discovery processes'

cognitive developmen ,. , -. _'-

problem space search 19 ABSTRACT (Continue on reverse if necessary and identify by block number)

SEE REi-ESE SIDE

.,

20 DISTRIBUTION / AVAILABILITY OF ABSTRACT 0 UNCLASSIFIED/UNLIMITED M SAME AS RPT 22a NAME OF RESPONSIBLE INDIVIDUAL

Dr. Alan L. Meyrowitz DO FORM 1473,

B4 MAR

21 C

,

ABSTRACT SECURITY CLASSIFICATION

OT'C USERS 22b TELEPHONE (Include Area Code)

(202)

83 APR edition may be used until exhausted. All other editions are obsoletU

696-4302

122c.

OFFICE SYMBOL

N00014 SECURITY CLASSIFICATION OF THIS PAGE

Unclassified

L

ABSTRACT The purpose of the three studies reported here was to formulate a framework for understanding the development of scientific reasoning processes. Subjects were placed in a simulated scientific discovery context by first teaching them how to use an electronic device To do and then asking them to discover how a hitherto unencountered function worked. conduct knowledge. prior their on based this task. subjects had to formulate hypotheses experiments, and evaluate the results of their experiments,. In the first study, using 20 adult subjects. we identified two main strategies for generating new hypotheses. One strategy was to search memory and the other was to generalize from the results of previous experiments. In a second study. with 10 adults, we investigated how subjects search the space of hypotheses by instructing them to state all the hypotheses that they could think of prior to conducting any experiments. Following this phase, subjects were Subjects who could not think of the correct rule in then allowed to conduct experiments. the hypothesis generation phase discovered the correct rule only by generalizing from the In a third study. twenty-two 3rd to 6th results of experiments in the experimental phase. Only two of them grade children were given the same task as the adults in study 1. discovered the correct rule, but 14 of them asserted that they were certain that they had discovered it. At the level of subjects' global behavior on this task. there was little difference between the children and the adults. Both groups understood the nature of the task and realized that they could discover how the device works by making it behave, observing that behavior, and However, viewed at the level of overall success rates. generating a generalization about it. consequences of how this general orientation toward the in differences profound there were had a 95% success rate. while 90% of the children adults The discovery was implemented. of this performance difference. First, children sources main There were three failed. Second. the children did not abandon did. adults the than proposed different hypotheses a new frame, or use the results of for space their current frame and search the Hypothesis the children did not attempt to Third. experiment space search to induce a new frame. check whether their hypotheses were consistent with prior data.

N

.

These studies provided support for the view that scientific reasoning is a search in two By extending Simon and Lea's (1974) Generalized Rule Inducer, we problem spaces. present a general model of Scientific Discovery as Dual Search (SDDS) that shows how search in two problem spaces (an hypothesis space and an experiment space) shapes The model hypothesis generation, experimental design, and the evaluation of hypotheses. also shows how these processes interact with each other and suggests what their developmental course might "e

'O.

....

;

i,

t

c

I

!

Scientific Discovery

i

Table of Contents On the origins of discovery processes Developmental issues in Scientific Reasoning Development of experimental strategies Studying the discovery process: General procedure Study 1: Adults discovering a new function Procedure Protocol encoding Aggregate results Overall performance The hypothesis space The experiment space Strategic variation in scientific discovery: Theorists and Experimenters Experimenters: General strategy Theorists: General strategy Study 2: Hypothesis-space search and experimentation by Adults Method Results and Discussion Study 3: Scientific reasoning in Children Method Subjects. Procedure Results Partial hypotheses Exploring only one frame Search of the Experiment space Differences in search strategies Summary A Dual-Search Model of Scientific Discovery SDDS: Summary SEARCH HYPOTHESIS SPACE TEST HYPOTHESIS EVALUATE EVIDENCE GENERATE OUTCOME E-SPACE MOVE Memory requirements The multiple roles of experimentation in SDDS Discussion Different experimental strategies Testing hypotheses Generating new hypotheses Generating new frames Scientific reasoning skills: What develops? Conclusion Postscript: Acknowledgements to Herbert Simon References

2 3 5 5 6 7 7 8 8 9 9 11 11 11 12 12 13 13 14 14 14 14 15 15 16 16 17 17 17 18 18 18 18 19 19 19 21 21 21 22 22 23 24 24 25

2

Scientific Discovery

On the origins of discovery processes Questions about the origins of scientific reasoning have been posed by developmental psychologists many times throughout the last 60 years (e.g., KarmiloffSmith & Inhelder, 1974; Kuhn, Amsel & OLoughlin. 1987: Piaget, 1928: Vygotsky, 1934) The context of developmental questions about scientific reasoning can be expanded to include a number of broader questions -- both descriptive and normative -- about the nature of science, and scientific reasoning. Within psychology, one approach to these questions has been to consider science a form of problem solving (e.g.. Bartlett. 1958 Simon, 1977). The science-as-problem-solving view is stated most explicitly in Herbert Simon's characterization of scientific discovery as a form of search and in his elucidation For example, he has used the notion of many of the principles that guide this search. of search in a problem space to analyze what science is (Simon. 1977), how scientists reason (Langley, Zytkow, Simon, and Bradshaw, 1986: Kulkarni and Simon, 1988). and In this chapter. we follow a similar path. how scientists should reason (Simon. 1973). and apply the notion of search to the development of scientific reasoning strategies. A contrasting view treats scientific reasoning as a form of concept formation. In the paradigmatic investigation of science-as-concept-formation. subjects are given examples or instances of a concept and are then asked to discover what the concept is (e.g., The extensive body of literature accumulated using Bruner, Goodnow & Austin, 1956). this approach has revealed many differences between the reasoning processes used by However, other than simply asserting that adults and children when forming concepts. scientific reasoning is a type of concept formation, psychologists have not formally specified how the cognitive processes involved in concept formation tasks are similar to those involved in scientific reasoning. One way to specify this similarity is to build a model of the processes that are involved in both concept-formation tasks and problem solving, and one model which has proved useful in this respect is Simon and Lea's (1974) Generalized Rule Inducer (GRI). Simon and Lea have demonstrated how this single system encompases both concept learning and problem solving. Within the GRI, concept learning requires search in two problem spaces: a space of instances, and a space of rules. Instance selection requires search of an instance space. and rule generation requires search of a rule space. Simon and Lea's analysis also illustrates how information from each space guides search in the other. For example, information about previously generated rules may influence the generation of instances, and information about the classification of instances may determine the modification of rules. A number of theorists (e.g., Cohen & Feigenbaum, 1983: Kulkarni & Simon. 1988: Lenat, 1977) have argued that the dual space search idea at the core of GRI can be extended to the domain of scientific reasoning, which takes place in a space of hypotheses and experiments. Using this idea. we developed a task that enables us to observe subjects' search paths in both spaces (cf. Klahr & Dunbar, 1988). Specifically, we studied the behavior of subjects who were attempting to extend their knowledge about a moderately complex device by proposing hypotheses about how it worked and then trying to determine whether or not the device behaved in accordance with their hypotheses. In this chapter, we will use the task to investigate what components of the scientific reasoning process show a developmental course. Our goal is to understand how existing knowledge structures determine the initial hypotheses, experiments, and data analysis in a discovery task. Because we treat scientific reasoning as a search in two problem spaces. we will explore the issue of whether there are developmental differences

Scientific Oiscovery

3

in how the two spaces are searched, and how search in one space affects search in the other. Our subjects worked with a programmable, multi-functioned, computer-controlled robot We trained both adults and whose basic functions they had mastered previously. elementary-school children to the same criterion on basic knowledge in the domain This training before we asked them to extend that knowledge by experimentation. allowed us to analyze developmental differences among subjects who shared a rommon Our analysis will focus on their knowledge base with respect to the task domain. attempts to discover how a new function operates -- that is, to extend their understanding about the device -- without the benefit of any further instruction. In order to do this, our subjects had to formulate hypotheses and then design experiments to evaluate those hypotheses: the cycle ultimately terminated when they believed that they had discovered how to predict and control the behavior of the device. The chapter is organized as follows. First, we briefly review some of the relevant Following this, we describe literature on the development of scienitific reasoning skills. These adult subjects., using studies earlier two summarize our task in detail, and then In the third study, we studies provide a context for the developmental questions. describe the performace of 8 - 11 year old children on this task. On the basis of these three studies we propose a model for scientific reasoning, and then use it as a framework for understanding the development of scientific reasoning strategies.

Developmental issues in Scientific Reasoning Klahr We have reviewed research on scientific reasoning in adults elsewhere (cf. and Dunbar, 1988), and in this section we concentrate on developmental issues. Research on scientific reasoning has typically treated different aspects of the overall process in isolation. In the developmental literature this approach has tended toward a One position is that polarization of views about the ontogenesis of scientific thought. improvements in scientific reasoning abilities are a consequence of a knowledge base For example. Carey that grows as the child develops (e.g.. Keil. 1981: Carey, 1985). (1984) states that: the acquisition and reorganization of strictly domain-specific knowledge (e.g., of the physical. biological and social worlds) probably account for most of the I have argued that in cognitive differences between 3-year olds and adults. many cases developmental changes that have been taken to support formatlevel changes. or changes due to the acquisition of some tool that crosscuts domains, are in fact due to the acquisition of domain-specific knowledge. (Carey, 1984, p62) Under this extreme view, the actual processes that children use only appear to be qualitatively different from that of adults because children do not have the necessary knowledge to perform at adult levels.

fReported in Kiahr & Dunbar. 1988

4

Scientific Discovery

The other view, exemplified by the work of Piaget (1952). is that while there are obviously changes in the knowledge base as children grow older, they are not the primary source of the radical differences in the behavior of children and adults. Rather. children have qualitively different representations of the world and strategies for reasoning (eg., Inhelder and Piaget. 1958: Kuhn and Phelps. 1982) Research in this about it. tradition has used tasks in which the role of knowledge has been minimized and the With respect to the different developmental strategies are made transparent. development of scientific reasoning strategies. this latter view makes very specific claims Flavell (1977) has succinctly described the difference between the reasoning strategies of adults and children as follows: The formal-operational thinker inspects the problem data, hypothesizes that such and such a theory or explanation might be the correct one. deduces from it that so and so empirical phenomena ought logically to occur or not occur in reality, and then tests his theory by seeing if these predicted phenomena do in fact occur. .... If you think you have just heard a description of textbook Because of its heavy trade in scientific reasoning. you are absolutely right. hypotheses. it is also called hypotheses and logical deduction from hypothetico-deductive reasoning, and it contrasts sharply with the much more nontheoretical and nonspeculative empirico-inductive reasoning of concreteoperational thinkers. (Fiavell, 1977, pp 103 - 104) Taken literally, this claim would lead to the conclusion that most adult subjects have not acheived the formal-operational level, because it has been well-established that adults find it extremely difficult to design experiments that provide a logical test of their Indeed, even well-trained scientists often draw invalid hypothesis (e.g., Wason, 1968). conclusions from the results of their experiments (e.g., Greenwald, Pratkanis, Leippe. & Furthermore, the view of science as a hypothetico-deductive Baumgardner. 1986). process is not consistent with recent descriptions of how scientists really work (cf. Harre 1983: Kulkarni & Simon, 1988). Whether or not children's thinking is empirico-deductive is While there has been a considerable amount of research on an open question. to design experiments that test hypotheses. there has been little children's abilities research that allows children to generate experimental results and then form hypotheses on the basis of these results. Therefore, one of the aims of our work with children was to discover what strategies they use in a scientific reasoning task. and how these strategies differ from those used by adults. We believe that instead of framing the developmental question in terms of the dichotomy between a broadening of the knowledge base and a qualatitive change in reasoning skills, it is more fruitful to provide a detailed characterization of the proceses that are involved in scientific reasoning, and then to ask about the development of these The specific approach in this chapter is based on the dual-space search processes. idea introduced earlier, and our focus is on developmental differences in the search processes. By using the same task to investigate the types of hypotheses that subjects generate, and the types of experiments that they conduct, we avoid the problem of studying knowledge and strategies in isolation. This enables us to answer some more focussed questions about the development of scientific reasoning skills.

5

Scientific Discovery

Development of experimental strategies Many developmental investigators have looked at the ability to design informative experiments. One common approach is to allow children to design (or select) simple experiments that will reveal the cause of an event (cf Case. 1974: Inhelder & Piaget, For example. 1958: Kuhn & Phelps. 1982; Siegler & Liebert, 1975: Tschirgi. 1980), to isolate the attempting children old to 1 1-year 10studied Kuhn & Phelps (1982) critical ingredient in a mixture, They discovered that children's performance was severely impeded by "the power and persistence of invalid strategies". i.e., experimental designs that were invalid, insufficient, or inefficient. Subjects commonly behaved as if their goal Tschirgi was not to find the cause of an effect, but rather to generate the effect. (1980) found that this tendency to generate a particular effect depends on whether the effect under investigation represents a good or a bad outcome. When the result of an experiment is undesirable (i.e.. a bad outcome). subjects' tendency is to (correctly) vary However. only the hypothesized causal variable: in order to eliminate the bad outcome. for good outcomes, subjects tend to simultaneously vary everything but the hypothesized cause of the good outcome. Tschirgi found that adults were as likely to make this error as children. Recent work on children's experimentation strategies by Kuhn and her coresearchers (Kuhn, Amsel, & O'Loughlin. 1987) has shown some developmental changes By presenting a large number of possible causes in the ability to evaluate evidence. that might produce an effect and asking children to state what factor or combination of factors are the cause of the event. Kuhn et al. discovered that children are more prone to ignore evidence that is inconsistent with their theory and are satisfied even when they know that their theory only accounts for some of the data. Furthermore, when children are asked to think of what data would be needed to disprove their theory, they have great difficulty. Taken as a whole, these studies suggest that children -- and under some circumstances adults -- frequently fail to distinguish between the goal of understanding a phenomenon and making it occur. The approach to experimm ntation that we will take is one of discovering the strategies that subjects use to both design and evaluate the results of experiments. When experimentation is considered as a form of search it should be possible to delineate what types of cognitive processes govern the search of the experiment space and then specify the differences between adults and children with regard to these In the following sections we will describe the task and the type of processes. hypothesis and experiment spaces that the subjects work in. This will make explicit the types of processes in which we expect to see developmental differences.

Studying the discovery process:

General procedure

The device we use is a computer-controlled robot tank (called "BigTrak") that is programmed using a LOGO-like language. 2 It is a six-wheeled. battery-powered vehicle. The device is used by approximately 30 cm long. 20 cm wide and 15 cm high. pressing various command keys on the keypad on the top of the device, which is illustrated in Figure 1. BigTrak is programmed by first clearing the memory with the

2

This same device was first used Klahr. 1986)

n a study of "nstructionless

learning"

(Shrager.

1985: Shrager and

Scientific Discovery

6

CLR key and then ing a series of up to sixteen instructions, each consisting of a function key (the command) and a 1- or 2-digit number (the argument). When the GO key is pressed BigTrak then executes the program Insert Figure 1 about here

The effect of the argument depends on which command it follows For forward (1) and backward (1) motion, each unit corresponds to approximately one foot. For left (