Evaluating a Transition Personnel Preparation Program: Identifying Transition Competencies of Practitioners

Evaluating Transition Preparation

1

Running head: EVALUATING TRANSITION PERSONNEL PREPARATION Evaluating a Transition Personnel Preparation Program: Identifying Transition Competencies of Practitioners

Mary E. Morningstar Kyeong-Hwa Kim Gary M. Clark University of Kansas Final Manuscript as Accepted Final Published as: Morningstar, M. E., Kim, K., & Clark, G. M. (2008). Evaluating a transition personnel preparation program: Identifying transition competencies of practitioners. Teacher Education and Special Education, 31(1), 47-58.

Evaluating Transition Preparation

2

Abstract To date, research has not sufficiently determined whether personnel preparation programs produce secondary special educators who have the specialized transition competencies needed to impact student post-school outcomes. Therefore, the purpose of the two research phases of the study described in this article was to determine and evaluate transition competencies gained by secondary practitioners involved in a long-standing transition teacher education program. We were particularly interested in examining whether graduates of our programs reported increased levels of competencies upon completion of the program. These responses were compared to a pre-assessment measure to determine significant levels of change. Using both quantitative and qualitative methods, we have found the answer to be overwhelmingly positive. Over time, our evaluation methods have increasingly expanded in terms of sophistication, and therefore, this article also provides an illustration of an ongoing approach to evaluating personnel preparation that can be replicated outside of the transition content area.

Evaluating Transition Preparation

3

Evaluating a Transition Personnel Preparation Program: Identifying Transition Competencies of Practitioners The issues and challenges of providing coordinated transition services are complex and pressing, and it is not surprising that secondary special education teachers often feel unprepared to provide effective services. This lack of preparation could be due in large part to the fact that relatively few special education personnel preparation programs include even one course devoted to transition (Anderson, Kleinhammer-Tramill, Morningstar, Lehmann, Bassett, Kohler, Blalock & Wehmeyer, 2003). Furthermore, states often report that most transition training takes place on the job with often limited systems for planning, delivering, supporting, and evaluating training activities (Kochhar-Bryant, 2003; Storms, 1999; Storms & Sullivan, 2000). Given that schools rely heavily on special education teachers to implement and manage transition planning and services, it is disconcerting to note that teachers feel they do not have the training to do so (U.S. General Accounting Office [U.S. GAO], 2003). It is unfortunate that schools are counting on practitioners to perform tasks in areas in which they have not received adequate training. Indeed, in a recent national leadership summit on improving results for youth with disabilities, more than 250 agency leaders, policymakers, educators, parents and youth with disabilities identified professional development for transition as one of the highest priorities for states (National Center for Secondary Education and Transition [NCSET], 2004). Complex Role Demands of Secondary Special Educators The role of secondary special educators has shifted from one involving traditional schoolbased services to one requiring collaboration among all stakeholders during the transition process (deFur & Taymans, 1995; Morningstar, Kleinhammer-Trammil, & Lattin, 1999). Some speculate that practitioners fail to collaborate effectively during transition due to the misperceptions,

Evaluating Transition Preparation

4

negative attitudes, and lack of training for such roles (deFur, 1997; U.S. GAO, 2003; Knott & Asselin, 1999). Recent research has exemplified strategies and capacities used by educators and transition practitioners; therefore, we know it is possible for local practitioners to shift their roles and responsibilities to enhance transition outcomes (Noonan, 2004). Today’s secondary special educators must be prepared to assume culturally-competent services that promote challenging standards as well as opportunities to link academic learning to transition goals leading to successful post-school results. Unfortunately, secondary special educators feel poorly prepared to address the majority of transition needs of their students (deFur & Taymans, 1995; Prater, Sileo & Black, 2000; Wolfe, Boone, & Blanchett, 1998). Given these findings, it is disconcerting to note that teachers who perceive they have limited transition knowledge are less likely to implement transition activities with their students (Blanchett, 2001; Knott & Asselin, 1999). Limited Transition Preservice Training Given such changing professional roles, it would seem that higher education training would respond accordingly. Unfortunately, this is not always the case. In fact, findings from a national survey of special education personnel preparation programs revealed that fewer than 50% currently addressed any transition standards (Anderson et al., 2003). In addition, less than half of all programs surveyed (45%) indicated offering a stand-alone course devoted to transition. While 70% of the instructors reported infusing transition content within other existing courses, they indicated devoting less time to transition competencies when using this content delivery method. These findings are consistent with earlier research conclusions that embedding transition content does not allow for adequate emphasis or coverage of important transitionrelated content (Severson, Hoover, & Wheeler, 1994).

Evaluating Transition Preparation

5

While transition-specific content for personnel preparation has been validated over the past decade (Flexer & Baer, 2005; Knott & Asselin, 1999; Kohler, 1998; Langone, Langone, & McLaughlin, 1991; Severson, et al., 1994), we have not resolved how best to offer such a knowledge base. In a recent review, Kohler and Greene (2004) offered multiple approaches used by higher education faculty to address both how to teach transition content as well as where this content fits within their curricula. Similar models of transition personnel preparation have been identified and described by others as well, such as Morningstar and Clark’s (2003) four approaches for teaching transition-specific content: (a) transition master’s programs; (b) transition specialization programs (15 credit hours or fewer); (c) transition class or classes; and (d) transition content infused within existing courses In summary, given the needs associated with transition implementation and personnel preparation, research has not sufficiently determined whether existing programs produce secondary special educators who have the specialized transition competencies needed to impact student post-school outcomes. Therefore, one purpose of this article was to determine whether secondary practitioners involved in a transition personnel preparation program do indeed gain competence in transition education and services. The lack of research regarding this issue was a compelling reason for our endeavors. Both quantitative and qualitative methods were utilized across two research phases to answer the primary research question: Do graduates of a transition personnel preparation program report increased levels of competence at a level that is significantly higher than upon entering the program? A second purpose of the research was to articulate an evolving model for evaluating personnel preparation programs. While we were particularly interested in examining whether graduates of our programs reported increased levels of competencies related to transition

Evaluating Transition Preparation

6

education and services, the methods used by our program evaluation process can be easily adapted to other program areas. Over time, our evaluation efforts have increased in sophistication and, therefore, a related purpose is to provide the results of an ongoing evaluation process and to offer strategies for expanding the evaluation efforts of special education teacher education programs. The University of Kansas Transition Personnel Preparation Efforts The University of Kansas (KU) has supported transition personnel preparation well before the existence of federal initiatives or mandates. Indeed, the topic of secondary special education has been of critical interest to faculty at KU since the 1970s (Younie & Clark, 1969; Clark & Oliverson, 1973; Albright & Clark, 1977) and has continued into the present, focusing specifically on strategies for alleviating fundamental gaps in the infrastructure for transition personnel preparation (Clark, 1984; Clark, Kleinhammer-Tramill, & Morningstar, 1999; Clark, Morningstar & Kleinhammer-Tramill, 2003; Kleinhammer-Tramill et al., 2003; Morningstar & Clark, 2003). These early efforts have laid the foundation for the development and implementation of the Secondary/Transition Masters of Education at KU in 1994 and its subsequent configurations. The current content for the KU Secondary/Transition program is based upon exhaustive reviews of literature related to teacher education in special education, personnel preparation for serving adolescents and young adults with disabilities, interprofessional personnel preparation, and effective online instruction. The core competencies currently addressed by the Secondary/Transition Masters program were refined after reviewing Council for Exceptional Children (CEC) and National Council for Accreditation of Teacher Education (NCATE) standards for training special educators (CEC, 2003), the CEC standards for transition specialists

Evaluating Transition Preparation

7

(DCDT, 2000), as well as state standards for training special educators (Kansas State Department of Education [KSDE], 2003). Based on this effort, we developed and created teacher competencies for the secondary/transition program and a curriculum for training graduate level special educators. The conceptual framework for the KU Secondary/Transition program focuses on the array of options for secondary school students with disabilities, ranging from support within the general education curriculum to employment and independent living options. We have found the model to be particularly relevant for states where secondary special education teachers are particularly difficult to recruit and shortages exist, and little emphasis is placed on transition services within preservice education. Moreover, Kansas, along with many states, is moving toward a less categorical approach to personnel preparation. As a result of these factors, this teacher education program was conceived as a noncategorical program that would prepare special educators to provide the necessary services to support adolescents and young adults in secondary education, transition, post-secondary education, and community participation. The program has adhered to five broad topical domains: (a) knowledge of principles and basic concepts of transition education and services; (b) knowledge of models of transition education and services; (c) skills in using strategies for developing, organizing and implementing transition education and services; (d) knowledge and use of collaboration competencies; and (e) knowledge and skills to address systemic problems in transition services delivery (Morningstar & Clark, 2003). Over time, we have enhanced and refined the KU Secondary/Transition program and expanded our efforts with subsequent spin-off programs (see Figure 1). Our evaluation efforts have informed activities such as the addition of new courses, revising content, and how programs are offered. Data are reported below for two

Evaluating Transition Preparation phases of an overall evaluation program that have taken place over a 10 year time period. The intent of this effort is not only to answer the broader research question, but to demonstrate an approach of a personnel preparation evaluation process that has evolved over time. -------------------------------------------Insert Figure 1 -------------------------------------------Phase 1: Transition Master Program Evaluation: Reported Post-Program Student Competency Gains Method Beginning in 1998, as part of final federal grant reporting procedures, we collected survey data to identify the perceived level of transition competence from among program graduates. In addition, we implemented a qualitative data collection process consisting of focus group discussions with recent program graduates. Post-program survey procedures. A survey consisting of 72 program competencies was developed by identifying specific transition competencies across all core and elective transition coursework. As mentioned previously, program competencies were derived from recommended and effective transition practices, national professional standards (e.g., CEC; DCDT) and state teacher licensure standards. We convened a statewide consumer advisory group made up of faculty and staff from Kansas institutions of higher education, state department of education staff, family members, adult services providers, and consumer advocates to socially validate the identified competencies. The final KU Transition Competencies Survey comprised of three sections. Section one contained 10 items related to demographic information of program participants (e.g., gender,

8

Evaluating Transition Preparation

9

ethnicity, current professional role, current graduate degree program, etc). Section two consisted of open-ended questions addressing the impact and value of the program on current and future professional roles. Section three consisted of 72 competencies designed to measure respondent perceptions of level of preparedness using a 4 point Likert scale (i.e., 4= well prepared to 1=not at all prepared). Survey mailings to 44 program graduates were completed during the summer and fall of 1998 with a response rate of 37%. Qualitative research procedures. In addition to the mailed survey, we held two focus groups with 9 program graduates during the spring of 1999. Focus groups were held at two different central locations in order to draw program graduates from a wide geographic area. Both meetings were held in the early evening, with dinner being provided. Focus group questions were developed in order to elicit both general and specific information about the transition master’s program and as a follow up to the survey results (see Table 1). The interview protocol consisted of seven broad categorical questions and specific probes used to clarify responses, follow-up participants’ comments, and elicit more detail. Results Post-program survey results. We collected descriptive statistics from the post-program surveys returned by program graduates. Demographic data indicated that the majority of respondents were between the ages of 30-39 (39%) and 40-49 (31%), with almost all of the respondents being women (85%) and Caucasian (93%). None of the respondents had enrolled in formal transition coursework other than that offered at KU, while 76% indicated they had completed transition inservice training. Over half of the respondents (56%) were school professionals (e.g., teachers, administrators, or transition coordinators) with the remaining graduates identifying themselves as adult agency professionals. All of the respondents indicated

Evaluating Transition Preparation

10

that they directly applied content from the transition masters program in their current positions. Examples of how they used the coursework included: (a) collaborating and supporting families during transition; (b) evaluating and implementing transition services and activities; (c) helping students make their transitions to adult and postsecondary educational program; (d) developing a new employment program; and (e) assessing students needs and interests. The overall mean score for level of preparedness across the 72 competencies was 3.53 (out of a total of 4.0), indicating high levels of perceived competence among the program graduates. In addition, a majority of the respondents (70%) stated that the program prepared them for professional roles different from the ones they had when they entered the program, with 62% stating their roles had changed due to involvement in the program. Role changes were interpreted as taking a new job (i.e., transition coordinator/specialist) as well as deepening and expanding the professional position they currently held. Open-ended comments included: (a) being viewed as experts and resource persons for their district; (b) giving them more confidence to pursue their current positions; (c) moving into a state department of education position; (d) being more knowledgeable about services; and (e) developing better transition plans that tie to adult services after high school. Focus group results. The participants in the focus groups represented the range of students trained under the Secondary/Transition Masters Program. Five of the nine participants were secondary special education teachers and/or transition coordinators, while the remaining four represented adult agency staff (e.g., family disability advocacy organization, vocational rehabilitation, and MR/DD agencies). All of the transition professionals indicated that they moved into their current positions as a direct result of having completed the KU transition program. In terms of overall satisfaction with the program, participants described the importance

Evaluating Transition Preparation

11

of the program content linking school and adult agency issues, leading to “real world” perspectives. Former students provided numerous examples of the skills and competencies acquired while in the program. The recurrent themes that emerged from the discussions included: (a) the benefits of collaborating and networking with classmates with differing perspectives and roles (i.e., school and adult agency professionals), (b) the focus on family-centered models of services; (c) learning about person-centered planning methods; and (d) exposure to interagency and community services and resources. They also highlighted positive aspects of the program such as, the practical application of concepts, optimal class sizes with a good mix of people, acknowledgement of the students’ professional expertise, guest lecturers, and flexible and relevant class projects. In terms of program weaknesses, participants indicated that there was some redundancy across course content, with similar projects being required across classes. In addition, for some class projects (i.e., required within the interagency and community services class), teachers indicated they had limited access to the outside agency resources required and, thereby needed to devote extensive time to complete these activities. When asked to identify content missing from the program that would have been beneficial to their current positions, participants identified the need for conflict negotiation skills (both across agencies and also between special and general education teachers); as well as more information pertaining to transitions to postsecondary educational experiences. Others felt there was too much of an emphasis on students with more significant disabilities, and not enough information specific to students with learning disabilities. Summary of Phase 1: Overall, students indicated high levels of competence gained from the KU Secondary/Transition Program. The descriptive statistics from the survey as well as the

Evaluating Transition Preparation

12

qualitative comments appeared to support the contention that the program was effective. A significant limitation of this phase of our research was the lack of baseline data of student competencies prior to entering the transition program. Given this, we were unable to ascertain whether such high levels of reported competence were the result of our training efforts or due to external variables. In addition, because of the low number of respondents, it was not possible to calculate an internal reliability estimate of the Transition Competency Survey, thereby further limiting our results. Given these limitations, during the next phase of our program evaluation, we developed a more rigorous evaluation design. Phase 2: Pre-Posttest of Student Competencies Method For the second phase of our evaluation efforts, we were particularly interested in addressing the limitations of the post-program survey results. Therefore, we implemented a research design to determine whether secondary practitioners involved in a transition personnel preparation program reported significant changes in their transition competencies upon completion of the program. Beginning in 2000 we collected pre- and post-program data across the 4 years of the KU Interprofessional/Transition (KU I/T) Personnel Program. This program differed from previous personnel efforts in that we targeted both special educators as well as related services personnel enrolled in KU graduate programs in school social work, school psychology, speech and language pathology, occupational therapy, and guidance and counseling. In 2003, we included a qualitative method of individual interviews with recent program graduates. Pre-post program survey procedures. The original 72-item transition competency survey was adapted for the purposes of this research phase, primarily because the KU I/T Program did

Evaluating Transition Preparation

13

not require students to complete all of the transition core courses offered in the master’s degree emphasis in transition services. Graduate students enrolled in the KU I/T program and completed an 8-hour block of coursework (i.e., an introductory course in transition education and services, a course on interagency and community systems, and a 2-credit hour seminar on interprofessional collaboration and issues). Therefore, the Transition Competencies Survey was pared down to the 22 indicators that corresponded to the specific courses in the 8-hour block. Similar to the original competencies survey, the KU I//T Competencies Survey consisted of three sections, including demographic information, program impact information, and competency indicators. The first two sections had open-ended questions and the third section used a 4-point Likert scale (i.e., 4= well prepared to 1=not at all prepared) to measure participants’ perceptions of competencies. Internal reliability estimates were calculated using Cronbach’s alpha and resulted in a coefficient of r = .95. This score indicates high levels of reliability of the survey, given that a reliability coefficient of .80 or higher is considered as "acceptable" in most social science applications (Cohen, 1988). All students who were admitted to the KU I/T Personnel Program completed the transition competency survey on the first day of the first transition course as the pre-assessment measure, and again on the last day of the last transition course for the post-assessment. Over a period of 4 years (2000-2004), we collected 43 pretests and 30 posttests from among 53 students who completed all or most of the coursework. Therefore, the response rate for the pretest survey was 81% and for the posttest, 56%. Because almost half of our participants were graduate students in related service programs, we had less control over their programs of study. Therefore, from this particular subset of graduate students, we saw higher rates of program attrition than from the special education graduate students. We employed two measures to analyze the data

Evaluating Transition Preparation

14

an independent samples t-test in order to evaluate all 73 completed surveys, and a paired samples t-test of 20 students from whom we had matched pre- and post-test surveys. Qualitative interview procedures. To complement the quantitative data collection of related services providers, we were interested in better understanding the experiences and perceptions of transition master program graduates (Benitez, 2003). In-depth personal interviews were completed in 2003 with three program graduates to further expand upon our existing data sets. A contact list was compiled of 24 program graduates who met the following criteria: (a) graduates of the program one year out; (b) secondary level professionals; and (c) school-based educators, as opposed to adult service professionals. Contact was made through email and telephone contact. From the initial sample of 24, we received no response from 8 individuals, with 5 graduates agreeing to be interviewed and the remaining 11 choosing not to participate. Of the initial five, 2 withdrew due to time conflicts; therefore, 3 former masters students in transition completed the in-depth interviews. All 3 were working professionals, with two being classroom teachers and the third serving as a vocational/transition coordinator. The interview questions focused on eliciting the graduates’ perceptions of the skills and experiences they felt they gained from the program; experiences and skills that were either not offered or that they did not gain from the program; and suggestions for program improvement. The first series of questions were posed to be global in nature (e.g., “Tell me about your educational and professional background.”) and then focused more specifically on several transition competency areas. Results Data from the surveys were entered into SPSS 11.5 for Windows and analyzed. The analyses from the quantitative and qualitative data are summarized in the following section.

Evaluating Transition Preparation

15

Demographic data results. The demographic data for the 53 respondents in the program for whom we had collected data indicated that the majority of participants were women (87%) and Caucasian (74%). Over half of the participants (62%) were graduate students in special education (e.g., high incidence disabilities, transition, etc.) with the remaining participants enrolled in KU related services graduate programs (i.e., school social work, school psychology, speech and language pathology, and occupational therapy, etc.). In terms of current professional roles, over half (53%) of the of participants were educators; with most serving as secondary special education teachers or transition coordinators; a few teaching in general education (19%) and the remaining education personnel serving as administrators or paraprofessionals. Of the remaining students, nearly one-fourth (23%) were full-time graduate students, and the rest (11%) were serving as related services professionals (school counselors, school psychologists, speech and language pathologists, and social workers) or in other roles (13%). Fifty-one of the 53 respondents had never enrolled in formal transition coursework other than the classes offered at KU. Special education teachers indicted they are currently working with students with disabilities and felt the program would be beneficial to their jobs. For school-based related services professionals, it was clear that they were concerned about having the knowledge to work with adolescents with disabilities: “As a counselor, I need to be knowledgeable about post-high school plans and transition programs.” The second reason was the importance of improving student outcomes during transition. For example, one general education teacher stated: “I know that my students in the education classroom really need good transition services.” Another common reason related to overall professional growth and knowledge of students with

Evaluating Transition Preparation

16

disabilities and transition services. This seemed particularly prevalent among the related services providers. Perceptions of preparedness among program participants. To examine the degree to which the program participants changed their perceived competence, two t-tests were employed: an independent samples t-test in order to evaluate all 73 completed surveys, and a paired samples t-test of 20 students for whom we had matched pre- and post-test surveys. Using the total number of completed surveys (i.e., 43 pretests and 30 posttests), an independent samples t-test was conducted to evaluate the effect of the program on student competence. As shown in Table 2, the independent samples t-test was significant, t (71) = 11.734, p = .000. The effect size, using the standardized difference between groups (Cohen, 1988) as measured by d, was 2.79, which indicates a large effect. This is based on the assertion that any d score above .80 is considered to have a large effect (Cohen, 1988). These results indicated that the group who had received coursework in the program (i.e., posttest group) was different from the group who had not taken coursework in the program (i.e., the pretest group). That is, the posttest group (M=3.44, SD= .40) perceived significantly higher levels of competence in transition than the pre-test group (pretest) (M=2.24, SD= .45). For the 20 matched pre- and post-tests, a paired samples t-test was conducted to evaluate whether students performed better after taking the required transition coursework. As shown in Table 3, the result of the test indicated that the mean score on the posttest (M=3.46, SD=.44) was significantly higher than the mean score on the pretest (M=2.41, SD=.45), t (19) = -10.426, p=.000. Using Cohen’s d, the standardized effect size rating, was 2.33, indicating a large effect given the d score was well over .80 (Cohen, 1988).

Evaluating Transition Preparation

17

Qualitative interview results. The three participants in the in-depth qualitative interview were all female educators. However, they were involved in very different educational experiences, thus giving a broader perspective of the impact of the KU Transition Master’s Program. One was a teacher of junior high school students with significant disabilities, the second a resource room teacher in a high school, and the third was a vocational/transition coordinator for a rural school district. All three entered the transition master’s program because of their interest in furthering their expertise in transition, as well as for purposes of career advancement (i.e., all three were interested in obtaining a transition coordinator position). All of the respondents indicated the importance of learning IDEA’s transition regulations as well as focusing on adult outcomes. All of the respondents stated that knowing about IDEA helped them to advocate for their students as well as assist families in advocating for themselves. The respondents also identified the importance of the program in providing skills related to the writing better transition IEPs and focusing on advocacy and planning for postschool outcomes. A second theme centered on the importance of understanding interagency supports and related services. All of the respondents indicated how learning about interagency collaboration helped them to plan for transition. The respondents indicated greater capabilities and confidence in working in this area. The third strength identified by the interviewees was the capacity of the program to apply what they learned in class to their professional roles. Respondents offered comments about the limitations of the program and suggestions for improvement. The first issue focused on those participants who were not currently teaching, and therefore, had limited access to students with disabilities to complete projects. A second limitation of the program was identified as a limited focus on social skills for students with

Evaluating Transition Preparation

18

disabilities. In terms of recommendations to strengthen the program, the respondents all suggested lengthening the vocational training and employment course beyond a single course. Summary Over the past decade, the University of Kansas has been intensively involved in providing secondary special education and transition personnel preparation. Given our increased attention to transition personnel preparation, it was important to evaluate our program with a specific focus on improving not only practice, but also evaluation methods. Program Evaluations Summary Over the course of 10 years, the KU Transition Program has developed two main tracks for training: (a) a 30-hour transition master’s program offering five core transition courses plus elective courses, and (b) an abbreviated configuration of transition coursework for professionals. Through implementing these two successive transition training programs, and now a third track consisting of a fully online certificate program, the core transition courses have become increasingly responsive to program graduates’ feedback and the results of the competency evaluations. Our program content has infused organizational standards (CEC, 2003; DCDT, 2000) and recommended practices supported by research. In addition, the results from our ongoing research agenda has informed our course content and sequence, thereby ensuring high levels of effective training. Specifically, the Secondary/Transition Program graduates (1994-1998) perceived that they developed competence in knowledge and skills pertaining to transition services after finishing their coursework. The benefit of the program appeared to be the practical applications of course content and the linking of school and adult agency issues to real world perspectives and practices. However, these results were tempered by the limitations to the evaluation design.

Evaluating Transition Preparation

19

The KU I/T Personnel Program was supported from 2000 to 2004 and made an effort to cover a broad context of special education and school reform and to strengthen interdisciplinary collaboration. We were pleased to see an increase in diverse and underrepresented professionals enrolling in the program (7% in Phase 1 increasing to 26% in Phase 2). The Phase 2 research refined the original evaluation procedures and obtained more robust data results. Based on these results, we can say with greater confidence that students who completed the 8-hour block of transition coursework showed significantly increased perceived levels of competence. All of the respondents from both of the program evaluation phases reported that they directly applied content from the program in their current positions. Special education teachers and/or school-based related services providers also indicated that their roles have deepened, as well as professional positions have expanded. The results of our evaluation appear to mirror other researchers who are concerned with impact of transition teacher education (Flexer & Baer, 2005; Webb, Asselin, Bassett, Hutchinson & Wandry, 2005). Therefore, the results of our research can indeed inform not only the efforts of transition personnel preparation programs but also provide guidance to others involved in evaluating teacher education irrespective of the content area. Recommendations for Future Research The two research studies described in this article evaluated transition competencies gained by secondary practitioners involved in long-standing transition personnel preparation. We were particularly interested in examining whether graduates increased their competencies specific to transition education and services. Using both quantitative and qualitative methods, we have found the answer to this question was overwhelmingly positive. Over time, our evaluation

Evaluating Transition Preparation

20

methods have increasingly expanded in terms of sophistication and, therefore, this article also provides an illustration of an ongoing approach to evaluating personnel preparation. Researchers closely associated with transition teacher education and professional development have called for better and more effective methods for evaluating the acquisition of transition competencies among practitioners (Blalock, et al., 2003; Flexer, Baer, Simmons, & Shell, 1997; Kohler & Greene, 2004; Morningstar & Clark, 2003; Morningstar & KleinhammerTramill, 2005). Therefore, an objective of this research was to do just this; we were not only interested in how effective our training programs were, but hoped to articulate the evolutionary process of our efforts, thereby sharing our model so that others do not have to recreate the wheel. As evidenced by the phases of our evaluation methods, over time our efforts have become increasingly more comprehensive. However, there is clearly room for an expansion of these initial research efforts. Indeed, our current research design for evaluating the newest configuration of transition coursework (i.e., 14-block of online courses) includes several additional measures for evaluating our personnel preparation efforts. While we continue to collect pre- and post-program competency data using a repeated measures design, we are also collecting a variety of additional measures in an effort to further determine the impact of transition preparation on secondary special education teachers and indeed upon the students with disabilities whom they serve. This research design now includes performance-based assessment measures using student online portfolios; data collected by students regarding the frequency with which they implement research-based transition practices; outcome data collected on students with disabilities; and comprehensive self-assessments of district and individual teacher programs. In this way we

Evaluating Transition Preparation

21

anticipate a much richer picture of the effects of personnel preparation across differing levels of impact. While the results of this research point toward a better understanding the efficacy of personnel preparation efforts, it is not yet clear what really matters when it comes to preparing special education teachers for the complexities of transition education and services. In other words, we need to begin systematically to evaluate content and delivery methods of personnel preparation. Still to be answered are questions articulated by Morningstar & Clark (2003): 1. How effective are current delivery methods? 2. Are some approaches more effective than others? 3. What is the critical mass needed to make infused content effective? 4. What are the courses into which transition content can and should be infused? 5. How do we ensure consistency of content and implementation? 6. Does specific content instruction make a difference in terms of outcomes for students with disabilities and postschool indicators of success? Continuing to refine and expand the evaluation efforts across the field of transition teacher education is a necessary first step if we are to effectively answer these questions. In addition, the evaluation methods described in this article can certainly be replicated across program content area and therefore, is relevant to a broader group than just those who are involved in transition professional development. Certainly learning from the limitations of our efforts, as well as replicating what works will make the path that much easier for others involved in the difficult task of evaluating the effectiveness of personnel preparation.

Evaluating Transition Preparation

22

References Albright, L., & Clark, G.M. (Eds.). (1977). Preparing vocational and special education personnel to work with special needs students: A teacher education resource guide. Bureau of Educational Research, University of Illinois at Urbana-Champaign, Urbana, Illinois. Anderson, D., Kleinhammer-Tramill, P.J., Morningstar, M. E., Lehmann, J., Bassett, D., Kohler, P., Blalock, G., & Wehmeyer, M. (2003). What’s happening in personnel preparation in transition? A national survey. Career Development for Exceptional Individuals, 26(2), 145-160 Becker-Staab M., J., & Morningstar, M. E. (1995, July,). Report of survey of institutions of higher education in Kansas on preservice training in transition. Unpublished manuscript, University of Kansas. Benitez, D. (2003). An evaluative inquiry of the KU Interprofessional Transition Project: Teachers’ perceptions of transition competencies. Unpublished manuscript. University of Kansas: Lawrence, KS. Blanchett, W. J. (2001). Importance of teacher transition competencies as rated by special educators. Teacher Education and Special Education, 24(1), 3-12. Blalock, G., Kochhar-Bryant, C., Test, D., Kohler, P., White, W., Lehmann, J., et al. (2003). The need for comprehensive personnel preparation in transition and career development: A position statement of the division on career development and transition. Career Development for Exceptional Individuals, 26(2), 207-226. Clark, G. M., Kleinhammer-Tramill, P. J., & Morningstar, M. E. (1999). Interprofessional Transition High Incidence Personnel Preparation, USDOE-OSEP, Division of Personnel Preparation, High Incidence Disability. Lawrence, KS: University of Kansas.

Evaluating Transition Preparation

23

Clark, G. M., Morningstar, M.E., & Kleinhammer-Tramill, P J. (2003, October 24). Transition Personnel Preparation: Responding to Issues and Challenges. Roanoke, VA.: 2003 Division on Career Development and Transition International Conference,. Clark, G.M. (1984). Issues in teacher education for secondary special education: Time for hindsight and foresight. Teacher Education and Special Education, 7, 170-177. Clark, G.M., & Oliverson, B.S. (1973). Education of secondary personnel: Assumptions and preliminary data. Exceptional Children, 39, 541-546. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associate. Council for Exceptional Children. (2003). What every special educator must know: Ethics, standards, and guidelines for special educators (5th ed). Reston, VA: Author. Division of Career Development and Transition. (2000, March). Transition specialist competencies Fact sheet. Arlington, VA: Council for Exceptional Children. deFur, S. (1997). Collaboration as a prevention tool for youth with disabilities. Preventing School Failure, 41, 173-178. deFur, S., & Taymans, J. (1995). Competencies needed for transition specialists in vocational rehabilitation, vocational education, and special education. Exceptional Children, 62, 3851. Flexer, R.W., & Baer, R.M. (2005). Description and evaluation of a university-based transition endorsement program. Career Development for Exceptional Individuals, 28(2), 80-91. Flexer, R. W., Baer, R. M., Simmons, T. J., & Shell, D. (1997). Translating Research, Innovation and Policy: Interdisciplinary Transition Leadership Training. Career Development for Exceptional Individuals, 20(1), 55-67.

Evaluating Transition Preparation

24

Kansas State Department of Education. (2003). Certified personnel report State profile 20022003. Topeka, KS.: Kansas State Department of Education. Kleinhammer-Tramill, P. J., Geiger, W., & Morningstar, M. (2003). Policy contexts for transition personnel preparation: An analysis of transition-related credentials, standards, and course requirements in state certification and licensure policies. Career Development for Exceptional Individuals, 26(2), 185-206. Kleinhammer-Tramill, P. J., Baker, B. C., Tramill, J. L., & Fiore, T. A. (2003). The history and status of OSEP personnel preparation policy for transition. Career Development for Exceptional Individuals, 26(2), 131-143. Knott, L., & Asselin, S.B. (1999). Transition competencies: Perception of secondary special education teachers. Teacher Education and Special Education, 22, 55-65. Kochhar-Bryant, C. (2003). Building Transition Capacity Through Personnel Development: Analysis of 35 State Improvement Grants. CDEI, 26(2), 161-184. Kohler, P. (1998). Implementing a transition perspective of education: a comprehensiveapproach to planning and delivering secondary education and transition services. In F. R. J. Chadsey (Ed.), Beyond Transition from High School to Work School (pp. 179-205). Belmont, CA: Wadsworth Publishing Company. Kohler, P.D., & Greene, G. (2004). Strategies for integrating transition-related competencies into teacher education. Teacher Education and Special Education, 27 (2), 146-162. Langone, C.A., Langone, J., & McLaughlin, P. (1991). Evaluating the impact of a secondary transitional teacher preparation program. Teacher Education and Special Education, 14(2), 94-102.

Evaluating Transition Preparation

25

Morningstar, M. E., Clark, Gary M. (2003). The status of personnel preparation for transition education and services: What is the critical content? How can It be offered? Career Development for Exceptional Individuals, 26(2), 227-237. Morningstar, M.E., & Kleinhammer-Tramill, P.J. (2005). Professional development for transition personnel: Current issues and strategies for success. National Center on Secondary Education and Transition Information Brief, 4(3) 1-5. Morningstar, M. E., Kleinhammer-Tramill, P. J., & Lattin, D. L. (1999). Using Successful Models of student-centered transition planning and services for adolescents with Disabilities. Focus on Exceptional Children, 31(9), 2-20. National Center for Secondary Education and Transition. (2004). A National Leadership Summit on Improving Results for Youth: State Priorities and Needs for Assistance. Institute on Community Integration, University of Minnesota. Noonan, P.M (2004) Developing interventions to improve interagency collaboration in providing transition services for youth with disabilities. Unpublished doctoral dissertation, University of Kansas, Lawrence. Prater, M. A., Sileo, T. W., & Black, R. S. (2000). Preparing educators and related school personnel to work with at-risk students. Teacher Education and Special Education, 23(1), 51-64. Severson, S.J., Hoover, J. H., & Wheeler, J. J. (1994). Transition: An integrated model for the pre- and in-service training of special education teachers. Career Development for Exceptional Individuals, 17(2), 145-158. Storms, J. (1999). Summary of 1999 Funded State Improvement Grant Applications. Eugene, OR: Western Regional Resource Center.

Evaluating Transition Preparation

26

Storms, J., & Sullivan, L. (2000). Transition Requirements: A Guide for States, Districts, Schools, Universities, and Families. Eugene, OR: Western Regional Resource Center. U.S. General Accounting Office. (2003, July). Special education: Federal actions can assist states in improving postsecondary outcomes for youth. Webb, K., Asselin, S., Bassett, D., Hutchinson, S., & Wandry, D. (2005. October). Identifying barriers to transition according to teacher candidates: Preliminary results. Paper presented at the Division of Career Development and Transition International Conference, Albuquerque, NM. Wolfe, P. S., Boone, R. S., & Blanchett, W. J. (1998). Regular and special educator's perceptions of transition competencies. Career Development for Exceptional Individuals, 21(1), 87106. Younie, W.J., & Clark, G.M. (1969). Personnel training for cooperative secondary school programs for mentally retarded youth. Education and Traning of the Mentally Retarded, 4, 186-194.

Evaluating Transition Preparation

27

Table 1: Focus Group Questions 1. What is your current professional role? 2. Has your role changed since entering/completing the program? PROBES: - Did the program help you to change roles? - Have you been offered other roles since entering/completing the program? 3. How did you hear about the program? Why did you decide to enter? PROBES: - How important were stipends in your decision? - Would you have entered the program without a stipend? 4. What were the major skills/competencies you acquired from the program? (transition specific and general special education) PROBES: - Give some examples of how you have applied what you’ve learned in your current role. - Think about the course content from the core transition courses (e.g., interagency, assessment, employment, and adolescent development) and describe how this content applies to your current role. - What activities in each of the classes were particularly useful? 5. Are there skills/knowledge that you need that was not provided by the program? PROBES: - Think about the course content from the major transition courses (e.g., interagency, assessment, employment, and adolescent development) and describe how certain skills/knowledge were not included that would have been meaningful to your current role. 6. Are there things that were emphasized in the program that really are not very important given your current role? PROBES: - Give some examples of program material that was emphasized but is not particularly useful. - Should these topics be revised or removed? - What activities in each of the classes were particularly a waste of time? 7. What are the institutional barriers that challenge your ability to implement the best practices taught in the program?

Evaluating Transition Preparation

28

Table 2: Independent Samples Test Levene's Test for Equality of Variances

F

T E S T

Equal variances assumed

.002

t-test for Equality of Means

Sig.

t

.966

Equal variances not assumed

Sig. (2tailed)

df

Mean Difference

95% Confidence Interval of the Difference

Std. Error Difference

Lower

Upper

-11.734

71

.000

-1.2016

.10240

1.40576

-.99740

-11.992

66.941

.000

-1.2016

.10020

1.40159

1.00158

Table 3: Paired Samples Test

Mean

Paired Differences Std. Std. Error 95% Confidence Interval Deviation Mean of the Difference Lower

Pair 1

PREPOST TEST

-1.0526

.45151

.10096

-1.2639

t

Sig. (2tailed)

df

Upper -.8413

10.42 6

19

.000

Evaluating Transition Preparation

29

Figure 1: KU Secondary/Transition Programs & Initiative

KU Secondary/Transition Programs Personnel preparation

Transition Masters Program Special Educators

Inter-Professional Transition Program

Professional development Transition Coalition

National Secondary Transition Personnel Preparation Project

National Transition Outreach Project

Related Services professionals Secondary Special Education & Transition Professionals

Online KU Transition Certification Program Secondary Special Educators