Revealing effects of changes in middle school science teachers ’ practices

Grant agencies are requiring documentation that goes beyond anecdote. We work with inservice middle-school and high-school teachers in two high-needs urban school districts in Ohio. Teachers who attend the summer institutes received at least 133 hours of professional development, which leads to changes in classrooms. We report on several ways we measure changes in teacher practice, including self reports and staff observations. PACS: 01.40.-d, 01.40.J-, 01.40.gb, 01.40.jh


I. INTRODUCTION
The Inquiry Model for Professional Action and Contentrich Teaching (IMPACT) was conceived by Principal Investigator (PI) Aubrecht and Science Partner (SP) Schmitt in 2008 as a professional development (PD) initiative to reform the teaching practices of middle-and high school science teachers.[1] The program was first implemented in Marion City Schools, an urban school district located in the Rust Belt of north-central Ohio, classified as "high-needs" by the Ohio Department of Education (ODE).After documented successes in increased student standardized test results and gains in increased teacher content knowledge, the program was expanded to the South-Western City Schools (SWCS) in 2014 in an attempt to replicate IMPACT's successes in another ODEclassified "high needs" school system.
IMPACT's primary goal is to promote increased student achievement by providing students the opportunity to benefit from better-prepared teachers -both pedagogically and in terms of increased teacher content knowledge.IMPACT's triadic approach to PD involves three distinct components that, while implemented in unison, are distinct in their objectives: increased teacher-content knowledge support; the use of common formative assessments (CFAs) with students; and lesson development with an emphasis on research-based inquiry pedagogical approaches.[2] This paper does not further discuss increased teacher content knowledge or the use of CFAs in the classroom but rather focuses on reported and observable measures of teachers' pedagogical changes.
IMPACT's PD program for increased teacher pedagogical knowledge culminating with increased student achievement is a cyclic process that occurs throughout the calendar year.We estimate teachers attending all PD receive more than 133 hours of research-based pedagogical exposure annually: ~ 105 hours of Summer Institutes where participating teachers experience inquiry-based learning in the role of the student in which teachers are exposed to hands-on, minds-on approaches to science instruction that honor student thinking and reasoning; ~ 63 hours of grade level meetings held monthly during the academic year where teachers work with PI and SP to collaborate on creating comprehensive multi-week content units focusing on student-centered inquiry-based instruction where the primary role of the teacher resides as the facilitator of student learning; ~ 10 hours of inclass visits from IMPACT staff which serves (among other things) to help provide feedback to specific teachers' instructional approaches and allows the opportunity for staff to provide support for teachers implementing the inquiry-based content lessons they created into the classroom environment.The IMPACT team also uses this time to complete teacher evaluations as a measure of teacher growth.
In order to quantify changes revealed in IMPACTed teachers' pedagogical approaches in the classroom, we rely on various diagnostics completed by the teachers throughout the 2013-15 school years, including: a curriculum and instruction diagnostic (CID), self-reports using categories from the Reformed Teacher Observation Protocol (RTOP), and staff classroom observations.[3,4] This paper discusses the results of these diagnostics.

II. RESULTS FROM THE CID SURVEY
The Curriculum and Instruction Diagnostic (CID) is a self-reported measure in which teachers quantify classroom activities and characteristics, the extent to which they use specified standards or curricula, and also report on selfefficacy in teaching science.
The following analysis provides the difference between the pre and post testing within each section and also describes how it relates to the inquiry-based professional development.The diagnostic survey questions completed by the teachers and discussed here appear in the Appendix.

A.Instructional influences
The purpose of this portion of the CID is to help portray the classroom, school, and district demographics of treatment teachers.Teachers are asked to rate the effects of various instructional influences when planning for and implementing classroom instruction.Participating teachers report similar patterns in instructional influences at the beginning and end of the school year.In both fall and spring, textbooks influenced teaching the least, while district and state standards (as well as preparation of students for the next grade level) are top influencers.Interestingly, district and state test results draw the most negative influence on teaching in both cases.

B. Class instructional time
This section consists of six items, and respondents reported the amount of time their students spent in class on a given activity.They select None (0) through Considerable (50% or more time).For analysis, teachers are given a score of one through five corresponding to the task in question.Teachers are given high scores when the reported activity is more student-centered or inquiry-based than teachercentered, and low scores when opposite.With this scoring, the highest possible score for this section is 30, and the lowest possible score is six.When the pre and posttests are compared for all respondents, the average score is nearly identical between the groups.For those who took both the pre and the posttest, however, there is a statistically significant increase from fall to spring: they reported that they spent more time on student-centered or inquiry-based activities after the year spent in the IMPACT program.
Table 1 shows the paired t-test between pre and post for those who completed both tests.

C.Individual class time
This type of instruction, along with group instruction, was among the two larger changes in instruction reported.The maximum possible score is 20 in this section, and while no one reached this ceiling in either pre or post, data show a significant increase in scores from the beginning to the end of the school year.Teachers report directing more inquiry-based individual instruction time at the end as compared to the beginning of the school year.

Group instructional time
The change in group instructional time in class has one of the larger differences (along with individual class time) among the CID sections.There is a significant difference between the pretest and posttest averages.After spending a school year in the IMPACT program, teachers report holding more inquiry-based group instructional periods in their classrooms.

D.Self-efficacy
This portion of the diagnostic consists of sixteen items, and is the largest portion of the diagnostic.Given that there is a significant amount of focus on increasing teacher content knowledge through hands-on, minds-on instructional techniques in the program, it was expected that teacher self-efficacy would rise as the result of increased content knowledge and the teacher's ability to create inquiry-based content lessons for their students, but that did not occur.

III. RESULTS FROM MODIFIED SELF-REPORTED RTOP
RTOP observation and evaluations offer the opportunity for an objective observer to quantify the extent to which a teacher employs constructivist, inquiry-based pedagogical approaches in the classroom.[4] In the self-report RTOP used by IMPACT, teachers self-assess their teaching practices using a selection of questions taken directly from the RTOP categories (17 of 25 items).
We examined what teachers reported about their practices.The maximum possible for the self-reported RTOP score was 68.Selected questions from the modified self-report RTOP are found in the Appendix.Teachers seem to think they are "reformed" (Table 5; RTOP scores over 50% are considered reformed).[4] Well-taught traditional classes might have an RTOP score of 30-40.In order to compare teachers' self-reported values with the traditional RTOP, the PI and trained RTOP consultant conducted unannounced classroom visits to treatment teachers and completed the full RTOP evaluations.As IMPACT is attempting to measure changes over time as a result of teachers' participation in our PD program, we do not do specific interventions based on RTOP scores, and the individual RTOP results were not directly discussed with the observed teacher.
The PI's original RTOP scores (all items) averaged 38 (N = 22).Scores in general increased over the project to an average of 49 (N = 31), with an effect size of 0.95.Supporting this significant change, the PI identified just 2 teachers as "reformed" in his first assessment (March 2014), while he found 16 "reformed" teachers in the latest assessment (May 2015).A consultant to the project training the PI found similar results.[5] This indicates that (1) treatment teachers often report themselves scoring higher on RTOP than the scores reported by evaluators and (2) that significant changes are occurring in teachers' practices.

IV. RESULTS FROM STAFF OBSERVATION
Both Aubrecht and Schmitt conducted over 100 teacher classroom observations of IMPACT teachers in both highneeds districts throughout the 2013-15 school years.These observation reports are kept simple in order for the focus to remain on coaching the participating teacher.The observation has three components: teacher approach, student engagement, and an indicator of whether or not the students exhibited reasoning skills.During the 2014-2015 school year, staff observation time was also used to perform RTOP evaluations on treatment teachers.
Using the staff observation report, the project's internal evaluator is able to score the results of the classroom observations as a means of quantifying perceived changes in teacher practices.The rubric for classroom observations is included in the Appendix.
Points are awarded for student-centered approaches where there is evidence of high levels of student engagement in thinking and problem solving.In each of the three areas, improvement is found from the first half of the school year.Teachers show a shift from teacher-centered approaches (such as lecturing) to student-centered approaches (such as group projects/experiments), and also more evidence of student engagement.A more detailed explanation of the observation process and most recent data can be found elsewhere.[2] Figure 1 Classroom observations: So•uth-Western City School teacher observations 2013-14 school year

V. DISCUSSION
We present here preliminary evidence that IMPACT's approach supports teachers changing teaching practices.Results from staff observations and RTOP scoring indicate IMPACTed teachers spend more time allowing students to work in a collaborative group setting with a focus on inquiry-based instructional pedagogical practices.It appears, however, that teachers' self-reported RTOP scorings are higher than the findings of dispassionate observers.IMPACTed teachers appear to report themselves more "reformed" in their classroom teaching practices than they actually are.Because the CID is another self-reported measure, it is possible these numbers are inflated as well.
During the 2014-2015 school year, IMPACT was able to move from a pre-post method of data collection to a quasi-experimental design based on the fact that we were able to obtain a control group for our research.Our control group is comprised of SWCS middle school teachers who face similar demographic constraints as our treatment teachers.While we were able to obtain some preliminary data on student achievement during the '14-'15 school year, the 2015-2016 school year will be the first year we will have teacher-level data on content knowledge and pedagogical practices from a control group.
We expect more definitive evidence of changes in treatment teachers' teaching practices to emerge in the next year as treatment teachers are continually exposed to more in-depth science content knowledge, more rigorous handson minds-on research-based pedagogical approaches to content and opportunities to implement reflective teaching practices.