Comparing Traditional and Studio Courses through Gains and Losses

We analyze data from over 5500 students, spanning multiple semesters and instructors within a single institution, to explore how Gains and Losses in Force Concept Inventory scores vary between instructional approaches, including traditional lecture/laboratory courses, lecture-supported “mini-studio” and studio courses. We observe that the trend for Gain to increase and Loss to decrease with increasing pre-test score, initially observed across institutions, extends to individual student effects within an institution. The correlations between pre-test score and Gain/Loss were least pronounced for more minimally guided instructional strategies, such as studio. Additionally, we find similar or improved Gains and Losses for studio-type courses compared to traditional courses. These trends argue against the claim that novices require strongly guided instruction.


INTRODUCTION
In a recent paper, Lasry, Guillemette and Mazur [1] highlighted an intriguing trend of students switching correct pre-test Force Concept Inventory (FCI) [2] responses to incorrect responses on the post-test.This switching, or Loss, occurred at a higher rate than noise due to guessing [1].The rate of Loss increased with decreasing pre-test score, while Gain decreased.The authors proposed several explanations for the high rate of Loss observed, such as confusion induced by instruction, fragility and context-dependence of the knowledge building process, and dependence of the ability to add new concepts on existing concepts.The authors suggested that Gain and Loss be explored for various instructional strategies to test these explanations and identify efficient pedagogies with high Gains and low Losses.
At the University of Central Florida (UCF), we began collecting FCI data in all first semester introductory courses in Spring 2011, across a variety of instructional methodologies (described below).This dataset allows us to explore how the trends observed by Lasry et al. differ by course type within a single institution and test hypotheses about the efficiency of teaching methods.For example, Kirschner, Sweller and Clark argue that novices benefit from strongly guided instruction rather than being required to construct concepts and procedures for themselves.[3].Thus, we might expect to see a higher correlation between pretest score and Gain or Loss for more minimally guided courses.In this paper, we explore the correlation of pre-test score with Gain and Loss and the variation in Gain and Loss across instructional strategies.

SAMPLE AND METHODOLOGY
We explore FCI results from the first course of the two-course introductory sequence at UCF, which covers typical mechanics topics, for both algebra-and calculus-based physics courses.These courses have been offered in traditional, honors, studio and lecturesupported mini-studio (mini-studio) modes.Traditional courses met for three hours of lecture (with 50 to 300 students) and three hours of laboratory per week (with approximately 30 students); in most cases, one hour of lab was used for recitation.The calculus-based course was also taught as a traditional honors section, with classes of 20 students from the university's honors program.Both courses were additionally taught as more minimally guided studio courses (with 50 to 100 students).Studio courses are marked by flexibility to move between lecture, laboratory and recitation activities, with an emphasis on group work.At UCF, the algebra-based studio emphasized student-designed video analysis projects and most sections of the calculus-based course closely followed the SCALE-UP curriculum for Matter and Interactions with little lecturing [4].The algebra-based course was additionally taught as a mini-studio, which combined three hours of lecture with three hours of studentcentered engagement with concepts, problems and experiments, rather than a traditional recitation and laboratory.Table 1 summarizes the number of sections, instructors and students representing each mode.
To allow comparison to the Lasry data, we analyze performance on the FCI through question-level fractional Gain and Loss.(We note that the FCI may not be well matched to the curriculum used in the calculus-based studio courses [5].)Following Lasry, Gain is defined as the ratio of questions corrected from pre-test to post-test to the total number incorrect on the pre-test [1].Conversely, Loss is defined as the ratio of questions switched from correct on pre-test to incorrect on post-test to the total number correct on the pre-test.Initial visual inspection of the data revealed that scores were not normally distributed, and standard transformations for percentage data did not improve normality.Non-parametric statistical tests and bootstrapped standard errors, effect sizes, and confidence intervals and robust Cohen's d R were calculated with the bootES package for R [6,7].These methods and the specific statistical tests used below are described in Ref. 7 and 8.

Pre-test Correlation with Gain and Loss
Lasry et al. found that, across institutions, students with higher pre-test scores gained more and lost fewer concepts from pre-to post-test [1].We use our dataset to explore this trend within an institution and among different instructional strategies.Figures 1 and 2 display Gain (a) and Loss (b) versus pre-test score for the algebra-and calculus-based courses by course type.Due to the large sample size, these scatterplots are heavily populated.The R package ggplot2 [9] was used to layer semi-transparent data points to emphasize more and less heavily populated regions.The best fit line and 95% confidence interval (shaded region) were also added.These plots indicate that the previous trend observed extends to within institution effects and holds across course types.
We explore the correlations with a non-parametric version of Pearson's r, Kendall's τ; results, displayed in Table 2, indicate all correlations are significant, with effect sizes ranging from small to large (small: τ~.06; medium: τ~.19; large: τ>.33) [10].The pre-test score is more strongly correlated to Loss than Gain across all courses, and all correlations were stronger for the calculus-based than the algebra-based course.In most cases, correlations are slightly smaller with similar slopes for the less strongly guided courses, such as studio, indicating novices fare well in these courses.

Gain and Loss across Course Structures
Lasry et al. suggested that Gain and Loss analysis could be used to assess the efficiency of instructional approaches.We use our dataset to compare course   types and explore whether the Gain and Loss analyses yield similar insight into course effectiveness.Figures 3 and 4 display the differential Gain (a) and Loss (b) by instructional method for each section of the algebraand calculus-based courses.The overall average for each was calculated across all sections (weighted evenly for each section), and individual section averages are compared to the grand average.Exploring Fig. 3, we observe that in the algebra-based course, the mini-studio sections typically performed better than average, most traditional sections performed worse than average, and there was a wide variety in the performance of studio sections on Gain and Loss.For the calculus-based course, the honors sections typically performed above average, most traditional performed below average and studio sections were scattered about the average.Both measures lead to similar conclusions about the effectiveness of the course structures.However, the Loss analysis shows the greatest stability across semesters and instructors for the calculus-based courses.Individual sections show some variation across measures; for example, the Fall 2012 algebra-based traditional section had average Gain but better than average Loss.Non-parametric tests were used as the data violate normality assumptions.However, density plots revealed that certain groups of interest had very similar distributions of pre-test scores.Thus, we proceed with the Kruskal-Wallis test, a non-parametric ANOVAtype test, while noting this does not account for differences in pre-test.For the algebra-based course, there were significant differences by instructional method for Gain (χ 2 (2)=101, p<.001) and Loss (χ 2 (2)=28.5,p<.001).Wilcoxon-Mann-Whitney tests with Bonferroni corrections and calculations of robust Cohen's d R and confidence intervals were used to identify differences between sections.The results are summarized in Table 3, and indicate significant differences between methods for all three measures, except for Loss between studio and traditional courses.We observe the strongest differences between ministudio and traditional sections, and only weak differences in Loss scores across all three comparisons.For the calculus-based course, the honors students had higher pre-test scores than students in traditional and studio courses.Thus, we focus on only the traditional and studio courses here.The Wilcoxon-Mann-Whitney test indicated a significant difference but small effect size between studio and traditional sections' Gain (W=833735, p<.001, d R =.14 [.03, .26])and Loss (W=662961, p<=.002, d R =-.06 [-.18, .06]).As with the algebra-based course, Gain and Loss scores give us similar information about teaching methods, with smaller differences in Loss than Gain.

DISCUSSION AND SUMMARY
We observed that the correlation of pre-test with Gain/Loss observed by Lasry et al. across institutions held at the level of individual students within an institution and extended across course types.Students in the calculus-based courses had higher Gains and lower Losses than students in the algebra-based course on average (40.9% vs. 33.5% and -21.0% vs. -38.4%),which is expected given the typical preparation of students in these courses.However, it may be somewhat surprising that the correlations were stronger and steeper between pre-test and Gain/Loss in the calculus-based courses than the algebra-based courses.
We also observed differences in Gain and Loss between course structures, with stronger differences in Gains than Losses.In the algebra-based course, the mini-studio performed better than traditional and studio courses on both Gain and Loss and the studio performed better than the traditional course on Gain.In the calculus-based course, the studio performed somewhat better than the traditional course on Gain and Loss.These findings refute Kirschner et al.'s contention that novices benefit from more strongly guided instruction [3]; pedagogies that put more emphasis on student guided learning performed better than the strongly guided traditional course.

Figure 1 .
Figure 1.Pre-test correlation with Gain(a) and Loss (b) for algebra-based courses.Shaded area represents 95% confidence interval.

Figure 2 .
Figure 2. Pre-test correlation with Gain(a) and Loss (b) for calculus-based courses.

TABLE 1 .
Summary of seven semesters' data Course Type

TABLE 2 .
Summary of correlation analysis.