Transformative Experiences and Conceptual Understanding of Force and Motion

This research investigates relationships between students’ understanding of force and motion as measured by the Force Concept Inventory (FCI) and their out-of-school engagement as measured by the Transformative Experience (TE) Survey. Surveys were administered in an introductory algebra-based physics course (N = 68 completely matched). Positive correlations are observed between students’ overall TE scores and FCI pre and post scores (r = 0.4, p < 0.005). Correlations between TE scores and FCI gains and normalized gains, however, are not statistically discernable from zero. These preliminary results suggest that students with a better understanding of force and motion concepts may more readily engage with content outside of the class, but such out-of-school engagement may not be contributing to further learning gains. We conclude with a discussion of how responses to individual statements on the TE survey vary between students in the top and bottom quartiles based on FCI performance.


INTRODUCTION
In physics education, interactive engagement (IE) pedagogies have been strongly linked to improved student learning outcomes [1].This link has been demonstrated thoroughly in the context of student learning about force and motion using conceptual surveys such as the Force Concept Inventory (FCI) [2].Our interest in courses that use IE pedagogies is geared toward understanding the conditions that contribute to students' deepening interest and engagement with content outside of class.While we believe that out-of-school engagement is an important goal itself, we are also interested in understanding how out-of-school engagement may correlate with standard measures of learning.More specifically we sought to address the following question: "Do students who engage more deeply with concepts of force and motion outside of class also develop stronger conceptual knowledge of force and motion?" To conceptualize out-of-school engagement, we draw on a construct called transformative experience (TE) [3].Transformative experiences involve freechoice transfer of concepts to out-of-school settings, meaning-making in the everyday world, and a deepening interest and valuing of scientific ideas.In previous work [4], we have used surveys to document variations in the extent and depth of transformative experiences among different types of IE courses.
In our current study we explore the relationship between out-of-school engagement and conceptual learning about force and motion using correlational methods.Despite a small sample size, we find significant positive correlations between TE scores and student FCI pre-and post-test scores but not with FCI gains or normalized gains.

CONTEXT FOR RESEARCH
We investigate our research question within the context of a first-semester introductory algebra-based physics course.The course covers traditional introductory physics topics and is taken by a variety of different majors.The course utilizes some IE methods such as Peer Instruction [5] in a large-lecture setting and collaborative problem-solving in an integrated problem-solving / laboratory classroom.Despite the use of IE methods, the average FCI normalized gains achieved by students in the course has remained lowbetween 0.2-0.3 for many years.While these results are low, they are not inconsistent with efforts to introduce instructional reforms in algebra-based physics courses at similar institutions [6].

METHODS
To assess students' conceptual understanding of force and motion concepts, the FCI was adminstered to students on the first day of class (N = 198) and then again during the last week of class (N = 144).FCI scores were then calculated as number of questions answered correctly (out of 30).For students with matched FCI pre-and posts-tests, both gains and normalized gains were calculated (N = 104 matched).
The TE survey that was adapted for our research has been described in a previous publication [4].It consists of thirty-one statements to which students respond on a four-point Likert scale (see Table 1) and was tailored to the address concepts pertaining to force and motion.The survey was adminstered after six weeks of instruction on force and motion.A total of 68 students took all three surveys and could be matched.The TE survey can be scored using different methods.One common method is to first collapse "agree" and "strongly agree" into a single category and then report the percentage of agreement.This first method has been used to score other likert surveys that are commonly used in physics education such as the Colorado Learning About Science Survey [7].We will refer to this method as the Percentage Agreement (PA) method.A similar method involves calculating odds of agreeing.Since odds grow quickly, it is standard to take the logarithm of odds.We will refer to this scoring method as the Log Odds (LO) methods.
Both of the above methods have the disadvantage of losing information from the full scale.In order to utilize the full range, multiple odds can be calculated and combined.For each student, in addition to calcuating the odds of agreeing, we calculated the odds of strongly agreeing and the odds of not strongly disagreeing, as is done with ordinal regression analysis [8].For our purposes, we calculated these odds in order to combine them into a single score by taking the logarithim of their product.This effectively shifts the LO score up or down depending on levels of strong agreements / disagreements.We refer to this score as the Combined Log Odds (CLO) Method.

RESULTS
Mean FCI and TE Scores for the fully matched students are shown in Table 2.These scores are not statistically discernable from either the larger pool of unmatched data nor an even larger unmatched pool of data aggregated over several semesters (N > 500).The statistical similarly suggests that the sample is likely representative of the larger population, despite the fact that the sample is small and not random.

TE Correlates with FCI Scores Not Gains
Correlation coefficients were calculated as a measure of association between TE scores and four FCI-related measures: pre-test score, post-test score, gain, and normalized gain (see Table 3).Across all three TE scoring methods, statistically significant correlations are observed for FCI pre-test scores and post-test scores but not for FCI gains or normalized gains.Focusing in on the CLO method, Figure 1 shows four plots that illustrate the relationship between TE and FCI scores and the lack of evidence for a strong relatonship between TE and FCI gains.

Students Differ More Out of Class Than In
In order to obtain a more nuanced understanding of student responses, Figure 2 shows the percentage agreement on each of the survey items for both top and bottom quartiles based on post-test FCI performance.The groups differ the most on items related to talking, thinking, being interested, and noticing examples outof-school.For example, 11 out of 17 the students in the top quartile report thinking about force and motion concepts out of class compared to only 4 out of 17 in the bottom quartile.In contrast, the groups are nearly identical for most items related to in-class engagement.For example, 15 out of 17 students in the top quartile and 16 out of 17 students in the bottom quartile reported thinking about concepts in class.This suggests that, in terms of the TE survey, differences in scores are likely being driven by differences in their out-of-class engagement rather than their in-class engagement, which is uniformly high.

DISCUSSION
results provide evidence for (i) a weak to moderate positive relationship between TE scores and FCI scores and (ii) a weakly negative to weakly positive relationship between TE scores and FCI gains.Evidence for these relationships holds even when the TE survey is scored using different methods, including courser methods that are more commonly used in physics education and finer-grained methods that better distinguish student scores from each other.Ultimately, the precision and generalizability of our claims are limited by sample size and the need for replication both within and across contexts.
Correlational studies, of course, cannot tell us about the causal mechanisms that drive such relationships.With this in mind, however, we propose the following three mechanisms and consider the merits of each within the backdrop of our data set: Mechanism 1: Stronger conceptual knowledge of force and motion supports transfer, including transfer to outof-school setting.
Mechanism 2: Increased engagement with force and motion concepts, including out-of-school engagement, supports deeper learning of force and motion concepts.
Mechanism 3: Out-of-school engagement and stronger conceptual knowledge are driven by feedback loops associated with positive physics identity and interest.Mechanism 1 cannot be ruled out by our data.Students with higher FCI scores (pre or post) had higher TE scores, especially on items related to out-ofschool engagement.Our data set does provide evidence that is inconsistent with Mechanism 2. Students who reported more out-of-school engagement did not appear to learn any more than students who reported little.Mechanism 3 cannot be clearly evaluated from our data set alone.To further investigate, we compared the TE scores of physics majors (N = 8, 92% agreement) to non-physics majors (N = 152, 57% agreement).While this result is not particularly surprising, physics majors uniformly had the very highest TE scores.It seems unlikely that their high scores would be accounted for by stronger knowledge of force and motion alone.Such data may lend support to Mechanism 3, involving a more complex constellation of factors.
Returning to Mechanism 2, it is perhaps surprising that high out-of-school engagement did not correlate with high learning gains.One possible explanation could be that out-of-school engagement lacks the scaffolding and feedback characteristic of in-class IE.Students could, for example, spend time being engaged outside of class reinforcing alternative conceptions.Other differences could be more related to the intensity and duration of engagements or the social-discursive nature of in-class versus out-of-class engagements.Physics majors, for example, tend to participate in communities and have access to settings that can provide such opportunities-feedback from more experienced peers and sustained discussions.Such considerations suggest a need understand how students' lives outside of class afford TE in addition to what classroom facts may support it.
Our ongoing efforts are aimed at both honing the empirical precision of our results as well as pursuing theoritical clarity concerning underlying mechanisms.

1 .
During class, I talk about… 2. I think about…, when I see… 3 Outside of class, I talk about… 4.During class, I think about… 5.I talk about… just for the fun of it 6.Outside of class, I think about… 7. I find myself thinking about… in everyday life.8.During class, I use the knowledge I've learned about … 9. Outside of school, I use the knowledge I've learned about… 10.I use the stuff I've learned about…even when I don't have to.11.I look for chances to use my knowledge of…in my everyday life 12.During class, I see things in terms of the laws I've learned about… 13.When I am working on a class assignment about… I tend to think of them in terms of… 14.If I see a really interesting situation (either in real life, in a magazine, or on TV), then I think about it in terms of… 15.I can't help but see situations in terms of … 16.During class, I notice examples of… 17.I notice examples outside of class of… 18.I look for examples outside of class of… 19.Learning about…is useful for my future studies or work.20.Knowledge of…helps me to better understand the world around me. 21.Knowledge of… is useful in my current, everyday life.22.I find that knowledge of…makes my current, out-of-school experience more meaningful and interesting.23.Knowledge of…makes learning physics much more interesting.24.In class, I find it interesting to learn about… 25.I think…is an interesting topic 26.I find it interesting in class when we talk about… in terms of… 27.I'm interested when I hear things about…outside of school 28.I find it exciting to think outside of school about… 29.The ideas we learned changed the way I see… 30.I think about…differently now that I have learned about… 31.I pay more attention to…now.

FIGURE 1 .
FIGURE 1.The four plots above show correlations between TE scores (CLO Method) and four different FCI measures: pre-test scores, post-test scores, gain, and normalized gain.With respect to TE scores, both pre-test and post-test scores have a significant positive correlation coefficients, r = 0.2 -0.6 (p < 0.005); while gain (r = -0.2-0.2) and normalized gain (r = -0.1 -0.3) have correlation coefficients that do not statistically differ from zero (p = 0.86 and p = 0.36, respectively).

FIGURE 2 .
FIGURE 2. The two plots above show agreement /disagreement levels (percentage agree method) for the top and bottom quartiles based on FCI post-test performance (N = 17 for each group).Each bar represents 100% (of the responses to a survey item) and the bar's location is positioned vertically to indicate the percentage agreement (above the axis) and the percentage disagreement (below the axis).Darks shadings are used to indicate the fraction of strong agreement or strong disagreement.The top quartile, representing students with a post-test FCI score > 16, had an average agreement rate of 71%.The bottom quartile, representing students with a post-test FCI score < 10, had an average agreement rate of 49%.Individual items where there is both a statistically significant difference using Mann-Whitney (p < 0.001) and a median difference of one are indicated with asterisks (*).

TABLE 1 .
TE Survey Statement showing the "Stems"

TABLE 2 .
Mean Scores for FCI and TE Survey