Sensitivity of Learning Gains on the Force Concept Inventory to Students ’ Individual Epistemological Changes

Prior physics education studies have established a positive relationship between students’ learning gains and their epistemological views measured at single time points (i.e., either before or after instruction). These studies, albeit informative, miss the dynamic nature of students’ views about physics. In this study, we seek to investigate students’ epistemological shifts in relation to their learning gains. Students’ epistemological shifts were measured by the difference in their performances on the Colorado Learning Attitudes about Science Survey (CLASS) before and after a traditionally taught introductory mechanics course ( . Student learning gains were measured by their normalized gains on the Force Concept Inventory (FCI). Results revealed a large variation in students’ epistemological change after instruction despite a near-zero shift on average. Also, while no significant overall relation was detected between and FCI normalized gains, the FCI results seemed to be sensitive to epistemological changes among the lower-end students who remained below average on both the pre and post CLASS. In this case, those who experienced a positive epistemological shift achieved higher conceptual gains than those otherwise.


INTRODUCTION
One of the major accomplishments in contemporary science education is the recognition of learning as a multi-faceted, complex activity.[1] Researchers and educators have come to realize that learning can be influenced by various factors beyond just cognition.Among such factors, for example, are learners' epistemologies; or learners' views about the nature of knowledge and the nature of learning.In physics education, epistemologies are frequently reported to have a significant effect on the ways students approach the subject.[2][3][4] A body of studies has shown that students who view physics as a simple collection of disconnected facts and formulas and believe learning as knowledge being dispensed from experts tend to approach the subject matter by rote memorization.[4][5][6] On the other hand, those who view physics as a hierarchicallystructured, coherent enterprise hinging on a few fundamental principles are more likely to seek deep understanding through their own active knowledge construction.
Research along this line has further established a positive relationship between students' epistemological stances and their learning outcomes.Recent studies have found that students with sophisticated epistemological views-views aligned closely with those of physics experts-are able to achieve a higher performance on concept inventories (such as the Force Concept Inventory) [7] than those whose epistemological sophistication falls behind.[2,3] In determining student epistemological levels, a focus of these studies has been primarily placed on learners' "state of mind" at a specific time point either before or after instruction.In other words, prior studies have only investigated the relationship between students' learning gains and their epistemologies measured at single time points.
While these findings provide useful information regarding the (static) role of epistemologies in learning physics, they miss the dynamic nature of learners' growth on this matter (be it positive or negative) in relation to learning.In this study, we seek to fill in the gap by investigating how the changes in epistemologies relate to students' conceptual gains on a concept inventory; or put differently, to what extent the results from a concept inventory (i.e. the Force Concept Inventory) may be sensitive to, and hence be able to inform instructors of the different epistemological changes that students have experienced.

THEORETICAL BACKGROUND AND RESEARCH QUESTIONS
Epistemologies in general are considered as situated at the nexus of a learner's views, beliefs and attitudes about the nature of knowledge and learning.Because of its inherent differences from pure cognition, epistemology often is not explicitly addressed in typical classroom teaching.There is a whole host of research suggesting that desired epistemological growth is hard to come by.[2,9,10] Students' views about what physics is and how it should be learned often show little improvement even after they receive a full semester of carefully designed and implemented instruction.On the other hand, epistemologies are malleable and can be directly or indirectly shaped by students' various learning experiences in or out of classroom.[11,12] It is therefore reasonable to hypothesize that even within a relatively uniform student group in which there is little overall epistemological shift, the change at the individual students' level can still vary significantly.
Such variation, if verified, can provide us a great opportunity to better investigate the relationship between the changes in students' epistemologies and changes in their conceptual learning.Prior studies have shown that students with expert-like views (measured at a single time point either before or after instruction) are more likely to achieve a higher conceptual gain than those of novice-like views.[2,3] But whether a change in epistemologies relates to a change in conceptual outcomes is a different topic.In fact it delves into the dynamic interplay between the two; in other words, it pertains to the issue of how a learner's epistemological growth is translated into his/her growth in conceptual learning.
At least two reasonable hypotheses can be formulated on this matter.One is that a growth in a learner's epistemologies can be quickly manifested through his/her performance on conceptual learning.As a result, those who experience a large positive shift in one area will likely show a large positive gain in the other.A competing hypothesis, however, is that such a positive relation may not exist, perhaps due to a slow translation from epistemological growth to conceptual growth and thereby creating a meager chance for a concept inventory used for gauging learning gains to capture such a nuance.
This study is aimed at investigating this previously unexplored issue.Specifically, we seek to answer the following two research questions.(1) To what extent do individual students' epistemological shifts vary even in a class where there is little overall shift after instruction?(2) How does the variation in students' epistemological changes relate to their conceptual learning gains as measured by a concept inventory (i.e.FCI; see Methods for details)?Or from a different perspective, how sensitive are the learning gains on a concept inventory to the different epistemological changes that students may have experienced?

METHODOLOGY
Participants.Students of interest in the study are 167 college freshmen from a tier-one research institution in East China, whose scope and scale of academic operations are similar to those of the large State universities in the U.S. All the participants were science or engineering majors taking calculus-based introductory mechanics as a mandatory course to fulfill their degree requirements.The course followed canonical topical sequences, including kinematics, Newton's laws, work and energy, linear momentum and collision, angular momentum and rotational dynamics.The course consisted of 3 one-hour lectures, a one-hour recitation and a two-hour lab every week, similar in structure to those taught in the U.S. The lecture was traditionally delivered by a senior faculty member in a large lecture hall.The weekly recitation and lab sessions were led by teaching assistants using end-of-chapter problems and cook-book lab manuals.
Instruments.We used the Colorado Learning Attitudes about Science Survey (CLASS) [2] and the Force Concept Inventory (FCI) [7] to gauge, respectively, students' epistemological views and conceptual learning outcomes.Both instruments are research-based and have been broadly recognized for their value in empirical investigations and instructional practices.In this study, we used the Chinese version of the CLASS and FCI with our student participants.The making of our translated instruments followed a rigorous process involving both forward (English to Chinese) and backward (Chinese to English) translations that were carried out by a group of bilingual content experts and were empirically checked with Chinese students at different academic levels.[9] We administered the two instruments as both a pretest in the first week and a posttest in the last week of the course.The FCI was given as an in-class quiz, and the students were allowed a maximum of 30 minutes to complete the test.The CLASS was conducted as part of lab activities and was completed within approximately 15 minutes.In all cases, the tests were paper-based.Students received a small amount of credit for participation.In this study, the reliability of the two instruments in both the pre and posttests ranged from 0.74 to 0.81, sufficiently high for our results to be considered reliable.
To investigate the variations in student epistemological changes and their relations with conceptual gains, we calculated pre and post differences in student CLASS scores as well as their normalized gains on the FCI.The former ( ) was used to represent students' epistemological shifts before and after instruction, and the latter (FCI normalized gain) was used to indicate their conceptual gains over the same course; two commonly accepted indicators for gauging student growth in these matters.
[As done in relevant prior research, [2,11] we used absolute gains on the CLASS ( ) instead of normalized gains to indicate epistemological changes.]

DATA ANALYSIS AND FINDINGS
The results of student performance on the CLASS and FCI for both the pre and posttests are listed in Table 1.As seen, students demonstrated a significant conceptual gain on the FCI after instruction, but their overall change in the CLASS was noticeably small [paired-sample t(166)=1.38,p=0.16], representing a case in which the course instruction had little impact on students' epistemological growth.However, a closer look at the data shows that the change in CLASS scores ( in fact spreads out across a considerably wide range from as low as nearly -50% to as high as nearly +50% (see Fig. 1), with a kurtosis of -0.2.Here, a negative kurtosis suggests a flat bell-shaped distribution with a less distinct central peak compared to (although not significantly deviating from) the normal distribution.This wide range, which is otherwise masked by the average, reveals a great variation in student epistemological changes.A scatter plot of student pre and post CLASS scores, as depicted in Figure 2, further shows this wide variation.Here, the solid diagonal line is y=x, representing equal pre and post scores.Dots above this line indicate a higher post score (positive change), and dots below this line signify a lower post score (negative change).Vertical and horizontal dashed lines mark the pre and post averages respectively.
To investigate how students' epistemological shifts relate to their conceptual gains, we calculated the Pearson's correlation between and FCI normalized gains and found a near-zero coefficient (r=0.1, p=0.26).To better understand what may underlie this finding, we conducted a finer-grained analysis by dividing the students into six groups (I-VI) based on their pre, post and scores (see Fig. 2 and Table 2).Group I is those who achieved a positive change on CLASS with both pre and post scores above average.Group II consists of those who had a negative change on CLASS but still maintained an above-average score on both pre and posttests.Group III includes those who achieved a positive shift from a below-average pre score to an above-average post score.Group IV represents those who also achieved a positive shift on CLASS but remained below average on both pre and posttests.Group V is those who had a negative change on CLASS with both pre and post scores below average.Group VI consists of those who experienced a negative change from an above-average pre score to a below-average post score.
We compared students' FCI normalized gains across these six groups (see Fig. 2) and found that two groups (IV and V) were statistically different (Kruskal-Wallis χ 2 =4.4,p=0.03; small samples), while the other four (I, II, III, and VI) differed little [F(3,122)=0.81,p=0.49].These results suggest that while there is no detectable correlation overall between and FCI gains, a change in learners' epistemologies measured  by the CLASS does manifest itself onto the conceptual gains measured by the FCI in the lower group of students-students with below-average epistemological views both before and after instruction.Specifically, in this case those who made a positive epistemological shift were more likely to achieve a higher conceptual gain than those with a negative shift (and in fact the highest conceptual gain among all the groups).

DISCUSSION
The above results reveal two interesting aspects about students' epistemological changes in relation to conceptual learning.One is the large variation of epistemological shifts at the individual students' level despite the near-zero overall change after instruction.This otherwise masked information suggests that perhaps factors beyond course instruction have influenced (or mediated) students' views about the nature of physics.Had the instruction impacted students uniformly without influences from other factors, students would have demonstrated at least similar changes in their epistemologies.
Another interesting aspect emerging from our study is the lack of overall correspondence between students' epistemological growth and their conceptual gains.As we hypothesized before (which is worth future longterm verification), students' epistemological change may not be able to quickly translate to a measurable learning outcome within a semester, at least not in our case where we used the FCI to gauge student's conceptual gains.Also surprising is the fact that those (group VI) who decreased in epistemological sophistication from above average before instruction to below average after instruction still managed to obtain a conceptual gain just as high as those who remained at the top (groups I and II).This seems to imply that a high performance on the FCI (an instrument known for its capability of distinguishing sophisticated students from others) [7] can be achieved without necessarily understanding the nature of physics.That said, our results also show that those who had a positive shift but still remained at the lower end of the epistemological scale after instruction (group IV) were in fact the highest achievers on the FCI gains among all groups.In other words, the learning gains measured by the FCI, albeit insensitive to epistemological changes in general, are able to capture the nuance among the lower end of students.
Our results no doubt have further revealed the complex nature of learning and epistemology.Although more systematic investigations along this line of research are needed, our study provides a first step toward a better understanding of the dynamic role of epistemological growth in relation to conceptual learning in tertiary-level physics education.

TABLE 1 .
Average CLASS and FCI results (%).A scatter plot of student pre and post CLASS scores.The diagonal line is y=x.The vertical and horizontal dashed lines indicate pre and post average CLASS scores respectively.
FIGURE 1.A histogram of student epistemological changes measured by CLASS (Δ CLASS = Post CLASS -Pre CLASS ).FIGURE 2.

TABLE 2 .
Average normalized gains on the FCI for student groups I--IV.Nine students achieved a zero epistemological change ( ) and were not included in the analysis.