Evaluation Of The Translated Version Of The FMCE

The Force and Motion Conceptual Evaluation (FMCE) is a test that was originally developed in English to assess American students’ understanding of Newton’s laws of motion. The present study evaluates the Japanese translation of the FMCE (abbreviated as “FMCEJ”) using data consisting of the pretest results of 1095 Japanese students, most of whom were first-year students at a mid-level engineering school between 2003 and 2012 in Japan. The classical test theory indices of the FMCEJ indicate that its reliability and discrimination are adequate in assessing Japanese students’ views about force and motion. A comparison of these students’ understanding of the concepts of motion and force, as assessed with the FMCEJ, and that of American students, as assessed with the FMCE, indicates the validity of the FMCEJ.


INTRODUCTION
The use of standard examinations, such as the Force Concept Inventory (FCI) [1] and the Force and Motion Conceptual Evaluation (FMCE) [2], to test physics concepts has led to the development of many effective instructional methods in the United States [3].Japanese educators, however, do not have the means to measure the efficacy of their instructional methods.
The Concept Inventory is designed to probe not only students' understanding of Newtonian concepts but also concepts commonly held by students.The context and the writing style of the multiple-choice questions may influence students' responses to the test.The FCI and the FMCE are written in a style so as to measure accurately students' understanding of concepts.A technical translation must use consistent and accurate terminology and incorporate appropriate linguistic features.However, a translated version of the Concept Inventory must also be style sensitive because it needs to elicit not only correct concepts but also commonly held concepts.
The Japanese language, which belongs to the Altaic family of languages that includes the Turkic and Mongolic languages, is a very different language from English, which belongs to the Indo-European family of languages.Thus, the translation of English sentences into Japanese adds a layer of complication to the writing style.
Japan has developed its own educational system.As an educator working within this system, I have observed many students with the same commonly held naïve concepts about force and motion as those previously mentioned by Hestenes et al. [1].I assume that a properly translated concept inventory could produce similar outcomes as the original version.In this study, I evaluate my translated version of the FMCE, a test originally developed in English for American students, in terms of the validity of its administration to Japanese students.
The format of the FMCE [2] is suitable for assessing students' views of force and motion by cluster analysis.The structure enables the translated version to be validated by comparing Japanese data to American data in clusters.In this study, the FMCE, translated by the author into Japanese (abbreviated as "FMCEJ"), was administered under the same conditions to 1095 students.The FMCEJ was evaluated statistically based on the classical test theory [4].The clusters were then compared to American data to validate the efficacy.For the purpose of comparison, I used the data reported by Smith and Wittmann [5] to represent American data.

Surveyed Students
The surveyed students were mostly first-year students attending a basic mechanics class at the Kochi University of Technology (KUT) between 2003 and 2012.Each year, more than 500,000 university candidates in Japan take the National Center Test

PERC Proceedings
(NCT), an academic aptitude test.On average, KUT students surveyed for the present study earned a score of 63 points (of 100 points) on the NCT in physics, which is comparable to the national average of 62.68 points reported in 2013 [6].Thus, it is assumed that the data must be suitable for assessing the physics preconcepts of Japanese students at a mid-level engineering school.

General Statistics
The FMCE is designed to probe students' views of force and motion concepts using clusters with respect to subjects and representation.First, I used simple sum scores (SSS's) with all 47 FMCE questions for the statistical analysis based on the classical test theory.Second, I used the scoring systems used by Thornton et al. [7] and clustering suggested by Smith and Wittmann [5] to compare Japanese and American data.

Single Number Scores
Thornton et al. [7] determined a single number score (SNS) for the FMCE to characterize students' understanding of Newtonian concepts.These authors omitted 7 questions (5, 6, 15, 33, 35, 37, and 39) that they considered to be less reliable in probing students' understanding of the concepts.They suggested that 9 of the 36 questions be divided into three sets of 3 questions, with a score of 2 points being obtained if all 3 questions in a set are answered correctly.The rationale behind this scoring system is that the FMCE designers believe that students only understand the concepts involved if they answer all three questions correctly.They advocated SNS's for the questions (excluding those questions on energy conservation), producing a total of 33 points.
The SNS put forth by Thornton et al. [7] excludes the 4 questions on energy conservation.Because the FMCEJ includes these 4 questions, I used the 33 points designated for the SNS by Thornton et al. [7] and 4 points for the 4 questions on energy conservation and referred to them as the SNS in this study.The inclusion of the 4 questions on energy conservation thus changes the possible total score on the FMCEJ to 37 points.The basic statistical values of the SSS's and the SNS's are similar (Table I), and they are well correlated (r = 0.99).

Classical Test Theory Indices
All 47 questions were used to analyze statistically the FMCEJ, based on classical test theory [4].The reliability index (KR-20) and the discriminatory power of the test (Ferguson's δ) were 0.91 and 0.98, respectively, indicating that the FMCEJ is adequate in evaluating individuals' understanding of force and motion concepts (Table II).Table III shows each item's difficulty index (p), point biserial coefficient (PBC), and discrimination index (D) based on data from the top 25% of scorers and the bottom 25% of scorers.The columns labeled "MC" and "NA" in Table III show the FMCEJ distractor symbols for the most commonsense concepts (MC's) and the proportion of no correct answers (NA's), respectively.The 10 blank spaces in the MC columns of Table III indicate that no MC's were probed.The first 7 blank spaces correspond to the seven items regarded by Thornton and Sokoloff [2] as inaccurate for the assessment.The last 3 blank spaces correspond to velocity graph questions.Because the correct response rate was near 90% for these questions, the other responses were too minute to be listed.Almost all MC's of Japanese students agree with those of American students.The items with different MC's from those of the U.S. counterparts are items 23, 46, and 47 (presented in bold type in Table III).Item 23 regards the corresponding acceleration for a toy car with decreasing speed.The surveyed Japanese students most commonly used the F∝v model (52%), whereas Smith and Wittmann [5] found that American students most commonly used the brake model.In the brake model, the opposite force to the velocity direction causes the change in velocity.Items 46 and 47 refer to the velocity of a sled at the bottom of a hill when released from rest at the top of two kinds of hills.One hill is higher and less steep than the other hill.The most common incorrect response (around 20%) given by the Japanese students was that there was insufficient information to answer these questions.By contrast, the most common incorrect response given by American students was slope dependent; that is, they incorrectly attributed the higher speed of the sled to the steepness of the hill.III indicate that no MC's were probed.The first 7 blank spaces correspond to the 7 items (5, 6, 15, 33, 35, 37, and 39) regarded by Thornton and Sokoloff [2] as inaccurate for the assessment.The last 3 blank spaces correspond to velocity graph questions (items 40, 42, and 43); the correct response rate was nearly 90% for these items, and, therefore, these responses were too minute to be listed.c NA, no correct answer.

Cluster Analysis
Smith and Wittmann [5] used the following clusters to analyze FMCE data: Force Sled, Reverse Direction, Force Graph, Acceleration Graph, Newton's Third Law, Velocity Graph, and Energy.I used the same clusters to analyze the present study's data.The responses in the clusters of Force Sled, Reverse Direction, Force Graph, and Acceleration Graph reveal students' views of Newton's First Law and Second Law.The apparent proportions of the models used are roughly in the range of 10% to 20% for the F∝a model, 40% for the F∝v model, and 40% for the brake model.The percentage of students who used a particular model depended on the context of the question asked.However, the percentage of students who consistently used the F∝a model is estimated to be less than 5%.
The responses in the cluster of Newton's Third Law reveal that the most predominantly used commonsense concept models are the mass-dependent model and the action-dependent model.The percentage of students who consistently used the Newton's Third Law model is estimated to be less than 5%.
The responses in the cluster of Energy Conservation provide information about students' views of the law of conservation of mechanical energy.The most frequently used commonsense concept model is the steepness dependency model (three quarters of the Japanese students used this model).The percentage of students who used the energy conservation model is estimated to be less than 25%.
The FMCEJ data indicate that less than 5% of Japanese students at a mid-level engineering school have a firm grasp of the concepts of Newtonian forces and that less than 25% understand energy conservation concepts.

CONCLUSIONS
This study shows that the FMCEJ, a Japanese translation of the FMCE, is a high-quality conceptual multiple-choice test with levels of reliability and discrimination that are adequate for assessing Japanese students' understanding of the concepts of force and motion, based on classical test theory.The item analysis shows the majority of items with a discrimination index of more than 0.30 and a point biserial correlation of more than 0.20.A comparison of Japanese data and American data shows that Japanese students and American students provide the same most common answers.According to the cluster analysis, no intrinsic differences exist between Japanese and American students.

,
edited by Engelhardt, Churukian, and Jones; Peer-reviewed, doi:10.1119/perc.2013.pr.036Published by the American Association of Physics Teachers under a Creative Commons Attribution 3.0 license.Further distribution must maintain attribution to the article's author, title, proceedings citation, and DOI.

Table I .
Basic statistics for three scoring systems

TABLE II .
Evaluation of the FMCEJ

TABLE III .
Difficulty index (p), point biserial coefficient (PBC), discrimination index (D), and most commonsense concept (MC) Discrimination index (D) is based on data from the top 25% of scorers and the bottom 25% of scorers.b The 10 blank spaces in the MC columns of Table