Testing Students ’ Understanding of Vector Concepts

After four years of research we designed a 20-item multiple choice vector concept test (Test of Understanding of Vectors, TUV). In this article we analyze: 1) the reliability and discriminatory power of the test, and 2) students’ understanding of the vector concepts evaluated in the test. The final version of the test was administered in English to 423 students who were finishing an Electricity and Magnetism course at a large private Mexican university. In the first part of the article, we show results indicating that the TUV is a reliable assessment tool. In the second part, we examine students’ overall performance on the test and analyze the results of the five most difficult items for students: geometric interpretation of dot product, calculation of dot product of two vectors written in unit-vector notation, graphic representation of a unit vector, calculation of the direction of a vector written in unit-vector notation, and graphical subtraction of vector in 2D.


INTRODUCTION
In recent years, researchers have investigated students' understanding of vector concepts.However, we identified two specific needs in this line of research.The first need is that, to our knowledge, there is no multiple-choice test to evaluate students' understanding of vector concepts that has been designed following the steps recommended by physics education researchers [1,2].The second need, that to some extent is a consequence of the first, is that there has not yet been a study of a large population of students that analyzes their understanding of vector concepts.
To respond to these needs, after four years of research we designed a 20-item multiple choice vector concept test (Test of Understanding of Vectors, TUV).This article addresses two objectives: 1) to evaluate the reliability and discriminatory power of the TUV following the procedure suggested by Ding et al. [1] and 2) to analyze, using the TUV, the understanding of vector concepts of a large population of students upon completing the introductory physics courses at the university level.

PREVIOUS RESEARCH
Six studies [3][4][5][6][7] (from other researchers) identified frequent errors that university students make regarding vector concepts in problems without physical context.The methods used in these studies are individual interviews or tests that use open-ended problems.
In one of these studies, Van Deventer [7] designed isomorphic mathematics and physics multiple-choice vector tests to compare students' performance in both contexts.However, the test was only based on interview results, with the disadvantage that the study's sample was very small (eleven students).By contrast, we constructed the distractors of the TUV based on the results of several administrations of openended problems.

DESIGN OF THE TUV AND METHODS
The research was conducted in a large private Mexican university.To develop a complete taxonomy of the most frequent errors that university students make with regard to vector concepts, we first conducted several studies based on the administration of open-ended problems (as recommended by Beichner [2]) in which a total of 2,067 students participated.
In designing the open-ended problems, we took into account the results of the previous studies mentioned earlier [3][4][5][6][7].Note that the results of some of our other studies based on open-ended problems have been reported in previous articles [8][9][10][11][12].
Using the responses from the open-ended problems, we designed and administered a first version of the TUV to the students.From the analysis of this administration, we designed the final version of the TUV with 20 items.This test was administered in English to 423 students finishing a calculus-based course on Electricity and Magnetism.This course is the last of three introductory physics courses students take in this institution.Calculation of cross product of vectors written in unit-vector notation 77% The TUV covers all the vector concepts used in the introductory physics courses at the university level, as shown in Table I.It has 11 items that evaluate students' understanding of vector concepts posed in graphical form (items 1-5, 9-13, 19), 7 items that evaluate students understanding with the calculation of vector concepts (6, 8, 14, 15, 17, 18, 20), and 2 items that cover graphical and calculations aspects (7,16).Figure 1 shows items 13 & 17 of the TUV.Item 13 is an example of the problems in graphical form.For the majority of these items, we used a grid as recommended by Nguyen and Meltzer [4], and positioned the vectors in a tail-to-tail representation since some researchers have established that students have more difficulties with this representation [9,13].Finally, item 17 is an example of the calculation of vector concepts.

Reliability & Discriminatory Power of the TUV
In this section we address the first objective of this study.We evaluate the reliability and discriminatory power of the TUV test, performing the five statistical tests suggested by Ding et al. [1].The three measures focus on individual test items: the item difficulty index, the item discriminatory index and the item point-biserial.In this report we present only the average indexes of these measures.The two other measures focus on the test as a whole: the Kuder-Richardson reliability test and Ferguson's delta test.Next we perform these five statistical tests.item difficulty index (P) is according to Ding et al. [1] a measure of the difficulty of a single test question, and it is calculated by taking the ratio of the number of correct responses on a question to the total number of students who attempted the question (N).Ding et al. recommend calculating the item difficulty average index, which is the sum of all the item difficulty indexes of the test divided by the number of items.The criterion range for the averaged difficulty value is [0.3-0.9].For the TUV the average difficulty index is 0.68, which falls into the criterion range.
The item discriminatory index (D) is a measure of discriminatory power of each item in a test.To calculate this index (using the 25%-25% method) we divide the sample of students (N) into four different groups of equal size according to their score: a high group H (top 25%, students with an individual total score higher than the third quartile) and a low group L (bottom 25%, students with a score lower than the first quartile).For each item, we counted the number of correct responses in both H and L groups: namely, N H (top 25%) and N L (bottom 25%).The discriminatory index for an item is calculated as D = [N H (top 25%)-N L (bottom 25%)]/ (N/4).Ding et al. [1] recommend that the averaged discriminatory index be .For the TUV, this index is 0.48 which fulfills the criterion.
The point-biserial coefficient ( ) is a of consistency of a single item with the whole test, which reflects the correlation between students' scores on an individual item and their scores on the entire test.The formula to calculate this coefficient is where is the average total score for those students who score 1 for the test item, which means they correctly answered this item, is the average total score for the sample, is the standard deviation of the total score for the sample, and P is the difficulty index for this item.Ding et al. [1] recommend that the criterion for the average coefficient is ≥ 0.2.For the TUV the average is 0.44, which fulfills the criterion.
The Kuder-Richardson reliability index is a measure of the self-consistency of a whole test.The formula to calculate this reliability index is ] where is the number of items in the test, is the standard deviation of the total score for the sample, and is the difficulty index for the ith item.Ding et al. [1] follow the criterion that a test with a reliability index higher or equal to 0.7 is reliable for group measures.The index for the TUV is 0.78, which meets this criterion.
Ferguson's delta measures the discriminatory power of an entire test by investigating how broadly the total scores of a sample are distributed over the possible range.The formula to calculate Ferguson's delta is , where is the frequency of occurrences of cases at each score.Ding et al. [1] follow the criterion that a test with a Ferguson's delta higher than 0.9 offers a good discrimination.Ferguson's delta for the TUV is 0.97, which is greater than this value.
We present a summary of the five statistical tests in Table II.From the analysis we can conclude that the TUV is a reliable test with satisfactory discriminatory power.

Students' Understanding as Shown by the TUV
In this section we cover the second objective.

Students' Overall Performance
The average of the scores of the TUV (from the sample of 423 students who were completing an Electricity and Magnetism course) is 13.52 correct problems out of 20.If we consider that students are finishing the third introductory physics course, and that the TUV evaluates concepts that are frequently used in these courses, it is interesting to note that the students who score the average value (13.52) have difficulty in correctly answering 6 items of the TUV.

Students' Understanding of the Items in the TUV
Table I shows the proportion of students correctly answering all items of the TUV.The range of percentages is very wide, from 33% of item 3 (geometric interpretation of dot product) to 92% of item 10 (graphic representation of a vector).To analyze these results, we decided to cluster the problems based on the range of proportion of the correct answer.We classified problems as high difficulty level if they had a proportion of correct answers that was equal to or less than 60%, as medium difficulty level if they had a correct proportion of 60% to 80%, and as low difficulty level if their proportion of correct answers was equal to or higher than 80%.
The items with a low difficulty level are: 4, 20, 5, 9 & 10.The items classified as medium difficulty level are: 19, 16, 11, 14, 1, 15, 6 & 7. Finally, the items considered to have a high difficulty level, in order of decreasing difficulty are: 3 (interpretation of product), 8 (calculation of dot product of vectors written in unit-vector notation), 2 (representation of unit vector), 17 (calculation of direction of a vector written in unit-vector notation), 13 (subtraction of vectors in 2D), 12 (interpretation of cross product), and 18 (calculation of cross product magnitude).

Most Frequent Error in Each of the Five Most Difficult Items
The items with a high difficulty level have strong instructional value; therefore, we decided to analyze the most frequent error of each of the five most difficult items of the test (Table III).Consider that the subtraction vector A-B of A=-3î+3ĵ and B=-2î-2ĵ is -1î+1ĵ.26% Item 3 was only answered correctly by 33% of students.A large proportion of students (27%) think that the dot product of two vectors is the magnitude of a vector between the two vectors.In open-ended problems, we found that the most common incorrect reasoning is to relate the scalar nature of the dot product with the magnitude of a vector.We discussed this error in previous articles [10,12].
Item 8 was only answered correctly by 42% of students.A large proportion of students (27%) calculated A⋅B of vectors A=1î+3ĵ and B=5î as 5î+3ĵ.In previous reports based on open-ended problems, we found that these students calculate a dot product of vector 1î (from vector A) and vector 5î (from vector B), incorrectly obtaining the vector 5î and then add the vector 3ĵ (from vector B) to the result.
Item 2 was only answered correctly by 43% of students.In this item the students have to select the graphic representation of a unit vector in the direction of a vector 2î+2ĵ.The most common error (33%) is to select the vector î+ĵ.In an open-ended problem, we found that the students choosing this option believe that this vector has a magnitude of one.Note that we discussed this error in a previous article [11].
Item 17 (Fig. 1) was answered correctly by 54% of students.The most common error (15%, option C) is to state that the direction of the vector is 143.13° (not 126.87°).In open-ended problems we found that these students calculate first a 53.13° angle (as ) and then incorrectly add a 90° angle to this value.Item 13 (Fig. 1) was answered correctly by 56% of students.The most common error (26%) was to select vector -1î+1ĵ (option D).In open-ended problems we found two incorrect procedures: 1) students subtract the two vectors using components, and incorrectly add the y-components of the two vectors, 2) students sketch incorrectly vector -B as vector 2î-2ĵ (collinear to vector A), and add the vectors graphically obtaining vector -1î+1ĵ.Note that this error has not been reported in the literature.

SUMMARY
In this article we show that the TUV is a reliable test with satisfactory discriminatory power since it fulfills the tests recommended by Ding et al. [1].We also analyze the understanding of vector concepts of students finishing a series of introductory physics courses.We focus on the most common errors students make in the five most difficult problems on the test.This is the first analysis in the literature performed with a reliable test.In a future report we will include the entire TUV and present a detailed analysis of each item.

FIGURE 1 .
FIGURE 1. Item 13 (graphical subtraction of vectors in 2D) and item 17 (calculation of direction of a vector written in unitvector notation) of the TUV.

TABLE I .
The ten vector concepts in the TUV, the description of the items, and the proportion of the 423 students finishing a calculus-based Electricity and Magnetism course who selected the correct answer.

TABLE II .
[1]mary of the results of the five statistical tests suggested by Ding et al.[1]for the TUV.

TABLE III .
Most frequent error in the five most difficult items of the TUV.Items are in order of decreasing difficulty.