A Detailed Analysis of Isomorphic Problems : The Case of Vector Problems

The methodology of using two isomorphic (same form) problems focuses on detecting differences in students’ performances when solving one of the problems. In this article we present a methodology that not only focuses on that, but also analyzes the effect on performance when students have previously done the associated problem, as well as the way students’ answers change (“answer changing”) when doing both problems, and the effect of problem order on answer changing. We administered isomorphic tests with vector problems to 330 students finishing a mechanics course. We used two sets of problems: negative scalar multiplication (no-context and velocity context) and dot product interpretation (no-context and work context). In the first set, we detected a significant effect produced by the order in which students answered the problems. In the second set, we found that a great proportion of students answered the first work context problem correctly, but incorrectly answered the second no-context problem. This article presents the methodology so it can be used by other researchers.


INTRODUCTION
Studies based on two isomorphic problems focus on detecting differences in students' performances when doing the problems [1,2].In these studies one problem is administered randomly to half the population and the other problem to the other half.There are several reports of this type in the case of vectors [2][3][4] that focus on differences due to context.However, in these studies there are some issues that are still not well understood (e.g., the effect in students' performance of previously having done the associated problem).In this article we propose a methodology that provides a more complete view of students' performance in two isomorphic problems and covers four objectives: to analyze 1) the context effect, 2) the effect of previously having done the associated problem, 3) the way answers change when doing both problems, and 4) the effect of problem order on answer changing.In this article we focus on the potential of the methodology and illustrate it by analyzing two sets of problems.Note that the effect of previously having done the associated problem and answer changing have been studied [1,3] without the analyses proposed here.

METHODOLOGY
The designed methodology is shown in Figure 1.We randomly divided the population into two halves.
The first half did Exam Version 1, composed of two parts.In the first part, the students answered a nocontext problem (and some other unrelated problems).After turning in this part, they took the second one, which included the physical context problem (and some other problems).The second half did Exam Version 2, in which students took the test in reverse order (the first part with the physical context problem and the second part with the no-context problem).As shown in the figure, the analysis comparing the two exams met the objectives for this study.

TEST ADMINISTRATION
The study was conducted at a Mexican university.Figure 2 shows the statements of the two sets of the

Negative scalar multiplication problems (Exam Versions 1 & 2). Velocity context:
The figure shows a box (seen from above) that moves at velocity v. Choose the option that shows this box moving at velocity -3v.

No-context problem [text only]:
The figure shows vector A. Choose the option that shows vector -3A. -

Negative Scalar Multiplication Problems
This section is divided into four subsections covering each objective.To compare students' performances we used Fisher´s exact test to determine whether the differences were statistically significant, thereby avoiding issues with low cell counts [7].First, we determined whether there was a significant difference in the distribution of answers of the two problems.If there was, we then determined which specific option was significantly different.

Context Effect
To analyze this effect (obj. 1) we compared the students' performances on the problems of the first parts (first and third columns of Table I).We found three significant differences.The no-context problem triggered the selection of the correct option C (87% vs. 51%) and the velocity problem triggered the selection of two incorrect options: E with incorrect magnitude (21% vs. 2%) and B with opposite direction (13% vs. 3%).Therefore, we see that the no-context problem helped students and the velocity problem caused a significant proportion to overlook the correct magnitude and direction of the vector -3v.

Effect of Previously Doing the Associated Problem
In this subsection we cover objective 2. First, we studied the differences between doing the no-context problems with or without having previously done the velocity problem.When comparing the first and fourth columns in Table I, it can be seen that the effect of previously doing the velocity problem decreased the proportion of students who answered correctly (87% vs. 67%) and increased the proportion of students who answered incorrectly (B & E).Therefore, we note that previously doing the velocity problem has a negative influence on the understanding of the no-context problem.
Next we analyzed the differences between doing the velocity problem with or without having previously done the no-context problem.When comparing the third and second columns of Table I, it can be seen that the effect of previously doing the nocontext problem increased the proportion of students who answered correctly (from 51% to 82%) and decreased the proportion of students choosing option E (from 21% to 2%).Therefore, we note that previously doing the no-context problem has a positive influence on the understanding of the negative scalar multiplication of a vector in the velocity problem.

Answer Changing when Doing both Problems
We looked at the amount of answer changing for both versions (obj.3).This can be studied with two analyses.In the first one, which offers a general view, we compared the number of students choosing the different options in each of the problems and detected significant differences.In the second analysis, which offers a more specific view, we studied the different combinations of answers with a cross analysis.
Exam Version 2. In the first analysis we compare the third and fourth column of Table I and find a significant difference in the selection of the correct option (51% vs. 67%).In the second analysis we analyze the cross table of Exam Version 2 (Table II).First we identify the groups of students who correctly answered at least one problem.We note that 45% (gray cell) answered both problems correctly; 22% (green cells) answered the velocity problem incorrectly, but answered the no-context problem correctly; and 5% (blue cells) answered the no-context problem incorrectly, but answered the velocity context correctly.Finally, we also observed that 27% (red cells) answered incorrectly both problems.
We can also talk about the consistency of these students' answers.We note that 64% of them answered both problems the same way (45% correctcorrect and 19% chose the same incorrect option) and 36% changed their answer.We also observe a group (11%) that overlooked the correct magnitude in the velocity context (E-C).
Exam Version 1.In the first analysis we compared the first and second columns of Table I and found no differences.Then we analyze the cross table of this version (Table II).We summarize the significant results: 79% (gray) correct-correct, 3% (green) incorrect-correct, 8% (blue) correct-incorrect, 10% (red) both incorrect and 15% changed their answer.

The Effect of Problem Order on Answer Changing
To analyze this effect (obj. 4) we compare the cross tables.We focus on the proportion of students choosing the correct answer in both problems (79% vs. 45%, gray cells) and in the proportion of students changing their answers (15% vs. 36%).In both cases, we found significant differences.Therefore, the nocontext velocity-context order helped a significant proportion of students in their answers and reduced the changing of answers, whereas the inverse order had the opposite effect on a significant proportion of students and increased the changing of answers.Total 6% 10% 67% 5% 12% 100% Bold percentages = students answering both problems the same way.

Dot Product Problems
In this set of problems, we only found significant differences in answer changing (obj.3).With regard to Objective 1, we didn't find differences due to context.However, it's important to mention that in their first part, a greater proportion of students selected the correct option for the work context problem than for the no-context problem (54% vs. 46%).Regarding Objective 2, we didn't find differences either.This was unexpected, because we thought that students answering the no-context problem after doing the work problem would perform better because they would have more resources.For Objective 3 of Exam Version II (first part: work; second part: no-context), upon first analysis, we did find an unexpected significant difference in choosing the correct answer (option B).Table III shows the cross analysis of this version indicating that 54% answered the first work context problem correctly but only 38% did so in the second no-context problem.An interesting result not evident in the previous set of problems is that the proportion of students answering correctly diminished significantly.To study this decrease, we did the second analysis shown above.We find that 32% of students (gray) answered the problem correctly on both parts and that 22% (blue) answered the work problem correctly but answered the no-context problem incorrectly.Note that in the work problem (Fig. 2), the wording explicitly states that work is defined as the dot product F⋅d.Therefore, we can say that the latter group of students (22%) answered the work problem correctly, not because they understood the geometric interpretation of the dot product, but because they understood the work concept itself.
Comparing this result with the results of Exam Version 2 of the preceding section, we find interesting differences.In Version 2 (Table II), 22% of students revised their first selection and answered the second no-context problem correctly.By contrast, in Version II, 22% changed their answer to an incorrect option, thereby showing an incomplete understanding.This result shows that the methodology allows us to establish the extent of this incomplete understanding for some of the problems.Finally, Table III allows us to make the analyses as in the preceding section.

CONCLUSIONS
Here we focus in the implications of the proposed methodology.Unlike a "traditional" study that would have allowed us to address only the first objective, the analyses performed were able to also address the second, third and fourth objectives of the proposed methodology.This allowed us to establish in a quantitative way some important evidence of students' understanding of two isomorphic problems as shown previously.With regard to the second objective, we can establish the positive or negative influence on students' understanding of doing each of the problems (e.g., no-context problem has a positive influence in the first set of problems).The third objective allows us to characterize students according to their type of understanding in both problems and to analyze the consistency of their answers.It also allows us to classify groups with significant proportions that follow a specific combination of answers.Finally, in addressing the fourth objective, we can determine the order that most benefits the students' understanding and the impact of this order on the consistency of their answers.This evidence of students' understanding has strong research and instructional value.
In this report we present the proposed methodology so it can be used by other researchers.In order to explain the reasons behind these findings in more detail, it will be necessary to combine this methodology with interview-based research.Our research group is currently working on this.

FIGURE 1 .
FIGURE 1. Diagram of the proposed methodology.

2 .
problems (Exam Versions I & II): Work context: The figure shows a force F exerted on a box.The box moves a displacement d.Which option is the best interpretation of the work done by the force F, defining this work as the dot product (F⋅d)?(A) The magnitude of a vector between F and d pointing up to the right.(B) The projection of vector F onto vector d multiplied by the magnitude of vector d.(C) A vector between F and d pointing up to the right.(D) A vector perpendicular to both vectors.(E) A vector in the direction of d.No-context problem [text only]: The figure shows vectors A and B. Which option is the best interpretation of the dot product (A⋅B)?FIGURE Problems used in the study.The no-context problems have the same form and multiple-choice options.

TABLE III .
Cross table of Exam Version II.