Applying analogical reasoning to introductory-level synthesis problems

This study compared the effect of two types of interventions on subsequent student performance with a target synthesis problem. Students either solved two single-concept problems (priming) or compared worked solutions across one of four different analogical reasoning conditions. These four conditions varied according to the type of examples compared (single-concept vs. synthesis) and structural similarity to the target problem. Taken together, the analogical reasoning conditions performed significantly better than control (d=0.31). However, there were no significant differences between the different analogical reasoning conditions, or between analogical reasoning and priming. Although student responses on the target synthesis problem were similar across conditions, their responses to the analogical reasoning tasks showed potentially useful differences in referenced concepts and response grain size, from generic to more precise.


I. INTRODUCTION
We define synthesis problems as problems that explicitly require at least two distinct physics conceptsconcepts potentially separated both in the teaching timeline and across unit boundaries in the textbook [1].As such, synthesis problems have multiple potential bottlenecks; they require not only individual concept mastery, but the ability to recognize multiple concepts and jointly apply them in novel situations more complex than typical end-ofchapter exercises [1].
This investigation seeks to explore the potential of analogical reasoning (AR) as a training tool for synthesis problems.Analogical reasoning is a method of mapping information from a familiar context to a new, analogous situation [2,3].In addition to application in other fields (for example, Ref. [4]), analogical reasoning has been utilized in several studies concerning problem solving in physics [5,6].For this study, we focused on a specific method known as analogical comparisoneliciting student comparisons between two worked exampleswith the intent that students extract the necessary structure to tackle a related target synthesis problem.
There are two main motivations for such an investigation: first, a previous study found that student performance on a synthesis problem can be improved by first priming students with initial questions targeting the relevant physics concepts in isolation [7].Analogical comparison based on worked examples that focus and expand upon the relevant physics concepts may provide further benefits.Second, most of the previous studies employing analogical reasoning in the context of physics have focused on extracting structure from a single example [5,6], rather than comparison between two base problems.
As such, this investigation postulates three main hypotheses: H-1) Analogical reasoning can help improve student performance on quantitative synthesis problems.H-2) Analogical reasoning is more effective than priming.H-3) The effectiveness of an analogical reasoning intervention may be modulated both by the type of worked examples provided and their similarity to the target synthesis problem.

II. EXPERIMENTAL DESIGN
To test these three hypotheses, we designed a synthesis task that would require application of two physics concepts: Ohm's law (in the context of simple circuit analysis) and magnetic fields due to a current carrying wire.This task is shown in Fig. 1C.In addition, four analogical reasoning interventions and a priming treatment were designed.
The analogical reasoning interventions consisted of pairs of worked examples (an example of individual questions are shown in Fig. 1A and 1B) and several short answer questions that sought to focus student attention on key similarities and differences between them.These prompts asked students to identify similarities between the provided solutions (e.g."Consider lines 1-4 in solution one.Which lines in solution two serve the same purpose?Explain your reasoning."),compare solution elements (e.g."Explain any differences between lines 1-4 and your selection in terms of the structure of the two circuits") , and where applicable, drew attention to the use of multiple concepts (e.g."In solution one, why does the student transition from line 6 to line 7?").The final question for each analogical comparison was intended as a summative prompt; the question asked students to create a short guide explaining how to solve similar problems to a friend (a.k.a.create-a-guide question).
The four analogical reasoning treatments were created in a 2x2 design, crossing similarity of the worked examples to the target synthesis task (similar vs. switched) with the type of problems used as worked examples (single-concept vs. synthesis).Similarity to the target synthesis problem was varied by changing what variables were provided and requested.The two near analogical reasoning conditions matched the structure of the target synthesis question: features of the battery or the currents were provided and the total magnetic field was requested.The provided information was reversed for the two "switched" conditions (given a total B-field, a voltage or current was requested).The priming treatment consisted of two single-concept questions selected from the worked examples in the singleconcept analogical reasoning conditions-one involving circuits and one involving magnetic fields.Students solved both independently.The final condition included in the design was a no-training control.
A total of 278 participants were randomly assigned to one of the six conditions, as shown in Table 1.The participants were students enrolled in the second semester of a calculus-based introductory physics sequence at The Ohio State University.The tasks were administered to students in individual carrels in a quiet testing room, as part of a flexible homework assignment for course credit.An equation sheet used for exams in the introductory electricity and magnetism course was provided.
Students first completed a conceptual pre-test, followed by an unrelated physics task, their selected training, and finally the target synthesis problem.Students completed all tasks at their own pace.The analogical reasoning and priming tasks required most students 15-20 minutes to complete.The target synthesis problem required approximately 10 minutes.
After review of a subset of student solutions, a 15 point rubric for the target synthesis problem was agreed upon by two of the authors (RB and DRW).All solutions were independently graded and the scoring rubric for each student was subsequently compared.Initial agreement of scores was 68%, with the vast majority of disagreements consisting of a one point difference in score.Each disagreement was discussed and resolved, leading to the agreed upon scores presented in Table 1.

III. RESULTS AND DISCUSSION
In order to test the hypothesis that training with analogical reasoning (AR) is effective for synthesis problems (H-1), an ANCOVA was conducted between the combined AR treatments and control, using pre-test score and final course grade as covariates in order to account for differences in student ability.Together, the AR conditions performed better than control, F(1,227) = 4.593, p = 0.033, with a small-to-moderate effect size, d = 0.31.For comparison, a similar ANCOVA analysis showed priming had a smaller effect size (d = 0.22), and, with a smaller sample size, was not significantly better than control, F(1, 87) = 0.990, p = 0.323.
To directly compare AR training vs. priming (H-2), an ANCOVA was conducted between the combined AR treatments and the priming treatment.There was no significant difference between AR and priming after controlling for pre-test score and final course grade, F(1,230) = 0.652, p = 0.420.
Finally, a 2x2 ANCOVA with final course grade and pre-test score as covariates was conducted between the four AR conditions to test whether the similarity to the target synthesis problem (similar vs. switched) or the question Overall, the AR interventions had a small-to-moderate positive effect compared to control on student performance with the target synthesis problem, but a direct comparison between treatment conditions showed there was no difference between the combined AR conditions and priming, or between the four different AR conditions.

A. Solution Bottlenecks & Common Errors
An analysis of solution bottlenecks and difficulties with the target synthesis task suggest several potential reasons for the similar effectiveness of AR and priming.First and foremost, most students were successfully able to identify the correct physics concepts required by the target synthesis problem.This is different from previous studies where concept recognition was one of the driving factors of student performance with a synthesis task [7].As such, it appears that one major potential advantage of analogical reasoningfacilitation of students' recognizing pertinent conceptsis not manifested in this case as many students are already capable of doing so without intervention.
Although concept recognition was not a significant bottleneck, Table 2 suggests several common failure points that did hinder many students from correctly solving the problem.First, a significant portion of students failed to include all 3 relevant B-fields (approximately 30% across conditions).Slightly over a third of the students included all relevant fields, matched them to the correct physical current, and correctly solved for the component currents.The mismatch of individual currents to their corresponding magnetic field is a particularly interesting failure point.Approximately 10% of the students across all conditions made the serious conceptual error of finding the total current through the battery, and then using that current for calculating all of their identified B-fields.In this case, the main synthesis failure is students' inability to identify that unique-and-initially-unknown currents within the circuit each contribute to the total magnetic field.As such, this failure point provides some evidence that the failure modes of synthesis problems can extend beyond a product of difficulties found in simpler single-concept problemshere the complexity of the physical situation itself may represent an obstacle for students attempting to combine the two physics concepts.
As shown in Table 3, AR and priming treatments both proved ineffective in helping students address this difficulty compared to control, suggesting that this may be a useful idea to target explicitly in future implementations.In addition, AR and priming were equally effective at bringing attention to common minor mistakes (for example, missing the factor of ½ in formulating the field due to a half-circular arc or completely ignoring the magnetic field direction).Other errors, such as incorrectly assigning a B-field direction through a misapplication of the right-hand-rulea skill addressed only briefly in the analogical training conditionspersist equally across all conditions.

B. Analogical Reasoning & Grain Size
In addition to comparing the relative effectiveness of the different AR conditions on the target synthesis problem, it is worthwhile to compare how the students responded to the comparison prompts.The most interesting result involves the create-a-guide question included at the end of all four of the different analogical reasoning tasks.The students in the single-concept conditions were far more likely to include specific information and solution steps compared to those in the synthesis conditions.
To illustrate, Fig. 2 shows the percentage of students who included a subset of the potentially relevant components related to Ohm's law and circuits.Identifying parallel/series structure and applicable rules for such cases, reducing the circuit to equivalent resistances, and explicit citation of Ohm's law are more fine grained descriptions, whereas implicitly using Ohm's law (i.e."use the given voltage to solve for the currents") and procedural steps (e.g.identifying unknowns) are more course-grained.Whereas students in the single-concept conditions were more likely to specifically cite Ohm's law and identify series/parallel circuit structure, students in the synthesis condition discussed the concepts more broadly.The difference between the conditions is likely due in large part to student's expectations of the taskregardless of condition, students would respond to the prompt with a guide of approximately 3-4 unique steps.Since the prompt was included twice in the single-concept conditions (once for each concept-pair), it is reasonable that those students would include more detailed steps overall.
However, students in the synthesis conditions have at least two a priori options: respond with a subset of the important steps at the same grain level (due to availability, cue strength, etc.) or adjust grain level and describe the synthesis problem more generically.Almost invariably students in the synthesis conditions provided guides based on the latter.Although this difference did not seem to introduce a significant effect in performance on the target synthesis question, it may be that analogical comparison using synthesis problems provides an authentic way for students to practice recognizing relevant concepts and planning their application.On the other hand, it seems that single-concept examples might be more suited to highlighting the nuances necessary for concept mastery.

IV. CONCLUSION
Overall, these preliminary results are encouraging: students trained via analogical reasoning showed a measurable improvement on a complex problem solving task (d = 0.31), after spending only about 15-20 minutes with training.Perhaps even more promising, these performance gains occurred with a one-time exposure on a novel taskwithout any feedback to the student beyond the analogical reasoning prompts themselves.As such, this may suggest a potential implementation for analogical reasoning interventions via small groups, where feedback and practice could help reinforce the effectiveness of the analogical reasoning prompts.
However, these results come with several important caveats: first, given the similarity in performance between priming and the analogical reasoning tasks, it is currently unclear if the gains from the priming and analogical reasoning conditions are due to the same effect or two different underlying mechanisms.If the benefit from analogical comparison extends beyond just priming the relevant concepts, either by helping students extract relevant structure or some other advantageous mechanism, an increase in time between intervention and target assessment might introduce notable differences between the guided analogical reasoning tasks and priming.Whereas priming effects might decay away rather rapidly, it is possible that the improvement gained from analogical comparison would be more robust and persist even with a longer delay.Future studies will seek to disentangle these two potential effects.
The second caveat is that the current study only focused on a specific pair of physics concepts.We might expect that getting students to reflect on the underlying structure of synthesis problems could be broadly applicable to any synthesis problem, but we have not explored such potential for transfer.Future work can examine that possibility.
Finally, it is possible that the current target synthesis problem underestimates some of the potential advantages of analogical comparison.In this case, almost all students were able to recognize the relevant physics concepts.There may be further benefits with synthesis problems where concept recognition is not as apparent, either due to the complexity of the concepts involved or the existence of multiple competing conceptual cues.In such cases, analogical reasoning prompts, paired with worked synthesis and single-concept examples, may provide complementary ways to get students to reflect on different problem solving steps and conceptual applications.

FIGURE 1 .
FIGURE 1.An example of a single-concept problem (A) and a synthesis problem (B) provided as worked examples (solutions omitted), and the target synthesis problem (C).Resistance values included in the original figures are not shown due to limited space.To keep time-on-task as similar as possible across conditions, the analogical reasoning conditions based on synthesis problems included a comparison of only one pair of worked examples.Students in the single-concept conditions compared two pairs of worked examples: a pair of circuit problems and a pair of magnetic field problems.The priming treatment consisted of two single-concept questions selected from the worked examples in the singleconcept analogical reasoning conditions-one involving circuits and one involving magnetic fields.Students solved both independently.The final condition included in the design was a no-training control.A total of 278 participants were randomly assigned to one of the six conditions, as shown in Table1.The participants were students enrolled in the second semester of a calculus-based introductory physics sequence at The Ohio State University.The tasks were administered to students in individual carrels in a quiet testing room, as part of a flexible homework assignment for course credit.An equation sheet used for exams in the introductory electricity and magnetism course was provided.

A.TABLE 1 .
Consider the current carrying wire shown.Given r 1 = 0.5m, r 2 = 2.0 m, and I=0.25 A, find the magnitude and direction of the magnetic field at point P. B. Consider the circuit shown.Given that the battery provides 9.0V, r 1 =0.5m, r 2 =0.75m, and r 3 =1.25mfind the direction and magnitude of the magnetic field at P. C. Consider the circuit shown, where r 1 =0.25m and l 1 =l 2 =0.5m.The battery voltage is 25V, and the horizontal wires are very long.Calculate the magnitude and direction of the magnetic field at point P. Mean score on the target synthesis problem out of a maximum score of 15 points.Errors shown are standard errors.Combined AR represents the mean of the four AR conditions.vs. synthesis) used during training influenced treatment effectiveness (H-3).There was no significant main effect of either similarity, F(1,181) = 0.336, p = 0.563 or question type, F(1,181) = 1.645, p = 0.201.

FIGURE 2 .
FIGURE 2. Comparison of circuit related concepts/facts present in student responses to the create-a-guide prompt.

TABLE 2 .
Percentage of students successfully accomplishing key synthesis problem-solving steps.

TABLE 3 .
Percentage of students making selected errors in target synthesis problem.