Assessing the impact of a computer simulation in conjunction with Tutorials in Introductory Physics on conceptual understanding

This paper compares the effect on student understanding from using either real-world circuits or an interactive circuit simulation. Three groups of students who worked through a tutorial on multiple-loop circuits from Tutorials in Introductory Physics, one with real circuits and two with a simulation, were compared in terms of their conceptual understanding after instruction. Students who used the simulation completed the tutorial faster and had more time for discussion with the TAs, and generally scored higher on conceptual questions than did those who used real circuits.


INTRODUCTION
Many of the research-validated curricula that have been the fruit of physics education research since its inception have not been built around the use of computer simulations.The use of simulations has only recently been introduced and studied in PER-based curricula.Several studies have found that computer simulations can be just as useful, if not more beneficial, than paper-and-pencil activities or standard equipment.[1][2][3][4] Keller et al (2005) found in one study that there was a slight increase in student conceptual performance soon after using a computer simulation with Tutorials in Introductory Physics [5] compared to students with real equipment, but that this effect vanished upon re-examination several weeks later.In a related study, Finkelstein et al (2005) used the same simulation in a laboratory context and found lasting gains in student understanding.
Both of these studies used the simulation in the context of simple circuits, where students were given the basic building blocks and asked to construct various simple circuits to investigate the behavior of resistors, current, and voltage.However, there are no similar investigations of when students move to the next step and are expected to use Kirchhoff's rules to analyze complicated multiple-loop circuits, where it is often more difficult and time-consuming to have them construct the circuits on their own.Another benefit to using simulations instead of real circuit elements is that simulations can be made exact: identical elements in a simulation will behave identically, but no two real bulbs or batteries are in fact identical.If a student measures the voltage drops across each element in a loop, it is unlikely that they will sum exactly to zero.Poor connections, stray resistance in wires, and other sources of error in the real world require additional time to address and explain.Using a simulation then could remove the need to discuss those errors, allowing a student to more readily grasp the important aspects of a circuit.Simulations can also be more efficient since they require less set-up time than does real equipment.[6] Our primary research question is thus: Is a simulation as useful to teach more complex concepts like voltage in many-battery circuits, and Kirchhoff's laws, as real equipment is?
A second question, partially pre-motivated and partially suggested during the course of the study, is: are there temporal differences between simulation and real equipment in learning, i.e. are either of the approaches more time-efficient in teaching the material?
This study addresses these questions by using a computer simulation called the Circuit Construction Kit (CCK), in conjunction with one of the Tutorials on multiple-loop circuits.
We find that students who used the simulation show significant improvement on conceptual questions compared to those who used real circuits.These students also tended to complete the tutorial significantly ahead of those who used real circuits.

Circuit Construction Kit
The CCK is a direct-current interactive simulation developed by the PhET group at the University of Colorado, Boulder.It allows students to easily build simple DC circuits with batteries, wires, resistors, light bulbs, and switches.Students can adjust the values of each component (e.g.resistance of a bulb), and measure current and voltage with a virtual ammeter or voltmeter.An explicit model for current that shows electrons moving through each component can be turned on or off, and light bulbs light up proportional to the current through them.A screenshot of one of the circuits from the tutorial is shown in figure 1.

EXPERIMENTAL DESIGN
The study took place in a large calculus-based introductory physics course at the University of Washington.This course was the second in a threequarter sequence intended for engineering and science majors, covering electricity and magnetism.The course was split into three sections that met for 50 minutes three times a week, two of which were taught by the same instructor.Both instructors covered the same material with the same pacing to make all three sections nearly identical.Students weekly attended a typical two-hour laboratory section, and a 50-minute tutorial section.Each tutorial section had roughly 22 students, which worked on one of the Tutorials each week in groups of 3-5 with the help of two graduate student TAs.Each of the tutorials was given after students have seen all lecture instruction on that topic.An on-line pretest is given each week to students to gauge their initial understanding of the topics to be covered in that week's tutorial During the sixth week of instruction, all three sections worked through the third Tutorial on DC circuits, which focuses on multiple batteries in multiple-loop circuits and the use of Kirchhoff's loop rule.Section A (N=207, "control") worked through the tutorial with real-world equipment: batteries, wires, small light bulbs, and voltmeters.Due to time constraints each of the 9 circuits needed for the tutorial were pre-built for the students, and the TAs were responsible for handing out circuits when each group of students needed them.Sections B (N=161) and C (N=159) used the CCK instead of the real circuits, and again the circuits were pre-built in the simulation for them.Sections A and B were taught by the same instructor.
One of the authors (RH) was present in every tutorial section to ensure that there were no equipment problems with the simulations or physical circuits.He also recorded how long it took each group of students to complete the tutorial, and observed the discussions students had with the TAs.

Data Collection
Data was collected from three exams over the course of the quarter, two as ungraded on-line quizzes and one from a free-response midterm.The first online quiz was administered before the tutorial.Two questions, each about one of the circuits in the tutorial, were asked on the following week's quiz, which students took a few days after they had worked through the tutorial.This served as an immediate post-test of the effect of the simulation in the tutorial.A free-response exam question was asked on the last midterm of the quarter, in week 9. Students were asked to explain their reasoning on each question.
The pre-test used for this study was identical to what had been used for several years.It consists of three circuits from the tutorial, with questions about them that are nearly identical in wording to exercises in the tutorial.The immediate post-test consisted of two questions about circuits that students had seen in the tutorial, one from the beginning of the tutorial and one from the end.Historically this has proven to be a challenging tutorial, which many students struggle to complete by the end of the period.If the simulations are more time-efficient than real equipment, students who use them might have more time at the end of the period to discuss the concepts covered in the tutorial, and might score more highly on a question from the end of the tutorial than do students from the control group.
Figure 2 shows the second question from the immediate post-test.Circuit 1 is the last circuit they see in the tutorial, and circuit 2 (which is identical, but for the orientation of battery B) is taken from the tutorial homework.Students are asked to compare the brightness of bulbs 1 and 3 in each circuit, and then to compare 3 and 4 in the second circuit.The free-response question asked on the last midterm was very similar to the question asked on the immediate post-test.A second battery was added below battery B, with the same orientation, and an identical question about comparing the two bulbs was asked.The questions on the two post-tests were designed to be very similar, to look for the transient improvement that Keller found.
All the questions asked were related to the concepts covered in the tutorial, mostly Kirchhoff's loop rule.Some of the questions in the tutorial homework were also very similar to the experimental questions.As such, all responses were coded by the same investigator, using the same rubric that TAs used to grade the tutorial homework.

RESULTS
The tutorial was given after all lecture instruction, and the free-response question appeared on the third midterm.
All relevant lecture instruction was completed before the second midterm and was tested on the second midterm.To check that the student populations were similar between sections, student scores from the second midterm were compared.The scores for the three sections were not statistically different (p > 0.4), indicating matching samples.
The data is shown in table 1. Statistically significant results (i.e.p < 0.05, using a two-tailed ttest) are in bold.Column 1 shows the average score on the three questions from the pre-test.Group A scored higher than the other two groups, which is surprising since there was no significant difference between the groups on their prior exams.
The first question on the immediate post-test was about a circuit from the beginning of the tutorial, and there was no significant difference between any of the groups.There was a large difference in performance on the second question between the test and control groups, and there was no difference between the test groups, shown in column 2. Column 3 shows the scores from the midterm exam three weeks later.There is again a large difference between the control and test groups and no difference between the test groups.There is also no statistically significant difference in any group's performance between the two exams.
The time that it took students to complete the tutorial differed between the control and test groups.Many of the students in the control group did not finish the tutorial in the time given; in seven of the ten tutorial sections of the control group, no student had finished the tutorial by the end of class.Overall, only 13% of control group students finished on time, and none finished early.In contrast, 41% of test group students finished on time or early.

DISCUSSION
On the immediate post-test, all three groups scored equally well on the question drawn from the beginning of the tutorial, but the control group scored worse on the questions about the last circuit from the tutorial.Students from the control group did not learn the concepts from the end of the tutorial as readily as the test group students did, which makes sense given the fact that students in the test groups got farther through the tutorial than did control students.This suggests that the additional time afforded by the simulation compared to real equipment was of significant value in terms of student learning.
During observations of tutorial sections we noticed that in the control group a large fraction of the TAs' time was spent dealing with the real circuits; handing them out, collecting them, and trouble-shooting them for broken connections.There was not a lot of time for students to discuss their difficulties with the TAs.In the test group tutorial sections the TAs were busy for the first 5-10 minutes of the period setting up the simulation and showing each group how to use it, but after that they were free to listen to student conversations and help them with their conceptual difficulties.
The initial pre-test favored the control group, which makes the conceptual gain of the test groups even more pronounced.The concepts covered in this tutorial are difficult for students, and in each group less than half could give a correct answer and a correct explanation.However, less than half as many of the control students could correctly use Kirchhoff's laws to answer these questions as test students could.The midterm exam question (administered 3 weeks after the immediate post-test) had very similar results to the post-test.There was no significant difference between the performance of any of the groups between the two tests.Keller had found that there was a significant difference (though small) between the group with real equipment and the group with the simulation, but that this difference was transient.We also find a significant difference between these two groups, but our difference is larger and not transient.It should be noted however that the tutorials that Keller was using are more focused on the construction of circuits and the investigation of current and resistance, whereas the tutorial used here did not have students construct circuits, but gave them pre-made ones so they could investigate Kirchhoff's laws.The difference in instructional approach could account for the difference between transient and stable effects, but further investigation is needed to investigate the cause.
For all the questions, most of the incorrect answers in each group were due to treating the battery as a constant current source, which is a well-documented student model.[7] Many students in the control group were using explanations consistent with a constant-current model than were students in the test groups, and very few control students were correctly applying Kirchhoff's laws compared to test students.
The study was designed to make the student experience between the two groups as identical as possible, with the sole exception of real equipment or simulation.
Thus the observed difference in conceptual understanding can only be attributed to the presence of the simulation in the test group.However, that could have been because the simulation was inherently been a better teaching tool than real circuits, or because it afforded more time for students to work on the tutorial and discuss their difficulties with each other and the TAs.
To test this, students in section C were provided with the simulation and the virtual circuits from the tutorial for use as a study aid.There was no significant difference between the two test groups on any question, though, so further access to the simulation did not seem to increase students' conceptual understanding of Kirchhoff's laws.Unfortunately due to a miscommunication between researchers, no qualitative data on whether students in section C used the simulation, or how long they used it, was gathered.A quick informal poll of several tutorial groups suggests however that about half the students used the simulation on their own while working on the tutorial homework.
Also, since both test and control groups did equally well on the question from the beginning of the tutorial, the simulation was not teaching those concepts better or worse than the real circuits.
Thus using a simulation is not inherently a better way to teach these concepts, but its large impact is because students who use it work through the material faster and have more time to work on the parts of the tutorial which were difficult for them.

CONCLUSION
This paper looks at the effect of substituting a computer simulation for real-world equipment in a tutorial about multiple-loop circuits.The first question this study asks is whether using the simulation in this tutorial is as effective than using real-world equipment, as other studies have shown.We see that the groups that used the simulation scored significantly higher on two post-tests than groups that used real circuits, and the conceptual gain due to using the simulation appears to be lasting.
The second question is whether the simulation or real equipment is more time efficient.We find that the groups with the simulation go through the tutorial significantly faster, and a much higher fraction of students complete the tutorial during the class session.This effect is partly due to the fact that with the simulation, students do not need to wait on their TA to bring over circuits or trouble-shoot them.It could also be due to a lessened load on the TAs, allowing them to interact more with the students.That further exposure to the simulation did not improve conceptual understanding supports this idea, so perhaps the most valuable aspect of the simulation is the increase in TA availability.However, more studies are necessary to probe the time effects further, as well as to establish how robust the conceptual gains of the test group over the control group actually are.

FIGURE 1 .
FIGURE 1. Screen shot of CCK, showing the last circuit of the tutorial.

3 FIGURE 2 .
FIGURE 2. Circuits used in the immediate post-test.

TABLE 1 .
Percent of students with correct answer and explanation on each question.Statistically significant results in bold.