Assessing the efﬁcacy of a new online game and simulation to teach electric ﬁelds

,


I. INTRODUCTION
The abstract nature of electric fields makes them a notoriously difficult topic of study for introductory physics students. Students struggle to make connections between electric field lines and electric force vectors [20], understand the superposition principle in the context of electric fields [2], and determine the trajectory of a test charge in the presence of an electric field [2,20].
Electric fields are not physically observable or tactile, making them difficult for students to comprehend [18].
Research-based technological tools which have been developed to help students learn about electric fields in a more interactive way include, but are not limited to, several PhET Interactive Simulations and a series of Physlets. A study by Smith et al. assessed how the use of virtual reality impacted student understanding of electric fields. They found that students trained in using the technology showed higher gains and more positive attitudes toward the technology than did those who were not trained [16]. In a study by the PhET developers that focused on best practices using educational simulations, it was found that students are less likely to explore all the features of a simulation when given a heavy amount of guidance in the form of a written worksheet [4]. These results align with the purposeful design of the PhET simulations "to guide, but not excessively constrain" exploration [5].
While both commercial and educational games have been used in physics classes [12,15,17,22], studying the impact of such games on students' understanding is a budding field [1,3,19,24].
The purpose of the current study is to address some of the learning difficulties associated with electric fields by assessing the efficacy of two learning tools currently in development -an online simulation and a game -which have been designed and developed by undergraduate researchers to help students learn about this topic. The simulation and game are easily accessible from any internet browser via our electric field simulation and game landing page. The research questions are as follows: To what extent do these learning activities (a video, a simulation, and an instructional game) impact students' ...
1. understanding of electric fields 2. and their technological anxiety?

II. STUDY DESIGN
The participants in this study are primarily (83%) life science majors in their second-semester of an introductory algebra-based physics course at a medium-sized, primarily undergraduate institution. The class is composed of 6% firstyears, 53% second-years, 26% third-years, and 15% fourthyears. The entire course took place online via Zoom conferencing. Participation in the study was entirely voluntary, but all students were asked to engage with the activities as part of the class requirement. The sample of 85 students who consented to participate in this study was 74% female and 26% male. All treatments took place during a single twohour class session. We use the term "treatment" to denote the activity (video, simulation, or game) whose impact has been analyzed.
In preparation for the study, students were asked to read a section in their textbook about electric fields and complete a pre-treatment diagnostic. On the day of the study, each student was randomly assigned to one of three breakout rooms in the Zoom meeting, each utilizing a different learning tool to study electric field content. As shown in table I, the "video room," "simulation room," and "game room" initially received a short video playlist, the simulation, and the game, respectively. Along with their assigned learning tool, all students received identical digital worksheets which they were instructed to complete as they used their learning tool. This was meant to provide light guidance for interacting with each tool. The students in each room were given 25 minutes to interact with their assigned learning tool, then 13 minutes to complete a post-treatment diagnostic. Students completed all activities independently. Students in each breakout room then interacted with the remaining two learning tools for 18 minutes each. Then, the students were guided through the solutions to the worksheet. Finally, all students received a brief survey consisting of questions that would provide the research team with feedback on the game and simulation, as well as students' learning tool preferences.

A. The Video, Simulation, and Game
The intent of the video playlist was to mimic traditional lecture-based instruction. The video playlist consists of 5 videos, averaging about 3.5 minutes each, that introduced electric and gravitational fields, electric fields due to point charges, electric field strength, and electric fields due to multiple charges.
The electric field simulation was written using JavaScript and the visualization library p5 [14]. It has a sandbox-like design in that there are no implicit instructions built into the simulation itself. Rather, students are given a space in which they can add, remove, and manipulate charges while toggling on and off different visualization settings to observe the impact of their changes, as shown in Fig. 1. For example, students can increase and decrease the magnitude of charges, change the sign of charges, and observe the field lines, field vectors, and equipotential lines resulting from their charge configurations. Students also have the ability to add test charges to their configurations and observe their trajectories.
The development of the simulation was initially motivated by the common misconception that the trajectories of test charges align directly with electric field lines. Furthermore, differences between electric field lines and electric field vectors tend to be a stumbling point for introductory-level students. The simulation (paired with the activity worksheet) allows students to directly address these misconceptions and observe the impact of dynamic positioning of electric charges on the resulting electric field.
The electric field game, Dynamic Electric Field Interactive (DEFI), was written using Javascript and p5. The goal, as shown in Fig. 2, is for the player to create a net electric field using a charge configuration that results in a test charge traveling within the boundaries of a given track to reach a finish line. The player's score is determined by the time it takes the player to successfully reach the finish line and by the number of stars collected in the path of the test charge. Utilizing game design theory [10,11,13] and taking advantage of zones of proximal development [21], the tracks become more intricate as the player progresses through each level. Brief instructions about how to play the game were directly integrated into the game. The game's development arose out of a need to keep introductory-level students (particularly non-physics majors) engaged in material related to electric fields, while also addressing some of the common misconceptions described above. For example, students can directly see, while placing fixed charges, the impact of the resulting electric field and the trajectory of a test charge through the space.
Both the game and simulation were designed to be used with an accompanying worksheet in an active learning environment in which students are encouraged to explore electric fields with minimal instructor guidance, like the learning environment suggested by PhET developers and facilitators [5].

III. DIAGNOSTIC AND SURVEY
Due to the nature of this study, traditional E&M diagnostics (like BEMA and DIRECT) could not fully assess the narrow focus of our learning objectives. Instead, a diagnostic consisting of seven questions that emphasize the conceptual nature of electric fields was compiled from "Con-cepTest" assessments accompanying popular texts and The Physics Classroom website [6,7,9,20,23]. For example, the diagnostic gave students an electric field diagram and asked students to determine the signs of the charges and their relative magnitudes. Questions regarding student beliefs about technology were adapted from the Computer Anxiety Rating Scale (CARS) [8], used to assess anxiety about technology. The original CARS instrument is shown to be a reliable and valid measure of participants' anxiety about computers while using them.
After interacting with all three learning tools, the students were given three additional survey questions so they could provide suggestions about how the research team could improve the simulation and game and offer their most preferred learning tool.

A. Student Understanding
To assess how effectively each tool impacted student understanding, we analyzed their raw scores on the content questions in the pre-and post-diagnostic. We find that there are gains in understanding across all three rooms, with the students in the video room showing the greatest average gain (see Fig. 3). An ANOVA test was conducted and Cohen's d effect sizes are presented. The difference in raw score is statistically significant for students in the video room, simulation room, and game room with probability p < 0.0001 for all rooms, and effect sizes d = −2.06, −1.74, −2.11 respectively. The students in the video room performed higher than those in the simulation (p = 0.054, d = 0.507) and game (p = 0.0698, d = 0.504) rooms on the pre-treatment diagnostic. These data provide support for comparison of normalized average gain (see Fig. 4). Fig. 4 reveals that the average normalized gain is not significantly different between any two rooms. The difference in average normalized gain gives p = 0.208 and d = 0.326 between video and simulation rooms, p = 0.216 and d = 0.330 between video and game rooms, and p = 0.898 and d = 0.034 between simulation and game rooms. The authors recognize that the pre-treatment score was higher for the video stream room, but wish to emphasize that the groups were randomly selected.

B. Student Beliefs
The results shown in Fig. 5 correspond to students' responses to the questions in the diagnostic sampled from the CARS instrument. We find that the difference in overall percentage of favorable responses from pre-to post-diagnostic is not statistically significant for any of the three treatment groups. It is, however, worth noting that students in the simu- lation and game rooms experienced a non-statistically significant increase (demonstrated by decrease in comfort) in technological anxiety after interacting with their assigned learning tool. We hypothesize that this is because these learning tools required the most interaction from the students with the least amount of explicit guidance. The students in the video room were assigned a more passive (and likely less anxietyinducing) role regarding their interactions with technology. Additionally, the students were most familiar with the video format because of their recent experience with asynchronous learning during the pandemic. This is further supported by a lack of any significant change in favorable responses by students in the video room. These results are consistent with those regarding the positive correlation between technological training and student attitudes toward said technology found in the Smith et al. study [16].

C. Student Preference and Feedback
After interacting with all three learning tools, the students were asked which tool they most preferred using to learn about electric fields. Figure 7 shows a breakdown of these  responses based on the initial room of each student. It is evident that the majority of students in this study, regardless of initial room assignment, prefer learning about electric fields by watching an instructional video. The students were asked to provide suggestions to the research team in order to help improve both tools, and upvote any statements added by their classmates with which they agreed. The responses were coded using inter-rater reliability (IRR). The codebook contains tags with which the team used to code each student's suggestions. Student responses containing more than one suggestion were separated into multiple responses and tagged individually. On the first measure, the IRR was 88%. After discussion and clarification of some definitions, the IRR was 100%. Table III shows the types of suggestions that emerged from student responses in the form of tags, a count of the frequency each tag was logged, and upvotes.
From Table III, we see that many students would have appreciated more thorough instructions prior to utilizing the simulation and game. They shared that they felt confused at first and unsure of how to begin. Additionally, as indicated by the "existing features" tag, they also offered suggestions for additional features, which students were unaware were already present. These results are consistent with the findings in Porter et al. that showed the benefits of training students in educational technologies prior to beginning an assigned activity for which they are expected to use it to learn [16].
At the time of the study, the game had a brief set of text instructions with diagrams. However, feedback indicated that the instructions must be clarified and enhanced with a visual introduction to the game. Some students specifically suggested a video tutorial and/or a demo mode.

V. CONCLUSIONS AND DISCUSSION
Students' understanding of electric fields improved with all three treatments, with students in the video group demonstrating the greatest improvement. However, there were no statistically significant differences in the three groups' average normalized gain. This possibly indicates that all three of the treatments facilitated students' learning of the material at hand.
Students using the simulation and game experienced a non-statistically significant increase in technological anxiety while performing their activity, while students that were asked to watch the video did not. This is consistent with the student preference towards the video over the other two treatments, and with the largest performance gains on a test of their understanding.
The research group has many hypotheses for the video providing the largest understanding shifts, the strongest preference and the corresponding lack of anxiety. These results could be related to students' present familiarity and comfort with the video medium. Additionally, they may be due to a need for stronger step-by-step guidance in the simulation and game activities, and may indicate the need for an activity which strikes a different balance between exploration and guidance. These results may also indicate that passive activities with little required engagement induce less anxiety -technological or otherwise. Nevertheless, students' strong preference for the video was surprising, and contradicts the seemingly ubiquitous distaste for online instruction that became particularly widespread during the pandemic.
Thus, while differences in students' shifts in understanding were not statistically significant, students favor the instructional video. The effect sizes discussed in section IV. A indicate that there is a small to medium effect in favor of the videos over other tools, perhaps suggesting that the video lecture is preferred. It is also worth noting that each treatment resulted in significant gains indicating the value of each treatment independent of the other treatments. The results of this study underscore the need for further investigation into how these demographics (primarily female, non-physics majors) and combinations of treatments (for example, providing the video then the game) affect students. The authors are also currently developing a tutorial mode in the simulation and game, as well as an accompanying activity worksheet to address the most common concerns about anxiety and lack of instructional guidance raised by students.