Epistemic Games Analysis of Common Exam Questions Across Course Formats

: This study investigates differences in problem-solving performance between three different introductory physics course formats at Florida International University. The course formats—lecture+laboratory (LL), inquiry-based (IQB), and lecture+laboratory+recitation (LLR)—all incorporated two Advanced Placement (AP) questions into their final exams. Students’ written responses were evaluated via an AP scoring rubric, and during this scoring, we observed marked differences in solution behavior between the three course formats. To further investigate these differences, we used the framework of epistemic games to analyze student responses. To apply this framework to written work, an epistemic game rubric was created. Application of this rubric yielded game profiles for each of the course formats, allowing us to highlight and compare course characteristics. These profiles of epistemic game distributions were then examined via chi-squared tests to quantify differences in the tools and strategies students used in their solutions.


INTRODUCTION
Epistemic forms are structures for organizing knowledge used both by practicing scientists and by students in the classroom [1]. To create these forms, we invoke epistemic games, which are a set of rules and strategies that guide inquiry [2,1]. This framework of epistemic forms and games has been used to study student problem solving in physics [2]. A basic example of forms and games would be making a list. When making a list, the epistemic form would be the list itself. The way we construct the list, by adding, subtracting, or grouping items, is the epistemic game [1].
In the present study, we use epistemic games to understand how students from three undergraduate course formats at Florida International University (FIU) solved a particular physics problem on their final exam. The three course formats consisted of traditional lecture+laboratory (LL), inquiry-based (IQB), and lecture+laboratory+recitation (LLR).
Two sections of LL, with 85 total students, reflect a classic learning environment where the professor gives the students the majority of their instruction straight from the board. The three IQB sections, with 91 students, are studio-classroom learning environments where students engage in small group activities and whiteboarding sessions. LLR, with a single section of 117 students, takes half the lecture time of the LL and uses it for recitation sections run by teaching assistants and undergraduate Learning Assistants [3].
To understand how the students from these course formats solved problems, the six instructors agreed to incorporate two advanced placement (AP) questions into their final exams, one of which is discussed here (for preliminary results on the other see reference [4]). The student responses were scored according to the AP rubric. The results showed statistically significant differences in mean scores between the course formats.
The written student responses, which would be normally graded, represent an organization of their thoughts and concepts into the target structure of a solution, thus reflecting an epistemic form. The steps used to create this epistemic form would be the epistemic game or games played by the student. Originally the idea of epistemic games and forms were used in physics education research with video data of student problem solving. [2] For this analysis, we adapted the Tuminaro and Redish set of epistemic games into a format that could be applied to written exam data.

THE SPRING PROBLEM
To examine the student problem-solving process in more depth, we selected the more challenging of the two problems for this analysis. We chose a subpart of that problem where students exhibited a wide range of solution strategies.

Prompt:
The student now attached an object of unknown mass m to the cord and holds the object adjacent to the point at which the top of the cord is tied to the rod, as represented above. When the object is released from rest, it falls 1.5 m before stopping and turning around. Assume that air resistance is negligible. Question: Calculate the value of the unknown mass m of the object. This question comes as a subpart of a much larger problem. At this point, students have calculated values for the spring constant k, max height y max , and spring elongation x. Students then should use conservation of energy as in equation 1: (1) With some minor algebra, students can then solve for the mass of the unknown object. This sub-problem invokes a change from the previous sub-problem by adding an additional force to counter that of gravity. With the addition of this force, the students should identify that energy is the easiest way to solve the problem.

METHODS
We wish to identify the epistemic game (or games) that a student played in the course of writing their solution. Considering each student's written answer as their epistemic form, we modified the six epistemic games, seen in table 1, from Tuminaro & Redish [2] to create a rubric that would allow us to apply the epistemic games to the student responses. During the course of our analysis, a seventh epistemic game was published by Chen et al. [5]. However, the format of the students' answers prevented it from being included in our rubric. Table 2 shows an example (only game 1) of the finished rubric. Each epistemic game is broken down into several steps as in Tuminaro & Redish, and each step is coded as absent or present in the written work.
Via the rubric, we can detect elements of the games based on the student's responses. Each of the epistemic games was broken up this way leading our final rubric to contain six games.
To ensure we had a robust rubric, it was subjected to inter-rater reliability (IRR). Thirty students' solutions were randomly selected from the three course formats; two researchers for each epistemic game applied the rubric to the question. The scores from the graders were correlated and were found to have an R-value of 0.93. The rubric was then used on the remainder of the exams, allowing us to attain game profiles for each of the course formats.

RESULTS
The epistemic games rubric was applied to each of the 293 students. From the rubric we were able to detect elements of the epistemic games and create "game profiles" for each of the course types.
Game profiles, seen in Table 3, show the percentage of students within a course format who played a particular epistemic game. The students are counted as playing an epistemic game if they have used more than 50% of the steps in that game in their solution. For example, if the student was counted as playing the Mapping Meaning to Mathematics game, the student would have to have shown evidence of working at least through step 3, relating mathematical entities to the physical story. From Table 3, we have found that students in LL predominantly use game 5, Recursive Plug and Chug, implying that for this problem they strive to quickly find an equation that utilizes their known variables and attempt to solve for the unknown. In this situation [6], the students focus on the surface of the problem, and this game is often unsuccessful because the problem implicitly requires the students to develop an energy approach. For example, if the student selects the classic F=ma=-kx they may believe they have all of the variables they need to solve for the mass; however, they are not accounting for everything in the problem (substituting g for the acceleration, usually part of such solutions, is inappropriate in this case).
Staying within LL, the next most popular was game six,, Transliteration to Mathematics. This game has students tap into the previous subpart of the problem as a template or example. Using this game will cause some issues with the student's logic, due to the fact that the prior subpart focuses on just the force of gravity, while the stretching cord introduces non-constant acceleration. Students may invoke a solution pattern similar to that of game five listed above.
Less commonly, students in LL use games one and two (Mapping Meaning to Mathematics and Mapping Mathematics to Meaning). These games are story and concept focused respectively, and these factors may help students to understand that the problem has to deal with a balance of forces (and thus energy provides a simpler solution). These games more often lead to a sound answer that mimics the AP expected response.
IQB students have a very broad selection of games with the roughly 40% of students showing evidence of games two and six respectively. This may seem problematic due to the pitfalls of game six; however, the IQB students seem to begin by referencing portions in the previous problem but then identify that there is more then one force at play. They then apply steps from game two filling in the gap and progressing forward to a sound solution. IQB students also seldom limited themselves to the "standard" solving space; a number of IQB students attached multiple pages to solve this problem.
Ten percent of the IQB students used games three and four. These games are picture-based games that have the students draw out what happened and in some cases pair the drawings with a logical story. This bleeds into the logic of games one and two stated above. With 40% of the IQB students playing multiple games, it was common to see students solving a problem one way, finding holes in logic, and then working their way back to correct and move forward. This flexibility with representational formats also suggests a higher level of expertise in these solutions. [6] The LLR students utilized games five (40%) and six (49%) with a few that explored different options. All of the games have been discussed above and their applications to this problem speak to how the LLR students approached this problem.

ANALYSIS
In evaluating these results we will take two paths: profile distributions between course formats, and the difference in number of students playing multiple games.
In Table 3, the individual course format percentages add up to more than 100%. This is due to the fact that within each of the course formats we found students with enough written evidence to support playing more than one epistemic game. Table 4 shows the percentage of students exhibiting evidence for multiple games, a value that markedly varies among the course formats. Having the ability to invoke multiple games will give a student more tools to solve the problem, a point developed further in forthcoming work. From Table 3, we see that student responses did not distribute equally among the six available games. To quantify this impression, we performed chi-squared tests on the profiles of the game counts to mitigate student variability. To do this, we used the LL as our expected values, making it our baseline.  Table 5 shows the chi-squared value for the comparisons of the other two formats to LL. In both cases, the differences were statistically significant. Thus, we can say that the distributions are not similar among course formats.

DISCUSSION
We present this analysis as a demonstration of the utility of the epistemic games framework for characterizing written student solutions. From this characterization we have found interesting differences between the three course formats and how the epistemic forms and games apply to the selected sub-problem.
Based on the game profiles, we have found that LL, our standard, shows heavy usage of game 5 with clusters of games 1, 2, and 6. This implies that their students are equation driven and that they reflect on the previous problems worked before constructing an answer or moving on to another concept. IQB, our studio style, shows a full gamut of games played with heavy usage of games 2 and 6.These students also show the highest chance of playing more then one game. This implies that when attempting to solve this problem the IQB students may have been reflecting on the previous problem, however, through their story forged from game 2 or inputs from the other games, the IQB students have more tools to solve the problem.
LLR, 50% lecture and 50% recitation problem solving, shows heavy usage in games 5 and 6 with small amounts of other games played. LL also has the lowest percentage of students invoking multiple games. This suggests that these students are more "one track", most likely plugging and chugging or looking back to previous work to formulate their answer. At the moment it is not clear as to why we see these trends in LLR, however, it may be due to what students do during the recitation sections.
The fact that games 3 and 4 were rarely invoked speaks to the format of the problem. Since this problem required a numerical answer, these games, being story-and picture-focused, were not an optimal choice for answer building.
One obvious question that stems from the different game distributions is if certain games lead to a correct solution more often, and how selection of games or facility with multiple games connects to expertise.
The next step of this analysis, connecting the games and number of games played to score, is in progress. Other questions for the future pertain to how the most favorable games change with the problem context, and if game-playing behavior by course type is consistent across different problems.