A case of productive conﬁrmation framing in an introductory lab

Students’ framing of an activity – their understanding of “what is it that’s going on here” [1] – shapes how they act, think, and learn. Prior research suggests that framing instructional physics laboratory activities as conﬁrming known results is problematic for learning [2, 3]. Here, we complicate those ﬁndings by presenting a case-study of students who exhibit conﬁrmation framing as they engage in productive behavior. In this case, data that are inconsistent with the theoretical model of the lab motivate a genuine problem for the three students, who troubleshoot their apparatus and analyze their data to construct an explanation for this anomaly. We claim that their productive behavior is supported by their conﬁrmation framing; put another way, we claim that their conﬁrmation framing engenders their productive behavior: the students seek to explain how they could have caused this error. The case-study reported on here is part of a larger project studying student behavior in non-traditional physics labs.


I. INTRODUCTION
National data on students' epistemological views of experimental science indicate that a majority of students in first-year courses expect the purpose of physics experiments to be confirming known results [4][5][6]. While a follow-up study by Hu et al. [7] indicates that students who expected to confirm previously known results see this work as valuable for learning, RF Lippman [3] found that students with these expectations found lab to be "useless," often attributing any discrepant data to human error without subsequent investigations. Another recent study suggests that confirmatory expectations are problematic for learning in lab, connecting them with questionable research practices [2] -students would disregard or manipulate their data to confirm the provided theoretical model.
That said, Smith, Stein, and Holmes [2] qualify their analysis: "Our data do not allow us to claim that confirmatory expectations necessarily lead students to engage in questionable research practices. For example, not confirming an expected result may suggest an error was made, and productively send the student into a troubleshooting mode" (p. 13) In this paper, we present an episode of productive "troubleshooting mode" from students' work in an introductory lab during the Spring 2021 semester. This episode complicates the notion that confirmatory expectations are problematic for learning.
Next, in section II we overview the key theoretical constructs that guide our analysis and inform our sense of productive behavior. In section III we describe the context of this data and the data collection methods. Then, in section IV we present and interpret the case. We conclude in section V, discussing the implications for future research and lab instruction.

II. THEORETICAL CONSTRUCTS
Like earlier work [2,3], our research is organized by the construct of framing. Framing was originally developed in anthropology and linguistics to describe how an individual or a group interpret what is taking place [8]. Framing an event shapes one's understanding of what could happen, what features of the event require attention, and what qualifies as appropriate action; to frame an event is to tacitly answer the question "what is it that's going on here" [1]. It is a dynamic process that is co-constructed and negotiated between different participants through meta-communicative signals in moment-to-moment interactions [9][10][11]. How students frame * Please address correspondence to ian.descamps@tufts.com an activity structures their expectations, shaping how they engage with and learn from activities [12][13][14]. For labs, students have had many experiences that were about confirming known results, which may structure their expectations for future empirical work [2].
Our sense of what constitutes productive behavior is informed by the long-standing goal of science education research and reform to engage students in "doing science" themselves [15][16][17]. Empowering students to construct knowledge in disciplinary ways can enable them to develop both deep understandings of phenomena [18][19][20] and an awareness of scientific norms and practices [21][22][23][24]. Despite the goals of researchers and education reform, data indicates that it is rare to see students doing science in classrooms [25].
The data presented in this paper come from a larger study on student behavior in non-traditional physics labs. This larger study aims to further our understanding of what supports and disrupts sensemaking in introductory, instructional laboratories. We take sensemaking to be "a dynamic process of building or revising an explanation in order to 'figure something out' -to ascertain the mechanism underlying a phenomenon in order to resolve a gap or inconsistency in one's understanding" [26]. This study will extend prior work on motivating and stabilizing factors for sensemaking [27].
In addition to sensemaking, problematizing is another theoretical construct that informs our interpretation of this case study. We define problematizing as the "work of identifying, articulating, and motivating the problem that needs solving" [24,28]. Phillips, Watkins, and Hammer [24,28] persuasively argue that problematizing is a significant activity in scientific inquiry. Further, Engle and Conant propose problematizing as a key feature of learning environments that support Productive Disciplinary Engagement [29] -students asking questions and challenging content can enable to them to construct their own knowledge, potentially in disciplinarily authentic ways.

III. COURSE CONTEXT AND DATA COLLECTION
This case study comes from an introductory physics lab in a large, four-year research university in the northeastern United States. These labs are intended to promote student agency in designing experimental methods and drawing their own conclusions. There are four lab activities a semester with each activity lasting three classes; in the first two classes, students' work consists of normal lab experimentation, while the third class is an extended whole group discussion where students present their work.
Our data come from spring of 2021 when the lab was operating in a hybrid format due to COVID-19. Students worked in groups of two or three, with some students in-person and others virtual. In this episode, two of the students, Peter and Holly, are in-person at the same lab table; their other lab partner, Judy, is virtual (these names are pseudonyms). They are in their second week of the first lab activity, which asked stu-dents to "design and carry out an experiment to investigate and test the predictions of Galileo" as precisely as they are able. The instructors anticipated students would find evidence against one of Galileo's claims, that the period of a pendulum does not depend on amplitude. Part of the goal was to promote students' framing lab as about what their own evidence supports.
With the hybrid modality of labs, we collected data by having students record their group video calls and the TA record the virtual whole class discussion. We also had a camera and audio recorder in the lab room. The first author watched the videos, logging student activity in 5-minute intervals and taking notes. Then they selected candidate episodes for workshopping [30] with the whole research team. This selection was based on prior literature on sensemaking and scientific inquiry; indicators included mechanistic reasoning [31,32], vexation or problematizing [24,27,28], not-understanding or uncertainty [33,34], and argumentation [35,36].
The first author prepared analytical memos [37,38] to present, revising based on feedback and discussion during workshopping sessions. Throughout, the process drew on discourse analysis [8], interaction analysis [39], and video research methods from the learning sciences [40]. For the larger study, we plan to compare across the data for patterns and themes [30] of various factors that contribute to sensemaking. In this paper, we present one episode that stood out as surprising and puzzling: we did not expect productive behavior to co-occur with such clear expectations of confirmation. It seems to us that it provides an existence proof for productive confirmation framing.

IV. DATA AND ANALYSIS
Holly, Judy, and Peter produced data for amplitudes of 10 • , 20 • , 30 • , and 40 • by timing five swings of their pendulum and dividing by five to find the average period; they did five trials at each amplitude. They estimate their uncertainty in timing to be ±0.2 seconds, which makes the uncertainty in their period measurements ±0.04 seconds. Fig. 1 is a graph of their data, although the particular graph shown was not produced until later in this episode. This episode begins immediately after they finish collecting data. Their data is not what Peter expects. Per line 3, Peter expects the amplitude to have no effect on the period; Peter expects that the theoretical model is correct and that their goal is to confirm this model. Thus, Peter locates the problem of their anomalous data in the doing of their experiment. Peter does waiver slightly on whether this data is truly inconsistent with the model, commenting that the changes are "insignificant" in line 3. Nevertheless, his lab partners take up this issue: in line 4 Holly begins to generate ideas and, in the lines below, Judy begins to suggest a different explanation for the trend.
8. [00:55:00.15] Judy: But I feel like the correlation is too strong to ignore, like it-like it makes sense, like it's decreasing very slightly as you decrease the amplitude 9. [00:55:17.09] Peter: Yeah, I mean, I think looking at our data, it would seem that it is related, but just at a really small like ratio I guess, so like the, uh, the amplitude has a really small effect. But like, I know, theoretically we shouldn't be seeing any effect. So I'm wondering what about what we're doing is making it look like that. In line 8, Judy begins to argue that the amplitude does have an effect on the period. She implies that Peter and Holly are ignoring the trend in their data, which is not true: Peter and Holly are invested in finding an explanation. While Peter and Holly are considering the experimental factors that could have produced their data, Judy is suggesting that the trend in their data might be a feature of the phenomenon. Judy's reasoning invokes a mathematical intuition -that the change is so consistent and clean as opposed to noisy, that it makes more sense that these data reflect a physical phenomenon rather than some experimental error. Peter acknowledges this logic, admitting that if you just look at their data it would seem amplitude has a small effect, but he goes on to overrule this empirical evidence because of what they "know" from the theoretical model (line 9).
From these initial exchanges, we can see that Holly and Peter are framing their overall task as confirmation of Galileo's claims. The formulation of their problem as "there must be something else that we're doing that's making it change just a little bit" (line 3) seems to foreclose the possibility that Galileo was incorrect. At the same time, the intellectual work they do here is disciplinarily authentic and productive. In the lines above they are problematizing [24,28].
In the discussion that follows, Holly and Peter attempt to figure out how their experimental apparatus could have produced their results. This conversation continues for another minute, with Peter suggesting a different idea that also doesn't pan out. Still, the intellectual work Peter and Holly are doing here is to come up with a viable explanation for their data, with a particular focus on how their experimental technique may have introduced extraneous dynamics. They may not be successful, but we claim they are engaged in building an explanation, in sensemaking [26]. They are thinking hard about what in the doing of their experiment could introduce error into their data, they are thinking about mechanism, and they are trying to figure out whether this mechanism scales with amplitude.
That the error might be in the drop is not a sufficient explanation for them, but rather it is the beginning of a search for a mechanism that would consistently increase the period length as they increase amplitude. Immediately after suggesting the idea, Holly wonders how it would work (line 19), and Peter is focused on how error in the drop could cause a correlation between amplitude and the size of the effect (lines 20 and 24).
Peter and Holly's sensemaking conversation ends when Judy shares a spreadsheet so they can graph and further interpret their data. It is beyond the scope of this paper to examine Judy's participation in this episode, which is interesting in its own right but not the central focus of our current analysis. It takes a few minutes for them to properly format their plots, but once they do they continue their attempts to figure out their anomalous data. [reformatting the graphs for another 1.5 minutes] Okay, so that, yeah that's our graph with like really exact numbers. This, this still feels very within margin of error, given our...
Holly makes an observation about the data (line 69) and generates a potential explanation (line 71); Peter articulates this explanation is his own words and elaborates on the mechanism for this explanation (line 72). Nearly seven minutes after their initial sensemaking discussion, Holly and Peter return to the subject and continue to sensemake.
This sustained engagement makes clear that, for Holly and Peter, they frame what they are doing as examining their methods for specific flaws that would account for the small dependence on amplitude. That their data defy their expectations drives their intellectual work, and it seems that their expectations shape the explanations they develop; their framing engenders their productive behavior As line 74 indicates, Peter continues to reformat their graphs. Eventually they add a trend line, which spurs this following exchange: Line 92 is a succinct description of how Holly, and previously Peter, have been interpreting their data: "Unfortunately it's not super horizontal" -what they are seeing in their data is not expected and a problem. While Peter had previously shared in this interpretation of their data (see line 1), his response in line 93 is a departure: it's actually not a problem. His use of "actually" in line 93 indicates that he is making a bid to shift from their previous line of thinking.
Based on this data, whether this discussion is sensemaking or a questionable research practice is ambiguous. Peter proposes an explanation in line 93 -"it's close enough to horizontal because of how small our axes is" -but this explanation is lacking in physical or mechanistic reasoning: that the graph looks horizontal is sufficient to them (lines 93, 95, 98). At the same time, this move by Peter could be considered novice uncertainty analysis, a simple attempt to gauge the relative scale of an effect.
There is also an element of intellectual honesty in Holly's suggestion to show both graphs. They are not manipulating their data to hide or obfuscate this anomaly. Given that the trend in their data is somewhat contained within their margin of error, their conclusion that the trend is insignificant and their data is consistent with Galileo is plausibly appropriate.
Following this exchange, it appears that Judy agrees with Peter's explanation. Shortly after, the TA enters their video call, which changes the activity and ends the episode.

V. DISCUSSION AND IMPLICATIONS
We chose to study this episode because the productive behavior we observed challenged our expectations that confirmation framing is problematic. Holly, Judy, and Peter's encounter with anomalous data -more specifically, the inconsistency of their results and their expectations -leads to a rigorous, critical examination of how their experimental apparatus could have skewed their data. Their problematizing and sensemaking not only co-occurs with their confirmation framing, but the data indicate that their confirmation framing engenders their productive behavior.
The way that the students are productive here provides concrete evidence for the speculation from Smith, Stein, and Holmes [2] about a tension between expectations and results sparking a "troubleshooting mode" (p. 13). It is in their repeated efforts to troubleshoot their apparatus and reconcile this data with the theoretical model that we see Holly, Judy, and Peter doing science. In their problematizing and associated troubleshooting, they act as epistemic agents [41]; they take their data seriously as a meaningful reflection of the phenomenon they have constructed. By seeking to reconcile their results with the model, they implicitly position their experimental work and results as legitimate sources of information to be listened to. They may frame this data in a particular way but that they listen to their data is important.
Consider the historical examples of the Cosmic Microwave Background radiation and the Pioneer Anomaly [42][43][44][45]. In both cases, scientists observed results that were not expected and challenged existing physical paradigms. In both cases, scientists exhaustively investigated potential instrument or measurement effects; to disprove existing theory is no small task and scientists often approach unexpected experimental results with skepticism, as Holly and Peter do in this case.
Another key element to their productive behavior is their engagement. Smith, Stein, and Holmes observe an association between confirmatory framing and a "desire to be done" in students that may motivate the questionable research practices [2]. We do not see such a desire to be done in our data. On the contrary, Holly, Judy, and Peter engaged with this problem over an extended period of time and make repeated efforts to construct an explanation for the data.
This paper demonstrates the importance of research on student dynamics in learning environments, in particular the dynamics of "doing science." Our central conclusion is that confirmatory expectations are not necessarily problematic and can indeed lead to productive behavior. Perhaps lab instruction supporting student epistemic agency can enable confirmatory expectations to be leveraged into productive behavior. That said, the lab in this paper and in [2] were both designed around epistemic agency, so our divergent results prompt the question of what leads students to take up opportunities to act as epistemic agents.