Physics graduate teaching assistant use of error framing in recitations and laboratories

Despite the positive gains towards student learning outcomes and engagement, active learning has been shown to potentially increase student anxiety due to a fear of negative evaluation. A pedagogical strategy proposed to mediate this issue is known as error framing; it asks instructors to encourage a perception of errors as being a natural part of the learning process. Previous work on this project investigated how graduate teaching assistants (GTAs) operationalized error framing during their training in a mixed-reality simulator but did not investigate their usage of it in their classrooms. This analysis characterizes the error framing statements made by GTAs during a set of classroom observations. We find that GTAs who employ error framing effectively avoid statements that might decrease student comfort and instead tend towards implicit, indirect strategies.


I. INTRODUCTION
As active learning has grown in popularity, many people in the STEM education community have realized the importance of training graduate teaching assistants (GTAs) in pedagogical techniques that help them facilitate active learning classrooms [1].Additionally, researchers and educators have described various strategies and techniques instructors can use to encourage student participation.One such technique is error framing.
Error framing is an instructional technique where an instructor attempts to encourage participation by creating an environment where errors are seen as a natural part of the learning process [2,3].This technique is theorized to increase participation by reducing student anxiety through reducing the "fear of negative evaluation," the fear that a student will be judged by their peers and instructors for making a mistake [4,5].By framing errors as an important part of the learning process, instructors can mitigate this fear and reduce student anxiety.Error framing can occur when an instructor is engaged with a student or as part of the introduction to an activity [6].For this paper, we will call error framing statements which occur directly after a student's mistake "direct error framing", as these statements attempt to directly frame a student's incorrect answer.We refer to statements made during an introduction or as a general statement as "error climate statements", as these statements are not preceded by a student error and instead aim to create a climate where students feel comfortable making mistakes.
We argue error framing is an important skill for GTAs as we believe instructors have a role in helping students "save face" when a student publicly offers an incorrect answer.This is rooted in the "facework" framework, where "face" refers to a student's self-image they hope to present to others [7].By having a student answer a question publicly, an instructor presents a "face threat," as an incorrect answer may reflect negatively on the student.However, by error framing, instructors can engage in "facework" meant to protect students from this threat.This positive facework helps make classrooms a safer space for students to engage in risk taking and independent thinking.
In contrast to the established benefits of error framing, in previous work, students identified parts of example error framing statements that decreased their comfort [8].For example, students disliked when GTAs started by directly commenting on how the student was wrong, focused on the student's mistakes, hedged when acknowledging the student's ideas as understandable, or used certain negative language such as "mistake" or "error".This creates a potential concern that, if used incorrectly, error framing could increase student discomfort instead of decreasing it.Thus, it is important to see how GTAs operationalize error framing while teaching students.
Prior research by the team characterized the error framing statements made by GTAs during their professional development in a mixed reality simulated classroom [8].However, GTAs' implementation of trained pedagogical techniques in their classroom varies from how they were trained [6,9].Thus, it is not sufficient to study how the technique is implemented in training.In this paper we characterize GTAs usage of error framing in their classrooms in comparison to the themes established in prior work.

A. Participants
This study was conducted at a large, research-intensive university in the southeastern United States.The data set was pulled from a sample of classroom observations of 27 different GTAs over three semesters: spring 2019, fall 2019, and spring 2020.Some GTAs participated in multiple semesters, and some did not, depending on both their teaching assignment and their choice to consent to research in a particular semester.The GTAs taught an introductory physics "mini-studio" with a class size of around 32 students [9].The mini-studio style classes featured a 75minute tutorial based on the University of Maryland Open Source Tutorials [10], followed by a 15-minute quiz, and then 80 minutes of an Investigative Science Learning Environment (ISLE) curriculum based lab [11].
GTAs participated in a professional development session where they practiced using various pedagogical techniques, including error framing [8,9,12].After this training, the GTAs were observed in select classes throughout the rest of the semester.Four classes were observed for each GTA in spring 2019 and three classes were observed in fall 2019 and spring 2020.Observation numbers varied due to hurricanes and the beginning of the COVID-19 pandemic.These observations were conducted using a modified version of the Laboratory Observation Protocol for Undergraduate STEM (LOPUS) [13].LOPUS is an observational protocol that involves observers recording the occurrence of specific behaviors, such as a student asking the GTA a question, during two-minute intervals.For our observations, we added an additional code for verbal feedback that contained error framing [14].Of the 27 GTAs who participated in the professional development during some of the three semesters, 14 used error framing during their observations and a total of 39 two-minute intervals were coded as containing error framing.

B. Positionality
Researchers working on this project were involved in various other aspects of the mini-studios.JC was involved in the curriculum development of the mini-studio labs and sometimes led GTA prep.CD and TW both at various times led GTA prep meetings during these semesters and conducted some of the observations considered.DS participated in this training in a later semester and taught a different version of this course as a GTA.

C. Themes and Codes
Prior research by members of the team established common themes related to the implementation of error framing [8].These themes and codes were utilized for this analysis.One set of themes, displayed in Table I, emerged from the research team's interpretation of error framing.A second set of codes, displayed in Table II, emerged from an analysis of student reactions to exemplar error framing statements during interviews.
The researcher-generated codes describe error framing statements in two ways: error indication and framing.Error indication describes how a GTA comments on a student's incorrect response and has two potential codes: explicit and implicit.Framing contains three codes: natural, beneficial, and positive acknowledgment.These codes describe how a GTA attempts to frame incorrect student responses in a positive light.A statement would be coded as natural if a GTA framed a mistake as being natural and common.It would be coded as beneficial if the GTA framed that making the mistake was beneficial for learning.If the GTA acknowledged an incorrect response with praise but did not explain why it was considered positive acknowledgement.
The student inspired codes contain two themes: framing and tone.The framing codes describe student perceptions of ways GTAs frame errors and how it impacts the student's comfort.This includes framing methods that students thought negatively impacted their comfort, which was not considered in the researcher defined codes.It should be noted the student generated codes for framing are different from the researcher defined codes.This is expected, as the undergraduate students interviewed were not formally introduced to the concept of error framing.
The other theme identified from the student interviews was tone.Tone describes how students perceive the diction and inflection used by the GTA and how it impacts their comfort.For this paper, tone was not considered for a few reasons.First, tone is more subjective than the other codes considered; students may hear the same statement and perceive the tone differently.Additionally, perception of tone depends on context that is unavailable in our data.We only recorded audio files of the observations, so actions such as gestures, facial expressions, and posture, which influence perception of tone, could not be considered for our coding.Due to our inability to satisfactorily code tone, it was not considered in this analysis.

D. Identifying and coding error framing statements
Error framing statements were identified using the time intervals marked as containing error framing feedback from the LOPUS observations.If the observation had an associated audio file, the flagged time interval was then transcribed by hand for further analysis.This resulted in 31 segments.Two coders initially analyzed these segments using the codes from Table I, and discussed their coding with a third researcher, who had conducted some of the observations.The research team noticed that some of the statements identified by observers did not align with the error identification codes.For example, after no students volunteered to answer a question, a GTA said, "It's okay if you're wrong.There's, like I told you, there's no wrong answer, just focus on what you think."While this fits our definition of an error climate statement, since it does not address a student's incorrect answer, it did not match the error indication codes.Thus, we decided to exclude error climate statements from this coding analysis.Two coders coded the remaining statements with the additional framing  GTA uses hedging, such as "kind of", when acknowledging student reasoning as sensible Use negative words GTA uses negative words, such as "misconception", when referring to a student's idea codes from Table II.They discussed their coding with the third researcher and came to agreement.Then, a fourth researcher, who was also part of the observation team, conducted a peer review of the coding for all 31 statements.The research team discussed coding until consensus was reached.

III. FINDINGS
Of the 39 times error framing was coded during the observations, 31 were available for additional analysis because the GTA had consented to their audio file being used for research.After review of these 31 statements, we interpreted 13 as direct error framing, 7 as error climate statements, and 11 as not error framing.
These 11 statements, while initially coded by the observers as error framing, ultimately did not match with our definition upon further review.Because the LOPUS observations were made live with no opportunity to review, it is not surprising that some flagged statements did not fit the error framing codes upon further review.Often these statements contained similar ideas to error framing but lacked key aspects.For example, one flagged statement that was ultimately coded as not error framing was "Correct, and that's a big lesson to learn right now.When we're talking about weight, you need to be clear are we talking about mass or we talking about actual weight."Here, the GTA highlights a potentially confusing topic after a student asks him a question.However, the GTA is not responding to a student's mistake.Instead, the GTA is acknowledging the validity and importance of the student's answer.While providing this sort of validation to students is similar to the goal of error framing, ultimately this quote features a different behavior than what we want to investigate.
Coding with the researcher-defined and student-inspired codes for the 13 direct error framing statements is shown in Table III.Trends among the data are explained below.

A. GTAs avoid explicit error indication
Our sample of direct error framing statements were overwhelmingly coded as implicit for error indication.Only 1/13 of the direct error framing statements was identified as an explicit error indication.It is possible GTAs were not comfortable with telling a student their answer was incorrect.Prior research found that students dislike when GTAs directly refer to and focus on their mistakes [8].It appears that GTAs, having been students themselves, tend to avoid statements that create uncomfortable situations for the students.Prior literature describes the debate about how and when to tell a student they are incorrect [15].Future work will focus on investigating why most GTAs did not explicitly mention a student's idea was incorrect and how the direct indication of an error impacts students' feelings and their learning.

B. GTAs avoid student identified negative codes
Prior research identified four potential framing strategies that students identified as having a negative impact on their comfort level.Interestingly, all four of these negative-impact codes had a low occurrence in our sample.Only three of the 13 direct error framing statements aligned with codes which decreased student comfort, as shown in Table III.In two of those statements, GTAs used hedging language.In the other statement, a GTA used negative words.The overall low occurrence rate of these negative codes again indicates that GTAs likely understand and empathically know what might make a student uncomfortable and try to avoid doing so.

C. GTAs refer to their own mistakes while error framing
A common strategy we saw repeated by many of the GTAs was a strategy we refer to as "GTA selfidentification".This is where a GTA attempts to frame a student's error by saying that they have made similar or the same mistake in the past.For example, a GTA teaching a circuits lab said, "So you have accidently flipped both of your leads.That's okay.Every time I build a circuit, I also flip both of my leads, even though I'm supposed to be good at this."Not only does this normalize the student's mistake, but it also presents an opportunity for the GTA to build rapport.An instructor's ability to be relatable or perceived as a "real person" has been correlated with positive aspects of facework (working to protect or maintain a student's social identity) [16] which suggests this style of error framing might increase student comfort.Also, when GTAs discuss how they have struggled with the material or concept, students might be encouraged to feel comfortable with their own mistakes.

D. Error climate statements
Despite not being considered in this analysis, our results show that GTAs tend to make error climate statements frequently when using error framing.Out of the 31 statements transcribed, 7 were error climate statements.These statements were often made in response to student confusion about a topic.As an example, in response to a student's weary sigh, one GTA responded, "It's okay, you're supposed to be confused, it's going to take a little bit to get through this."In this example, we see that when the student experienced a moment of frustration or confusion, the GTA tried to normalize that experience.Error climate statements could help GTAs address situations where students are so confused they cannot even provide an incorrect answer.In these moments students are likely frustrated, and error framing may help decrease negative student experiences.

IV DISCUSSION
In this analysis, we characterized how GTAs implemented error framing in their classes.Through this characterization we noticed GTAs who used error framing statements tended to avoid statements that might decrease student comfort.GTAs may be aware of the impact their statements have on student comfort and avoid statements they think could have a negative impact.Thus, any potential negative impact of error framing statements seems to be moderated by the GTAs themselves.This indicates that error framing may be an effective method for reducing student anxiety, and GTAs should be trained to use it in their classes.
We also observed GTAs use a wide variety of different kinds of error framing statements.We believe it is important that GTAs are trained in how to use these different forms of error framing.GTAs tend to use pedagogical skills only if they align with their own perceptions of the learning process [9].By teaching GTAs many different forms of error framing, GTAs can likely find a form that they will use.
Future work should explore the impact of error framing when enacted by faculty instructors rather than GTAs.Students have reported GTAs to be more relatable [17] and better at guiding group discussion [18] than faculty.It is possible that faculty use of error framing statements may help to break down the power dynamic between student and instructor, allowing for open communication [19].On the other hand, faculty may have a more difficult time implementing error framing without negatively impacting student comfort, since students are typically more comfortable with GTAs from the start.

V. LIMITATIONS
There are a few factors which limit the generalizability of our claims.Notably this study only investigated a single style of class and only featured GTAs who chose to use error framing in their classes.While GTAs who chose to implement error framing implemented it well, no claims can be made about the GTAs who did not use the skill.It is likely that if they were coerced to use error framing their implementation would differ from what was observed in this study.Future work will explore why some GTAs did not use error framing.Additionally, live-coding observations is difficult, and it is possible that some error framing statements were not flagged during the observation, which might introduce bias as to what error framing statements were available for analysis.

ACKNOWLEDGMENTS
This work is supported in part by National Science Foundation Award #1725554.

TABLE I .
Researcher defined codes

TABLE II .
Student inspired framing codes

TABLE III .
Findings for frequency of codes, with exemplar quotes The conceptual process was right, but the value was wrong, but that's okay."Implicit 12 "Right, so we would think that, yeah, but, and in both cases we're gonna do the exact same amount of work."Researcher Defined: Framing Natural 10 "So, you just had something a little askew.That happens you're not used to building circuits."Beneficial 2 "No, no, but this is like not a thing about physics like this is about something with our instrumentation or measurement.See and this is actually a good lesson about real world stuff."Okay I see where you're coming from, I see because of the word gravity."The conceptual process was right, but the value was wrong, but that's okay.You just switched your voltage and leads, that's very common.All right try again.""I would like at least two variables on the graph, it looks like this is just your resistance but that's ok, you won't lose points for it because we're learning."Provide explanation to subject matter 4 "Usually that's what people think of, when I'm talking to you and I say take into account gravity, that's exactly what you're going to do, you write down FG.This is a problem with the English language, we're actually talking about gravity as a constant, as something that's pulling us, as an acceleration."