Modified color frames for analyzing group interactions during an online quantum tutorial

In this paper, we analyze video recordings of students working on tutorials in Zoom breakout rooms in an upper-division quantum mechanics course. We investigate group behaviors in this virtual environment, including the effects of instructor presence. To this end, we modify the Color Frames coding scheme introduced by Scherr to suit the virtual nature of the interactions. By broadening the frames and allowing for multiple overlapping frames, we are able to describe some group behaviors not otherwise captured. For example, in some instances, students take on an authoritative role in the group, and in other instances, groups engage in overtly casual behavior while nonetheless having on-topic discussions. We observe significant variation in how much time each group spends in each frame, but find that all groups spend some time in all frames. Instructors can be present without dominating or eliminating discussion between students, and their presence need not significantly impact the time students spent in an “informal/friendly” frame. However, instructor presence significantly reduces time spent working individually. Our findings will support additional research into the dynamics of student discussions during tutorials and aid ongoing development of online tutorials that can, e.g., be assigned for use outside of class.


I. INTRODUCTION
Tutorials are a commonly used tool for active learning in physics [1,2]. In tutorials, students work in small groups on concept-focused, guided-inquiry worksheets. Instructors (including teaching or learning assistants) play the role of Socratic questioner with the goal of guiding students to construct knowledge for themselves. Tutorials were popularized at the introductory level [1], where they are typically run in dedicated recitation sections [3]. Tutorial use has expanded to the upper-division [4][5][6][7], and also to large lecture halls [8]. In this work, we investigate student behaviors during tutorials in an upper-division quantum mechanics course where instruction took place entirely online, necessitated by the COVID-19 pandemic. To that end, we modified the existing color framework initially used for in-person group tutorials, and used it to analyze student behaviors in a single tutorial.
Our interest in online administration of tutorials predates the pandemic. With the goal of bringing tutorials to courses that may not have the instructional time or resources to implement them in class, we have developed an online tutorials website called ACE Physics (acephysics.net). We have been redesigning our paper-based tutorials [9] to fit the online format where students may work individually or in groups. For this study, students worked on an ACE Physics tutorial about quantum entanglement [10] in groups in Zoom breakout rooms, as described in Section II.
One challenge for instructors in virtual tutorials is that they cannot glance around the classroom and get a sense for how students are interacting. During in-person tutorials, instructors normally walk around the room listening in and only stop to chat if a group explicitly asks for help or if the instructor sees an opportunity to engage with students. In the virtual format, instructors must choose to visit one breakout room at a time. In this paper, we analyze recordings of students' Zoom breakout rooms to investigate how tutorial groups interact in this environment both with and without the instructor present. Our ultimate research goal is to better understand how students interact with our tutorials in an online format. However, we cannot begin to assess online tutorials without a robust observation tool for online settings. Therefore, our preliminary research goal is to modify an existing observation protocol to be suited for the online environment.
There is extensive precedent in physics education research for the analysis of group work in various environments, but mostly in in-person settings [11][12][13][14][15]. Scherr et al. created a framework called color frames that defines four categories (described below) for student behaviors. Scherr calls for applying a systematic observation protocol to gain insights into student learning while balancing "generalizability with interpretive validity" [16]. Color frames focus on the dynamics of student behavior, student framing of activities, and the substance of student reasoning, all of which are central to our own interests in developing and understanding student interactions with upper-division quantum mechanics tutorials.
We have adopted color frames for our analysis, but have modified the framework for two reasons. First, color frames were developed in the context of in-person group work, so we have adapted the framework to apply to student behaviors that are evident in the virtual setting. Second, we observed additional behaviors belonging in each category. After discussing our methods, we present our modified framework and provide examples of student interactions that fit into each frame. We then use the frames to analyze the group behaviors of eight groups working through the quantum entanglement tutorial.

II. METHODS
Data collection: Data collection took place during two semesters of upper-division quantum mechanics at a large, primarily white, R1 university. In both semesters, the course was taught by author SP using the materials available at [9] following McIntyre's text [17]. The course included three weekly lecture periods as well as an optional, once-weekly tutorial recitation section, all of which were conducted via Zoom. There were 58 and 96 students in the class in the two semesters, Fall 2020 and Spring 2021 respectively, of whom roughly 30% attended the tutorial sessions.
Tutorial sessions lasted 50 minutes during which students worked on ACE Physics tutorials in groups of 3-5 in Zoom breakout rooms. Instructors, including the professor, a graduate teaching assistant (TA; author GC), and an undergraduate learning assistant (LA) circulated through breakout rooms to answer questions and facilitate discussion. Each week, a designated student in each breakout room recorded their Zoom session and uploaded the video to a shared drive after class. Each student worked on their own copy of the ACE Physics website, but typically one student shared their screen. The recordings included the shared screen as well as thumbnail video feeds from every student in the group.
The present study focuses on only one tutorial, the "EPR & Entanglement" tutorial [18]. We have recordings from eight groups, three from the fall and five from the spring. The fall and spring semesters were nearly identical, but because there were more groups in the spring, instructors spent less time with each group during the tutorial.
Analysis methods: We have based our analysis on the color frames developed by Scherr [19,20]. These color frames were created by first identifying student behaviors and then interpreting those behaviors. In our work, we take these interpretations and add (and in some cases subtract) additional behaviors to fit the virtual environment. In some cases we expanded the interpretations of the frames to account for all time elapsed during the tutorial (see for example, the yellow frame). Additionally, we allowed for multiple codes to be applied at the same time. Rationale for our decisions and explanations for how this was done are provided in Section III, with descriptions of the modified frames.
We coded each minute of the recordings with one or more of the modified color frames. There were a total of 348 minutes of tutorial work recorded across the 8 groups. The modi-TABLE I. Modified color frames and the behaviors that were coded for each frame. The original color frame definitions were presented by Scherr et al. in [19]. Italicized text indicates behaviors that have been added or modified for the purposes of this study.
Green frame -Discussion fied framework was developed in discussion amongst the authors based on observations from the recordings. All recordings were coded by author BC. Additionally, 16 minutes from each of 4 recordings were re-coded independently by author GC. We found an inter-rater reliability rate of 87% for these 64 minutes. Additionally, we noted those minutes where an instructor was present in the Zoom breakout room (whether or not they interacted with the students).

III. COLOR FRAMES FOR ONLINE GROUP WORK
In this section we describe the four color frames and our modifications to them. The behaviors that make up each color frame can be seen in Table I. Italicized text indicates a modification to the original color frame behaviors from [19]. Some modifications to the frames followed clearly from the transition to a virtual environment, but other changes were made to more accurately record the student behaviors observed in our data. We include examples below to justify these changes.
Green frame: This frame is identified as the "Discussion frame" by Scherr [19,20], and we retain that description. It includes debate, disagreement, and attempts to build collaborative understanding between peers. Behaviors added to this frame were primarily the result of the virtual context. This frame consists of prolonged discussions or disagreements where students make substantial modifications to the discussed ideas. Discussions were typified by respondents' careful paraphrasing of or explicit disagreement with the original speaker's idea, including rebuttals and elaborations. To be considered a discussion, these modifications must be undertaken by the majority of members engaged in conversation. The tendencies for elaboration and collaborative understanding are what separates the green frame from conversations that occur in blue and red frames.
Blue frame: Initially subtitled "Worksheet frame" [19,20], we have modified the Blue frame to encompass any individual work. With the limitations of the online environment, we cannot always tell whether students who are working individually are working on the tutorial or are engaged in offtopic activities. We also occasionally encountered lengthy periods of silence, which would be coded in this frame.
The original definition of this frame included "check-ins", where students who are working individually briefly check in with other group members. One form of check-in we observed was "echo-back" talk without substantial modification to the content. For example, students working individually might announce their answer to a question, and group mates might echo the answer to signal their agreement before returning to individual work.
Several instances of echo-back talk may occur back-toback, creating a facsimile of discussion. However, these are included in the Blue frame when they consist primarily of agreement or consensus without substantial elaboration.
Yellow frame: Initially subtitled "Joking frame" [19,20], we have modified the yellow frame to include all off-topic discussions and overtly casual behaviors. This is a substantial change to the framework, but we often noticed discussions that took on an overtly familiar or casual element, and wanted our codes to capture this pattern. This frame now includes instances of particularly friendly, casual, and familiar discussion between peers, in addition to the outright jokes and laughter from the original definition of the frame. By expanding the frame in this way and by allowing sections of time to be coded with more than one frame, we are now able to distinguish between casual discussions that are off-topic (yellow frame only) and casual discussions that are on-topic (yellow and green frames simultaneously).
Red frame: This frame was initially described as the "TA frame" [19,20], and was activated only during periods where instructors or TAs engaged with students. We have made two substantial modifications to this frame. First, we have expanded the Red frame to include any instance where an instructor, TA/LA, or group member imparts their knowledge to the group in an authoritative manner. This change allows us to distinguish between collaborative discussion (green frame) and times when a single student is dictating answers to others (previously green frame, now red frame). We observed multiple such instances where students were treated as authority figures by their peers in absence of instructor presence, one of which is visible in Fig. 1 between the 20 and 30 minute marks. In some of these instances, the student in the author- ity role did not have the correct answer. Second, the presence of an instructor no longer automatically activates the red frame, because we found that student behavior in the presence of instructors was more varied than a single frame could capture. For example, there were multiple instances of friendly discussion between students and instructors (yellow frame), continued discussion between students with minimal instructor input (green frame), or minimal engagement from students as they continued to complete the tutorial individually despite instructor presence (blue frame).
While primarily a one-way frame with information being imparted, some exchanges occur within Red frame that mimic back-and-forth discussions seen in Green frame. However, exchanges within the Red frame are limited to clarifying questions while the Green frame will contain back-and-forth debates or rebuttals. The duration of such exchanges in Red frame is usually shorter than exchanges in Green frame.

IV. RESULTS FROM THE ENTANGLEMENT TUTORIAL
We have applied the modified color frames framework to recordings of eight groups working on the EPR & Entanglement tutorial. From these data we can create a timeline for each group showing which frame(s) the group is in at each minute over the course of a tutorial. An example timeline is given in Fig. 1. This example is not representative of all eight groups; in fact, each timeline was idiosyncratic and created a unique fingerprint for each group.
As can be seen in Fig. 1, there are many instances where multiple codes were given to a single minute. The average number of codes assigned for each minute was 1.4. A total of 225 minutes (65%) had one color frame identified, 116 (33%) had two colors, and 7 (2%) had three color frames coded. No minute was coded with all four colors. Fig. 2a shows the average time that groups spend in each color frame. Overlaid are the percent of total time that each group spent in each frame. Additionally, we have separated the data into times when the instructor is present and times when the instructor is not present (Fig. 2b).
As shown in Fig. 2a,  with yellow (friendly) arising in roughly a quarter of tutorial time on average. All groups spent some time in every frame, but there is significant variation between groups. The most pronounced outlier is the group that spent over 70% of the time in the Yellow frame. The timeline for this group is shown in Fig. 1, which shows that the time the group spent in the yellow frame overlapped with time spent in other frames. This example demonstrates how the yellow frame should not be interpreted to be unproductive without further analysis. Fig. 2b shows the effect of instructor presence on the breakdown of color frames. Unsurprisingly, instructor presence increases the red frame significantly: groups spent only 15% of the time without instructors in the red frame, but this increased to nearly 60% of the time when instructors were present. Although one group spent 100% of instructor time in the red frame, most spent less, indicating that instructors can be present without dominating the discussion.
On average, instructor presence reduced the green frame by approximately two fifths (from 34% to 20%). For three groups, instructor presence completely eliminated the Green frame, but other groups tended to have active discussions re-gardless of instructor presence. This was more common in the fall semester, which had a higher instructor-student ratio, meaning instructors were present for a larger percent of the time. Perhaps surprisingly, instructor presence did not significantly impact the time spent in the yellow frame. This may reflect the instructors' intentions of cultivating a friendly, informal tutorial environment.
Instructor presence significantly reduced time spent in the blue frame for every group (from 64% on average without instructors to 21% on average with instructors). With instructors in the Zoom breakout room, students tended to not work individually. To some extent, instructor presence tended to replace time spent in the blue (individual) frame with time spent in the red (authority) frame.

V. DISCUSSION
We found that all four color frames were present in every tutorial recording. The blue individual frame was the most common with an average across all groups of approximately 50%. The time spent in this frame was much lower when an instructor was present in the room.
It is not clear whether similar patterns would be found for in-person tutorial sessions. We expect that virtual environment may skew group dynamics in favor of more individual work interspersed with sporadic discussions, as students do not share a physical environment and thus may be less inclined (in general) to engage with each other the same way they did in person. Further study is required, the results of which would inform the development of online tutorials and help instructors better understand differences between group work online and in person.
Modifying the color frames framework allowed us to categorize student behaviors throughout a single tutorial such that we were able to gain insights into the varied ways students interact with each other and instructors/TAs/LAs, during online tutorials. By broadening the frames and allowing for multiple overlapping frames, we were able to describe some group behaviors not otherwise captured. For example, we distinguished overtly casual, but nonetheless on-topic discussion (yellow and green frames simultaneously), from distracted or unfocused discussions which were not relevant to tutorial completion/comprehension (yellow frame only). We also discovered that the presence of an instructor does not necessarily place students in the authority (red) frame. This is especially true for groups which spent the majority of their time in blue frame, and for authority figures who were not the professor (i.e. the TA or LA). We found that blue frames and red frames rarely overlapped, and that an ongoing yellow frame did not necessarily prevent students from entering the green frame.
While we cannot use these results to make generalized claims regarding "typical" group behaviors, it does appear that all frames likely serve important and necessary roles in group completion of tutorials (given that all frames arose in all groups). We believe it is possible that the lack of overlap between blue and red frames results from instructors "pulling students out" of the blue (individual) frame, where they can then engage in any of the other frames.
Because not every instance of red (authority) frame is initiated by instructors, we believe it is beneficial for both instructors and students to notice when authority frames are repeatedly initiated by students without an instructor. Although we did not analyze student understanding within the scope of this paper, we did notice that students who act as authorities do not always relay correct information to their peers.
We were pleased to discover that the yellow (friendly) frame did not eliminate the possibility of productive discussion between students. Although the previously discussed group (timeline shown in Fig. 1) is certainly an outlier among the eight analyzed groups in terms of percentage of time spent in yellow frame, productive, on-task discussions overlapped with the casual, friendly, joking behaviors in multiple groups. Instructors might want to pay attention to students engaged in lengthy yellow frames, as not every instance of friendly and familiar behavior is inherently off-topic. In fact, we have observed instances of the yellow frame that encourage all students to participate in conversation and can lead to productive discussions.

VI. CONCLUSIONS
We found the Color Frames framework to be a fruitful starting point for analyzing videos of virtual tutorial group work. By modifying the framework, we were able to capture nuances in the dynamics of virtual tutorial groups in upperdivision physics. For example, we were encouraged to see that the presence of an instructor does not automatically place the group into the "authority frame" and that discussions still occur. We also learned that the presence of the "friendly frame" does not preclude productive discussion. This analysis will be used as a starting point for analyzing the ways students engage with tutorial content. Future work will look at the effects of different questions on group behaviors. Specifically, we are interested in whether or not specific types of questions are more likely to prompt discussion. We also plan to expand our analysis using epistemic games [13] to investigate the reasoning students use while in the green discussion frame. The results will help tutorial authors modify this tutorial and inform the development and refinement of other tutorials.