Characterizing the feedback that learning assistants give to faculty

Learning assistants are undergraduate peer educators that help facilitate learning in a university classroom environment. Jardine (2019) found that learning assistant feedback to faculty roughly fell into three categories: course logistics, student behavior, and student understanding. We built from this previous work by further characterizing the feedback given to faculty by learning assistants and found the following categories: student experience, classroom content, classroom structure, accessibility, empathy, and broad feedback. Using interview data with learning assistants and faculty working with learning assistants, we created a preliminary framework for the types of feedback and examples by learning assistants. This framework may be useful for both learning assistants and faculty members as they provide and elicit feedback.

The learning assistant (LA) model creates a unique type of faculty-student partnership that aims to better the classroom environment. Developed by the University of Colorado Boulder, the program consists of learning assistants, who are undergraduate peer educators that facilitate in-class learning, and university faculty members [1]. Learning assistants can help students learn effectively, keep students engaged with the curriculum, communicate with professors on how to better the classroom experience, and be a bridge between students and professors [1][2][3]. Learning assistants can also lead to various positive outcomes including increasing the number of K-12 teachers, improvement in the understanding of subject material and feelings towards science for students, and growth in faculty's attention to student learning [1,2,[4][5][6]. Through the support and training from a pedagogical course and weekly preparation meetings with their respective faculty members, learning assistants are primed to offer insight and help prepare their peers for success.
One important facet of the learning assistant-faculty relationship is the feedback that learning assistants provide faculty members. In their weekly preparation meetings, faculty members and learning assistants gather to discuss how to help students and prepare for the upcoming classes. Though the structure of these preparation meetings is not uniform among all partnerships, the goal of these meetings is to improve upon previous experiences in the classroom and prepare for new ones. To improve upon previous experiences, learning assistants may provide feedback on a range of topics, from how students are performing in class to how the professor can better their lecture slides. Recent research has begun to characterize different aspects of the relationships between faculty and learning assistants [7][8][9][10]. This paper will further build upon this by working towards answering the following two research questions: (1) What is the content of the feedback that learning assistants give to faculty? (2) What purpose does feedback between learning assistants and faculty serve?

I. PRIOR STUDIES OF LEARNING ASSISTANT PARTNERSHIPS
In an interview-based study at Chicago State University, Sabella, Van Duzor, & Davenport [7] characterize the relationships and dynamics between learning assistants and their partner instructors. The paper classifies these relationships into three categories: mentor-mentee, faculty-driven collaboration, and collaborative. Mentor-mentee relationships can be defined as a unidirectional relationship on the instructor's part within a learning assistant and faculty partnership. Relationships that are classified as faculty-driven collaboration are partnerships between learning assistants and faculty where faculty get feedback from learning assistants while still leading most of the dialogue. Finally, in collaborative relationships, faculty get feedback and inspiration for course material from learning assistants. After defining each classification of relationships, the paper goes further to discuss the benefits and drawbacks of each type of relationship.
Building from Sabella, Van Duzor & Davenport [7], Davenport et al. [8] utilize the classification of relationships between learning assistants and partner instructors to create a tool to analyze and label interactions between learning assistants and faculty members. The Preparation Session Observation Tool (PSOT), characterizes the behaviors and content of talk during meetings between learning assistants and faculty.
In a different study, Jardine [9,10] connects the type of feedback that learning assistants give to their partner instructors and the roles that they assume within their relationship. She identifies three different types of learning assistant feedback: course logistics and instructional materials, student behaviors and attitudes, and student ideas and conceptual understanding [9]. She also identifies the roles that learning assistants can be positioned as within their relationship with faculty. The role of the student is defined as a learning assistant whose "expectations for behavior were to listen to the faculty member, follow directions, complete assignments, or answer questions" [10]. The role of the informant is defined as a learning assistant that "inform[s] instructors what students were saying and doing in and out of class" [10]. The role of the consultant is defined as a learning assistant whose interactions with their partner instructors are where "their assumed right or duty is to provide advice to the instructor" [10]. The role of the co-creator is defined as a learning assistant who is "provided opportunities to develop instructional materials along with the faculty member, or on their own with support and feedback from the faculty member" [10].

II. METHODOLOGY AND DATA COLLECTION
Based on this previous work, we were interested in understanding the learning assistant feedback process within our own program. Specifically, we wanted to understand how differences in our local program compared to others affect the types of feedback LAs give. We were also interested in what feedback would emerge in interviews compared to Jardine's study which primarily relied on observations.
We developed two semi-structured interview protocols, one for learning assistants and one for faculty. The questions addressed several topics including course details, the learning assistant and partner instructor relationship, the feedback given by learning assistants, and the impact of LA training. Example questions included: "What type of feedback do LAs usually give? What do you do with this feedback?" and "Have you ever implemented feedback from LAs and if so, how did it go?" The San José State University LA program was created to serve introductory courses in the College of Science. The program is jointly supported by the College of Science and Peer Connections. Peer Connections is an organization that supports several forms of undergraduate peer educators (including LAs) and is housed in the Office of Student Affairs. In their first semester of the program, LAs take a pedagogical TABLE I. Categories of feedback developed from analysis of interviews with professors and learning assistants.

Category
Definition Example Student Experience Feedback that is rooted in specific student experience whether it is based on personal experience as a student or based on actual student experience witnessed in the classroom.
"Hey, as a student, when I was taking this class, I was also thinking like that. Maybe they have explained me about this part so it would have been easier to do diagram, or something like that."

Classroom Content
Feedback that helps to create or improve classroom content, including feedback on lecture materials, activities, and assessments.
"We'll go through the activities, and then we'll pick out certain things that maybe could be improved upon or certain things that they did well, and then things to look out for." Classroom Structure Feedback that addresses the order or structure of the class.
"I did an activity this semester that was new, so I had more of an outline that I brought. I really wanted [ training course in the College of Science. Those that complete the pedagogy course continue to be learning assistants through Peer Connections the following semesters. Indukuri has been an LA in the program for 3 semesters and Quan has been involved in the LA program for over three years.
We invited all the members of the LA program except for those who would have a conflict of interest with a member of the research team to participate in an interview. We interviewed two learning assistants who we refer to as Finn and Kai, and two professors who we refer to as Professor Blake and Professor Layne. One LA was in their first semester of the program and the other LA had been in the program for multiple semesters.
Our analysis began with collaboratively watching videotapes of the interviews and discussing moments where interviewees described LAs giving feedback to their faculty partner. We defined feedback to broadly refer to an LA giving input, recommendations, or participating in decision-making related to the course. We reviewed interview transcripts to identify all segments where interviewees described feedback. Next, we conducted an iterative process of characterizing the feedback according to the roles identified by Jardine [9] and developing categories for the content of the feedback. Through this process of characterizing feedback, we found that the roles in which LAs were positioned aligned well with the roles identified by Jardine. Using a progressive refinement of the hypothesis approach [11], we identified six cat-egories to characterize the content of all feedback found in interviews. Because we wanted to see if there were relationships between categories, we defined the categories, such that they do not overlap.

III. FINDINGS
We now define these six categories of feedback identified: Student Experience, Classroom Content, Classroom Structure, Accessibility, Empathy, and Broad Feedback.

A. Student Experience
We define student experience feedback as comments made by learning assistants that are rooted in specific student moments, whether it is based on personal experience as a student or based on moments with students in the classroom. We see evidence of student experience feedback when a learning assistant gives feedback from personal experiences or describes how students are responding to an aspect of the class. Student experience feedback differs from a general claim about students in a classroom because this type of feedback must be based on a specific experience faced or seen by the learning assistant. For example, in response to a question asking what faculty members expect from them as a learning assistant, Finn elaborated on a specific instance where they described their time as a student: Finn: Hey, as a student, when I was taking this class, I was also thinking like that. Maybe they have explained me about this part so it would have been easier to do diagram This is an example of student experience because Finn is pulling from their own history as a student in class to let their partner instructor know how a student might respond. Because Finn says "as a student, when I was taking this class..." we interpret this to mean they are drawing on a specific instance when they were a student doing this activity.

B. Classroom Content
Classroom content is a type of feedback that helps to create or improve classroom material, such as lecture slides, talking points, and course assessments. This type of feedback should not be about a specific student experience within the classroom, as that would fall under student experience. This type of feedback can be identified when there is feedback in the form of advice, input, or suggestions that pertains to activities or content in the classroom. For example, in response to a question surrounding the strength of the relationship between Professor Blake and their learning assistant, Professor Blake elaborated that: Professor Blake: One of [their class] assessments came from an LA idea. So we did a ... G2 pen, you could rearrange the parts in it and then kind of launch something out of it. This piece of feedback from Professor Blake's learning assistant is an example of classroom content as it leads to a summative assessment that is given to the class. We note that this particular piece of feedback does not have a specific and personal learning assistant experience tied to it.

C. Classroom Structure
We define classroom structure feedback to be statements that help to improve upon or recognize pitfalls within the order or structure of the lecture. Similar to classroom content, this type of feedback is not connected to a personal experience faced by the learning assistant. We can see evidence of classroom structure feedback when a learning assistant provides feedback that is related to the organization of the classroom and not the content of the class. For example, in response to a question asking about the general categories of feedback that learning assistants provide to Professor Blake, Professor Blake spoke about a particular piece of feedback that their learning assistant provided them: Professor Blake: [My LA said] 'Maybe next semester you switch the order of these two activities, or maybe we should spend more time on blah, blah, blah, something like that.' This is an example of classroom structure since rearranging the order of activities is changing the structure of the classroom, but not changing the content. By also elaborating that the professor should "spend more time", the learning assistant is providing their professor with feedback on how to change the organization of the lecture to give more time to certain concepts or activities.

D. Accessibility
Accessibility is any feedback that takes into consideration students' circumstances and designs for people who might be excluded. This type of feedback can be identified when a comment by an LA consists of actionable advice to make sure all students can be included in the learning that occurs in classrooms. For example, in response to a question asking about the impact of the feedback they provide on the environment in their classroom, learning assistant Finn described the following piece of feedback they provided: Finn: There was a video only on YouTube, so I just asked [the instructor] if there is any way they can download that video and then they can watch it on their own way rather than just every time going to the YouTube. Maybe some students have a poor WiFi, maybe their WiFi is not that strong as other people We categorized Finn's feedback as accessibility because it takes students' circumstances into consideration to make the learning experience available to all. By asking to make the video downloadable since some might have poor WiFi, Finn's feedback gives actionable steps the professor can take to ensure that more students can participate in the learning process.

E. Empathy
We define empathy feedback as comments made by learning assistants that address the emotional and identity aspects of the learning process for students. This type of feedback can be identified when a learning assistant makes observations or statements on what life is like for students outside of the classroom, ranging from discussions on diversity to the weight of particular statements, to help students have a better in-class experience. For example, in response to a question surrounding the impact of the learning assistant training on their experience with their learning assistants, Professor Blake elaborated on a piece of feedback that their learning assistant provided: Professor Blake: [My learning assistant] said a diversity, equity and inclusion kind of thing. So [they were] talking about how [they] switched from saying hey guys to hey y'all. This is an example of empathy feedback as this statement speaks to the emotional and identity aspects of the learning process for Professor Blake's students since it recognizes that not everyone in the classroom identifies as a "guy." By describing their switch from "hey guys" to "hey y'all," Professor Blake's learning assistant is giving advice to Professor Blake by setting an example of how to use more inclusive language. Though the piece of feedback wasn't a specific recommendation for Professor Blake, Professor Blake's learning assistant is providing input by highlighting their behavior to bring awareness to inclusive language, which can make students feel more comfortable in a class. We see the category of empathy as different from accessibility in that accessibility focuses on designing changes within the classroom so that everyone can participate, whereas empathy takes into account students' identities and emotions beyond the classroom.

F. Broad Feedback
Finally, the category of broad feedback can be defined as a response or observation that addresses multiple categories of feedback at once. Broad feedback can also be a response or observation that holistically describes the state of the class. This type of feedback can be identified by looking for any general statements made by learning assistants that are not tied to a specific experience or incident. For example, learning assistant Kai recounted the following interaction between them and their partner instructor: Kai: We had our prep meeting, she was asking, "Oh, what do you think? How do you think the students are feeling about this?" And we said... We were just upfront with her, we were like, "They're really confused." The input given by Kai is an example of broad feedback as Kai's feedback is a general statement that gives the professor a surface-level understanding of how students are doing in the classroom. The statement, "they're really confused" is general and not made about a particular student or group of students. Rather, it addresses the state of the whole class.
In the cases we found broad feedback, it did not live in isolation. Once broad feedback is given, learning assistants tend to elaborate on that broad feedback with feedback that fell into another category. Immediately after Kai's previous quote, they elaborate with feedback on student experience: Kai: ...In our breakout rooms they had a lot of questions, we weren't really able to go over the worksheet because they were really lost. After giving broad feedback, Kai further elaborated with justification and detail-that they felt that the students in class were confused based on specific encounters in class.

IV. DISCUSSION AND FUTURE WORK
The learning assistant-faculty relationship can lead to several forms of feedback. Within the interviews collected, both learning assistants and partner instructors described how the variety and depth of feedback impacted their students' expe-rience. From these examples, we were able to identify and define the following categories of feedback: student experience, classroom content, classroom structure, accessibility, empathy, and broad feedback.
While these feedback category definitions are preliminary, we hope that they can be used in future work to understand at a deeper level what is occurring within an instructor and learning assistant partnership. Though there is research on the benefits of learning assistants and their impacts on students, there is more work to be done in understanding the relationships between learning assistants and faculty partners. The categories that we have developed build on top of Jardine's work by finding subcategories of the types of feedback she defined and identifying new categories of feedback not explored in her work. Characterizing the feedback that learning assistants give to faculty is one facet of these relationships and complements previous research on LA-faculty partnerships [7][8][9]. Developing the language for describing the different forms of partnerships can help us to understand what impacts, if any, the learning assistant and instructor relationship has on what students experience within the classroom.
This research can also be beneficial to instructors and learning assistants. Being able to name the types of feedback can enable those in the LA-faculty partnerships to either give certain types of feedback or elicit certain types of feedback. We encourage instructors to think about the types of feedback that they solicit from LAs and whether they want to seek additional feedback. This framework may help learning assistant programs recruit instructors who may be seeking different types of feedback on their teaching. For learning assistants, understanding the types of feedback can help them gain awareness of the different types of feedback they can give, including some categories that they might not have known about. More knowledge about the diversity in learning assistant and faculty partnerships can help learning assistants and faculty envision new forms of partnership for themselves.
One area for further work is understanding the relationship between roles and types of feedback. Our data suggest a possible connection between the student role (as identified by Jardine [9]) and the feedback category of student experience. For example, we found instances where a learning assistant speaks of the students' experience in a classroom while positioning themself in a student role. Another area for further work is the relationship between different types of feedback. As discussed previously, broad feedback often leads to feedback that falls into other categories. Future research can understand the extent of this relationship and if there are patterns where one type of feedback leads to another type.
Finally, in order to do a deeper analysis of the relationships between categories and the relationships between feedback and roles, more data collection is necessary. We were only able to interview two learning assistants and two faculty members. New feedback examples will allow us to refine our framework further. By using a new form of data collection, such as observations, we may be able to gather more examples of feedback and refine our framework.