Observing teaching assistant differences in tutorials and inquiry-based labs

Through the use of the Real-time Instructor Observing Tool (RIOT) we examine the differences in actions of multiple TAs in mini-studios, which combine student-centered recitations with inquiry-based labs. TA actions observed include open or closed dialogue, passive or active observing, and clarifying or explaining to students. We observed five TAs teaching seven algebra-based first-semester physics labs to approximately 30 students per section. Individual TAs created an action profile that consists of the proportion of time spent on each action for that specific TA. These action profiles were found by averaging the duration of TA actions across multiple labs for a single TA. Surprisingly, with this method we found that TAs with a Learning Assistant tended to explain less and interact with the students less. We also found that there is a lack of consistency between the TAs in the overall time spent on each action. PACS: 01.40.-d, 01.40.Fk, 01.40.gb, 01.40.J-


I. INTRODUCTION
As more colleges and universities adopt interactiveengagement techniques, their efforts may not always yield the same positive results as the model course they are trying to emulate.For example, in his study of interactiveengagement methods, Hake found some institutions employing such methods had smaller than expected gains on concept inventories [1].Such differences may result from individual differences in implementation, which may stray from the intentions of the curriculum designers, as demonstrated in the diverse ways faculty use clickers in lectures [2].We may expect an even greater variation in implementation when interactive-engagement techniques are used in discussion sections and laboratories, which are typically led by graduate teaching assistants (TAs).While classroom approaches are evolving, educators must also be able to adapt the way in which they prepare the TAs to lead transformed classes.
Much like students bring prior knowledge and beliefs about physics into the classroom, TAs bring prior knowledge and beliefs about teaching to their teaching assignments [3].Thus, as classroom approaches evolve, we must also adapt the ways in which we prepare TAs.TA training meetings for classes with transformed curriculum require more than just going over the worksheet and lab for the next class [4]; rather, TAs need to learn about the design of the class in order for these methods to be implemented correctly.
In an effort to understand TAs' approaches to teaching a transformed curriculum and to create a preliminary baseline for our research, we use the Real-time Instructor Observing Tool (RIOT) [5] to quantify the amount of time TAs spend engaging in various actions in the classroom and compare this to the curriculum designers' expectations.

A. Research questions
Four questions guided our research and observations: 1) Is there consistency between the actions of individual TAs? 2) Is one observation a good indication of how a TA acts in general or are multiple observations necessary?3) Do TAs act differently when an undergraduate learning assistant (LA) is present?4) How do TAs' actions change between tutorial and lab portions of the class?

II. METHODOLOGY A. Participants and setting
We observed five TAs teaching seven sections of firstsemester algebra-based mini-studios with approximately thirty students each at the University of Central Florida (UCF).All TAs were male, and three of the five were firstyear graduate students.The first-year TAs and all LAs attended separate weekly professional development seminars for teaching in the fall, for TAs, or spring, for LAs.During the semester of our observations, one of the first-year TAs was leading a class for the first time, while the other four were teaching the same course for at least a second semester.Two of the five TAs, both with several semesters of teaching experience at UCF, were observed both with and without an LA present.The observations occurred each week between weeks ten and thirteen of a fourteen week semester.
The mini-studios at UCF consists of a 75 minute tutorial based on the University of Maryland Open Source Tutorials [6], a 15 minute quiz, and 80 minutes of lab based on the Investigative Science Learning Environment (ISLE) [7] curriculum.This course structure has been shown to result in higher gains on the Force Concept Inventory compared to traditional and studio courses at UCF [8].
The training meetings for the mini-studios, attended by all TAs and LAs for the course, took place during the week prior to teaching each unit.Our observations occurred during the spring semester.While the fall training meetings engaged the TAs and LAs in working through the tutorials like a student would, the spring training meetings simply reviewed the worksheets since most of the TAs had already worked out the tutorial.In both semesters, the labs were fully worked through in the meetings and most training meetings began with the opportunity to discuss any problems or comments on the previous class.

B. Coding TAs' actions with RIOT
The RIOT protocol organizes instructors' actions into four major categories: talking at students, shared student/TA dialogue, observing students, and not interacting with students.An observer watches the TA for specific actions and then codes these actions in an online applet in real time.The codes are designed to be mutually exclusive and exhaustive; at each point in time the TA is coded as engaging in one of these actions.A description of these actions and the coding rules are outlined in Table 1.
The observations were conducted by one undergraduate and one graduate researcher.Prior to taking data, the two observers agreed to 80% when observing the same classroom.Individual researchers' observations were binned into ten second intervals and compared using Cohen's Kappa.The researchers engaged in cycles of coding and discussion until they reached a mutual understanding of the coding rules and a Cohen's Kappa of 80%.
Once the data was taken, we determined each TA's action profile, which is the overall percentage of time each individual TA spent on the observed actions over the four weeks.We also determined the TAs' action profiles for the tutorial portion and the lab portion separately over all 4 weeks.Because the quiz should not involve the TA interacting with the students, the observers did not code during this time.

A. Consistency and one-time assessment
The TAs' four-week-averaged action profiles, excluding the sections with an LA so as to make valid comparisons, shown in Fig. 1, reveal little consistency among the five TAs.To further explore these trends, we binned the data into total minutes spent performing each action and used a contingency table χ 2 test.We find that there is a statistically significant difference with a small effect size among the TAs for time spent on the observed actions, χ 2 (28, N = 2813) = 485, p < .001,Cramer's V = .21.We expect that more effective TA training would lead to more similarities between the TAs' action profiles.
To predict what a favorable action profile would entail, we look at the design of the tutorials and ISLE labs.The  tutorials attempt to guide students to an understanding of physics concepts, taking a largely intuitive approach and building students up to physics concepts based on what they already know [6].In the ISLE labs TAs are not to provide students with physics concepts; rather, students are to observe phenomena and design an experiment on their own to test hypotheses based on their observations [7].From the design of the class we expect to see lower percentages of explaining and higher percentages of open dialogue or observing across all TAs.However, we find that explaining occurs most often in the mini-studios, and there is a large variation between every TA.This indicates that the training meetings were not effectively influencing TAs to teach in the intended manner.
Looking at Fig. 1, we can also see that the standard error of each action for each TA is fairly small.On average the standard error is ±1.4%.This suggests that one may observe an instructor once for an entire class and get a good sense of his or her overall teaching style.However, this finding has two important caveats.First, we note that our observations were conducted by student researchers, who may not have the same effect on the TAs as an authority figure such as the lab coordinator.Additionally, if the training meetings were more effective, we might expect that observations at the beginning and end of the semester would reveal different action profiles for the same TA.

B. With and without an LA
To compare TA actions with and without an LA, we analyzed each observed action of our two TAs with and without an LA using a Wilcoxon signed rank test and interpreted results with p < .05 as indicating a statistically significant difference in the percentage of time spent on an action.Statistically significant differences with a large effect size were found in explaining, passive observing, active observing, and not interacting (see Table 2).Looking at the significantly different actions for both TAs, seen in Fig. 2(a) and (b), we found that both TAs explained less, interacted with the students less, and observed more when an LA was present.The course designer intended LAs to provide additional student help since the mini-studios typically have a high student-instructor ratio (about 30:1).However, our data may suggest that the TAs we observed tended to use the LAs as a replacement to their teaching as opposed to a supplement.For both TAs, we see that observing occurred more often when an LA was present.This may suggest that only one group in the class needed help at a time, which the LA could have responded to.If no other group in the class needed attention, then the TA would resort to either observing or not interacting.However, it is the author's experience as a TA that often multiple groups need help at one time.It may also be the case that TAs and LAs together are responding to questions in a more efficient manner.The effect of which is more downtime for the TA resulting in more observing and less interacting.It is evident that more research and observation is needed to better understand the dynamic between our TAs and LAs.

C. Tutorial and lab differences
Lastly, we look at differences in TA actions between the tutorials and lab time.Figure 3 shows the difference in the percentage of time taken on each significant action, defined as an action where multiple TAs have a difference of more than 5% between the two halves of class.A positive value indicates that the action occurred more often during the tutorials.Explaining (all classes), open dialogue (three classes), and not interacting (five classes) were the only actions in which multiple TAs had a significant difference.We used a Wilcoxon signed rank test and found these differences were statistically significant, with a large effect  size (see Table 3).Explaining occurred more frequently across all TAs during the tutorials, while most TAs performed open dialogue and not interacting more often during the labs.While explaining is not preferred during the tutorials it is easy to understand that the option to explain something comes up more often when going over a concept or a word problem.Likewise, the labs give the TA an opportunity to ask questions like, "How do you plan on testing this?" and "How can you explain your results?" which merit a longer response from the student, resulting in open dialogue.

IV. CONCLUSIONS
The RIOT program is a powerful tool for describing instructors' actions for comparison to intended action profiles.We find two important results.First, the TAs observed tended to explain more than the course designers intended.Both portions of the class emphasize students developing concepts from their prior knowledge and experiences.TAs who frequently explain cut this process short.In respect of the TAs' beliefs [3] we consider that this could be due to the TAs' prior experiences learning and teaching physics, as well as their desire to make sure their students have a thorough understanding of the material.Second, we find that there is less consistency than we expected among the TAs' action profiles.This variation indicates that TAs are coming in with their own ideas about how to teach and the training meetings are not changing those ideas.In this study, analysis with RIOT revealed that TAs and LAs may be working individually in our ministudios, rather than working together as intended.We also found that "explaining" occurred more frequently during the tutorials than the labs, while the labs resulted in more open dialogue but less overall interaction.
As a continuation of this research, we will continue to use RIOT to observe the TAs throughout the semester in the mini-studios as we try to improve the training meetings to both better explain the intended teaching techniques and allow the TAs to practice those techniques.We will interview the TAs at the beginning and end of the semester to determine their background and their future plans or prior experiences in teaching the class.

Figure 1 .
Figure 1.Overall comparison for each TA without an LA.We compare the five sections without an LA.

Figure 2 .
Figure 2. The percentage of time spent on observed actions with and without an LA for two TAs.Only actions with p < .05are shown.(a) TA pseudonym Albert with and without an LA.(b) TA pseudonym Brady with and without an LA.

Figure 3 .
Figure 3.Time percentage difference between the tutorial and lab for all observed sections.

TABLE 1 .
[5]escription of the actions and coding rules for RIOT.See Ref.[5]for more details.

TABLE 2 .
Statistics of the significant difference in actions of a TA with and without an LA.

TABLE 3 .
Statistics of the significant difference of actions between tutorial and lab portions.