Student Satisfaction and Perceptions of Instructor Support in Studio Physics

This study investigates the pedagogical factors that impact students’ satisfaction with studio-style physics courses. Students enrolled in various sections of a SCALE-UP style course at EKU (N=184) responded to a reformed version of the Pedagogical Expectancy Violation Assessment (PEVA) and a course satisfaction questionnaire. Within the PEVA, students identified their satisfaction with particular activities based on how often they expected them to occur or how often they had experienced them. Based on experiences, the items in the PEVA clustered into five factors, one of which concerned the instructor support that the students received. We found that expected course grade and instructor were both correlated with course satisfaction, but even when those predictors were controlled, students’ satisfaction with instructor support was a significant predictor for course satisfaction but not for their expected course grade. Further, the more support students reported receiving, the higher their satisfaction. We discuss implications of these findings, especially in terms of how to respond to students desire for instructors to spend more time demonstrating how to solve problems on the board.


INTRODUCTION
Although active learning classes based on the SCALE-UP [1] model typically produce greater learning gains than traditional methods [2], student evaluations of such courses are sometimes mixed and, at times, discouraging.Faculty members who attempt such courses for the first time and experience student pushback and poor student evaluations may be discouraged from continuing with the reforms [3].On the other hand, having a class full of students who have embraced active learning reforms can make for an invigorating experience for everyone.Therefore, we are interested in understanding what causes students to have an overall positive or negative attitude about the class.

METHODS
At Eastern Kentucky University (EKU), introductory algebra-based physics is offered in 45-student SCALE-UP [1] style rooms and meets six hours a week for sixteen weeks.Classes integrate multiple modes of instruction, including hands-on activities, lecture, and problemsolving sessions.Students in five sections (Fall 2013: 4 sections; Spring 2014: 1 section) were surveyed twice.Instructor A taught a section each semester; each of the other three fall sections was taught by a different instructor.A total of 184 students (approximately 90% of en- Based on previous work [4], we modified the Pedagogical Expectancy Violation Assessment (PEVA) [5] to consist of 16 items representative of activities students did both during and outside of class (see Table 1).Notably, this version replaced the item "paying attention to lecture" with two items: "the instructor introducing new material to the class" and "the instructor demonstrating how to solve problems to the whole class." On the first survey (during the second week of the semester), students reported how often they expected to do each activity, reflecting on both their expectations when they enrolled for the course and their current expectations after a week of class.During the second survey (given during the fourteenth week), they reported their perceived experiences in the course.Students used a Likert scale that ranged from 0 (almost never) to 6 (almost constantly).
On both surveys, students were asked about their valence and preference for each activity.Valence came from students reporting how pleased they were with their expectations or experiences with each activity, ranging from very displeased (-3) to very pleased (+3).Preference came from students answering the prompt, "If I had my way, this activity would happen (Less/Same/More)." As part of the second survey, students were also given a 10-question measure of course satisfaction, an extension of previous work [4].Students answered one item inconsistently: "I would recommend to a friend to take a different version of the course," perhaps as a result of the reverse-coded phrasing.The Cronbach's alpha value when that item was dropped was 0.93.A total "satisfaction score" was created by adding the student responses to the remaining nine items; this overall score ranged from -27 to +27 and was considered to be continuous.
Demographic information was collected during both surveys.On the first survey, students indicated the minimum course grade with which they would be satisfied; during the second survey, students reported the final grade they expected to receive.In previous work, we found that students were good predictors of their grade [4].A total of seven students were interviewed, four from section 1 (instructor A) and three from section 3. Interviews typically lasted about 25 minutes.Responses to the interviews were used to identify weaknesses in the survey instrument and to get more detailed information about students' reasons for satisfaction or dissatisfaction in the course.

Factor Analysis
We found that 14 of the 16 items on the final survey clustered into factors based on students' perceived experiences in class (see Table 2).We used a Principal Component Analysis with Varimax rotation, selecting items into a factor when that item had a score of 0.60 or greater on that factor and no score above 0.40 on any others.Five factors were selected based on factors with eigenvalues greater than 1.0 and a visual inspection of the Scree plot.Factor 1 was particularly interesting because each item referred to activities that the instructor had at least limited control over and were related to ways instructors build relationships with students through coaching and support.Factors 2, 3, and 4 were about ways that students worked, and factor 5 was related to the teacher-centered portion of instruction.These factors were similar to those that emerged from an earlier version of the PEVA [4], but with an additional factor for student work outside of class.

Factor Valence, Satisfaction, and Grades
Student valence for each factor was calculated by simply averaging the valence scores for each item within the factor.Not surprisingly, valence for each factor was correlated with overall satisfaction in the course, each at the p < 0.001 level (r 1 = 0.67; r 2 = 0.46; r 3 = 0.66; r 4 = 0.66; r 5 = 0.64).We found that satisfaction was correlated with expected course grade (r = 0.33, p < 0.001) and varied by instructor (F = 11.7,p < 0.001) but was not related to student gender.Post-hoc testing revealed that course satisfaction was significantly higher for instructor A than the others, who were not significantly different from each other.However, there was no difference found between instructors in terms of students' predicted grades.
There was no relationship between students' valence on the instructor support factor and their expected grade in the course, which means that students who claimed more satisfactory levels of support as measured by those four items did not earn higher grades in the course.In fact, students' expected grades were not related to their valence for any factor other than factor 4. Students who were more pleased with the time they spent solving problems individually during class and working to understand physics concepts by themselves earned slightly higher grades (F = 3.5, p < 0.05).It was not generally true that students who reported spending more time on those activities estimated that they would earn a higher grade in the course.

Instructor Support
Because valence of experiences on the instructor support factor is strongly correlated with course satisfaction and activities within that factor should be fairly manipulable by instructors, we now focus on that factor exclusively.We expected and found that valence on that factor depended on the instructor (F = 11.0,p < 0.001), with significantly higher scores from instructor A's sections compared to sections 3 and 4, and significantly higher scores from section 2 compared with section 3.
Interviews with students from section 3 provide some insight into why students may rate those items low.Students claimed that the assistance offered by the instructors in the classroom for activities and problem-solving exercises were helpful.However, they did not happen nearly enough.The large size of the section (40 students), the relative inexperience of the assistant instructor, and the fact that the instructors worked with individual students for long stretches of time each limited the amount of support the students received.One student said that she stopped raising her hand during problem-solving sessions because she assumed the instructor would not get around to her anyway; consequently, she was left frustrated and unable to progress whenever she got stuck.She claimed that the instructor did not often go over problems on the board (item #8), further contributing to this student's feeling of being unsupported.
The interviewed student may have represented an extreme situation, but her example suggests a possible explanation for why students frequently complain about lack of lecture or, more precisely, not going over problems in class enough.Exams in all sections were largely problem-based, and students likely perceived class time as preparation for the exams.Hence, whenever the instructor demonstrated how to solve problems on the board, s/he was providing a form of personal support, which in some cases may have been an acceptable replacement for one-on-one interactions.Conversely, by choosing to spend considerable time with a small portion of the students and not solving problems for the whole class, the instructor was seen to be negligent, not providing what students believed to be necessary preparation.
The fixed factors "instructor" and "expected grade" were both significant predictors of course satisfaction (F = 12.1, p < 0.001, R 2 = 0.33).When we included valence scores for the instructor support factor as a covariate, we saw a substantial increase in the amount of variance explained by the model (F = 24.0,p < 0.001, R 2 = 0.55).There were no interaction effects between terms in either model.Because the instructor support factor was a significant predictor of course satisfaction even when we controlled for the instructor, we conclude that even within each section, students' level of satisfaction with instructor support contributed substantially to their over-all satisfaction in the course.That is to say, differences between the instructors did not account for all of the variance in terms of the perceived support individual students received.Thus, we need to understand what contributed to valence scores for the instructor support factor.
"More was better" for each item in the instructor support factor.That is, for each item in this factor, higher perceived frequency of occurrence generally meant higher valence for that item (r = 0.61, p < 0.001).This observation is verified by substituting students' total reported experience on that factor for their valence score in the second model above.In this case, we find that total experience is a predictor of course satisfaction without much loss of power (F = 22.4,p < 0.001, R 2 = 0.53).Students who claimed to have received more instructor support (greater frequency of items 4, 8, 9, and 10) had higher satisfaction in the course, even when controlling for instructor effects and expected final grades.
It is worth noting that three of the items that made up the instructor support factor (items 4, 9, and 10) were "individualized" activities.In other words, it's quite reasonable to expect that the instructor would speak with one student more frequently than another during class or provide more feedback or encouragement: while one of the purposes of a studio physics class is to increase the number of interactions between students and faculty, such interactions may not occur homogeneously.On the other hand, item 8 refers to an activity that occurs simultaneously for all students; when an instructor chooses to go over a problem on the board, she or he does so for the entire class.Why then does a difference in reported frequency exist between students in the same section?
One possible reason is that expectations can frame experiences.We found that students' reported frequency of experience for item 8 was independent of their expectations upon entering the course.However, after a week of the class, students who expected the instructor to go over problems on the board "very often" or "almost constantly" (a response of 5 or 6) reported that the instructor went over more problems compared to those with lower expectations (U(121) = 1255, p < 0.01, r = 0.28) and had higher reported satisfaction with both item 8 (U(121) = 1232, p < .001,r = 0.29) and the instructor support factor (t(119) = 2.6, p < 0.01).Because effect sizes are small, we should consider other explanations.
Differences between students in terms of both expectations and experiences may have to do with how they interpreted the "demonstrating how to solve problems to the whole class" item.For example, instructor A had students present their solutions to problems (written on whiteboards) to the rest of the class.He would then emphasize and reiterate aspects of that solution, often drawing key pictures or writing partial solutions, frequently skipping algebraic steps.Students who felt that doing so constituted "demonstrating how to solve problems" may have stated that it happened more often than students who were frustrated that the entire solution was not written out on the board.Likewise, it is possible that students (in any section) who were disappointed in the course may have "blamed" their frustration on a perceived lack of problem solving demonstrations, regardless of what the instructor actually did in class.It is thus difficult to be confident in any causal relationship between the reported frequency of instructor support and students' satisfaction without further research.
Finally, it is possible that the difference in reported experiences is related to a shift in epistemological stance regarding the appropriate way to learn physics.Perhaps as the semester progressed, some students' ideas about the role of the instructor and students developed such that by the end of the course, they believed that it was a better use of class time to work out problems themselves rather than watch an instructor demonstrate.To check this idea, we looked for cases where students reported a lower frequency on item 8 than they expected after orientation and who had changed their preference from wanting more problem-solving demonstrations to feeling that there were the appropriate amount.Only 12 (about 10%) of the paired surveys fit this description.Students in that group had a significantly higher overall satisfaction score than those who were not (t(20.5)= 4.5, p < 0.001).Thus, although the data are weak because such a small number of students seemingly "came onboard" this way, the results are consistent with our expectations that students who undergo a shift in beliefs about how to learn physics have higher overall satisfaction in the course.Notably, eight of the 12 who favorably changed their preference were enrolled in one of instructor A's sections.Further investigation would be needed to identify what, if anything, instructor A did to cause these shifts.Future work could also verify that epistemological shifts occurred by using established surveys such as MPEX [6] or CLASS [7].
Unfortunately, we received curious responses to the combination of valence and preference questions on the PEVA.A few students responded that they were "very pleased" with how often the activity was occurring but that they would want it "more" if they had their way.In interviews, at least one student revealed that she misinterpreted the valence question, instead responding with her overall satisfaction with the activity rather than her satisfaction with how often the activity happened, a subtle but very important distinction.It is also possible that some students interpreted the combination of questions differently from one survey to the next, which would reduce the validity of our results.In future implementations of the PEVA, we plan to clarify the wording.

CONCLUSIONS
When we control for students' instructor and expected grades, satisfaction on items related to the instructor support factor predicted a large part of students' overall satisfaction in the course.Further, the more frequently they perceived activities within that factor occurring, the greater their overall satisfaction tended to be.It may be tempting to interpret the results of this study to say that the way to improve satisfaction in studio physics is to understand and meet students' desires regarding how they want to be taught.Certainly for some items in the instructor support factor, one would be well suited to heed that advice and thus provide as much face time, encouragement, and feedback as possible.However, one purpose of studio instruction is for students to spend class time working through problems while instructors are available for help rather than for instructors to spend class time working examples.For an instructor to "give in" to students' wishes and spend more time working problems for them would weaken the pedagogical value of active learning classrooms.An alternative is to try to shift students' pedagogical expectations and epistemic values, perhaps through explicit framing of the instructional approach.While we saw some limited evidence that such shifts are possible, further work is necessary to verify the existence of such shifts and, if so, to determine factors that contribute to those shifts.In general, instructors should take advantage of opportunities to build supportive relationships with their students.This study suggests that while doing so successfully may not directly result in higher grades, it should lead to more satisfied students.

TABLE 1 .
Items used on the modified PEVA.

TABLE 2 .
(7,sters of activities for studio-based physics at EKU based on frequency of experience.Two items(7, 14)did not factor into a cluster.