Investigating the effects of course structure on students’ sense of belonging and course performance

,


I. INTRODUCTION
Physics and science education researchers have repeatedly determined measurable differences in students' learning in traditional lecture-based courses and research-based instructional strategies [1,2]. These research-based instructional strategies span a wide range of course structures, which have been further diversified by the need for remote and hybrid instruction during the COVID-19 pandemics. However, comparisons of research-based instructional strategies remain relatively infrequent and are complicated by factors such as the large timescales to complete studies and differences in populations. In this paper, we compare students' course performance and sense of belonging in three different course structures in a single introductory course required of nearly all students at the university.
Sense of belonging is generally regarded as an important factor in students' success at universities (e.g., Refs. [3][4][5][6]). In physics education, studies indicate that students' sense of belonging contributes to success in introductory physics courses and persistence in physics [3,7]. For example, Rainey et al. found that students who persist in physics majors report greater sense of belonging than those who choose to leave.
Here, we focus on students' sense of belonging in a single physics course and narrow our focus to identifying course structures that support and hinder students' sense of belonging. Within large introductory science courses, research hints that students sense of belonging may be tied to various course structures. For examples, Edwards et al. found that students' sense of belonging in a general chemistry course may change during a semester according to students' course performance [8]. Kepple and Coble found that students cite both interpersonal relationships with their group and their lab work as significant contributors to their sense of belonging in a physics course [9]. Wilton et al. identified that a research-based biology course with many different required activities correlated with students' increased sense of belonging compared to a traditional lecture course [10]. These findings suggest that different forms of research-based instruction may influence students' sense of belonging.
In this study, we compare students' overall course performance and sense of belonging in a course that implemented three different course structures over four semesters of instruction in a calculus-based introductory electricity and magnetism course. All course structures used research-based instructional practices by regularly incorporating both individual and small group problem solving. We find that: (1) a course structure with lower stakes assessment corresponds with both increased students' letter grades and sense of belonging, (2) hybrid instruction compared to in-person instruction did not appear to correspond to a difference in students' sense of belonging, and (3) reliably disentangling the effect of course structure on students' sense of belonging is greatly complicated by their earned letter grade.

II. METHODS
We analyzed data collected from two academic years in an introductory calculus-based electromagnetism (E&M) course at an engineering-focused university. Nearly all undergraduate students at the university are required to take the two-semester sequence of mechanics and E&M, except for those with AP or IB credit. Therefore, student demographics in the course are nearly identical to the demographics of the university, where about 70% of students are men and 30% are women, according to the university registrar data that allows only a binary election of gender identity.
The course structure of this E&M course consists of an out-of-class preparatory lecture video of the new physics content that is followed by an interactive one-hour lecture with extensive discussion among students and back-andforth questions with an instructor, called a discussion. In Year 1 of this study, students had the option of attending discussion either in-person or remotely. In Year 2, all discussions were fully in-person. The following day, students work in groups during a two-hour Studio that includes problem-solving and lab activities. For the Studio, due to the ongoing COVID-19 pandemic, in Year 1, students rotated through one group member attending in-person and the other group members attending remotely and worked with each other through Zoom or Teams. In Year 2, all students were required to participate in the physical classroom, with a few exceptions for students in quarantine or isolation. In both years, the students worked together in groups of typically three people in Studio.
As part of an ongoing effort to improve this course, in the second semester of Year 2, the instructional faculty restructured the course to replace three midterm exams (mostly multiple choice) with seven quizzes (entirely free response). Midterm exams were 90 minutes long, and the final exams were 120 minutes long. Exams were primarily administered in-person in all semesters, however, in the Year 1 semesters, students who elected to be fully remote and students who were in isolation or quarantine were video proctored as they took an online version of the exams.
In previous semesters, students were self-reporting high levels of test anxiety on post-surveys and informally conveying that the midterm exam structure was not supporting their learning in the ways that faculty had intended. The change in course structure was intended to improve students' learning by shifting the grading focus to the justification of steps in the problem solving process and to reduce effects of test anxiety by having more frequent assessments that cover less material. In all semesters, there was a cumulative final exam, worth 15% of the students' overall grade, that consisted of 20 multiple choice problems. Therefore, we have multiple course structures to compare students' sense of belonging between: (1) Year 1 when students had the choice of and/or rotation between in-person and remote participation and the midterm exam structure was in place, (2) Year 2 Semester 1 (S1) when students attended fully in-person and the midterm exam structure was in place, and (3) Year 2 Semester 2 (S2) when students attended fully in-person and the quiz structure was in place.
To measure students' sense of belonging in a course, we used items developed by Singh's group, presented in papers such as Refs. [11][12][13][14]. For example, one of the five items pertaining to sense of belonging prompts students about their level of agreement (with five points) with the statement "I feel like I can be myself in this physics class." The scores presented in this paper are means across the items targeting students' sense of belonging, so possible scores range from 1 (weak sense of belonging) to 5 (strong sense of belonging). Students received extra credit for completing the survey, which consisted of a more extensive set of items including one item to choose whether or not to allow the data to be used for research purposes. We administered the survey through the learning management system near the end of the course in all semesters.
To analyze the survey data in this paper, we use descriptive statistics to demonstrate to the reader the proportion of students for each analysis. These proportions are particularly drastic when comparing students' sense of belonging in different course structures, which is the focus of our study. Student participant numbers vary by semester: Year 1 S1 has 387 student participants, Year 1 S2 has 299, Year 2 S1 has 433, and Year 2 S2 has 349. These participant number represent approximately twothirds of students enrolled in each of the courses.
During Year 2 S2, we interviewed 14 students to discuss their experiences in the course that semester. The interviews were one-on-one with the first author of this paper. Students volunteered to participate in the interviews with no inducements. We intentionally selected participants to hear from students who experienced the course in different ways (e.g., repeating the course, prior physics experience, self-reported grade). We audio recorded the interviews and used a live transcribe feature to generate transcriptions. Here, we briefly report preliminary results based on opinions that were shared by most participants. In the future, we intend to perform more rigorous and detailed analyses but wanted to provide descriptions of students' experiences to supplement the quantitative data that is the focus of this paper.

A. Grade distribution shifts between semesters
As shown in Fig. 1, letter grade distributions in Year 1 (hybrid instruction with midterm exams) were consistent between S1 and S2. However, Year 2 had noticeably different grade distributions. In Year 2 S1 (fully in-person with midterm exams), a greater proportion of students received letter grades less than B than in either of the Year 1 semesters. In Year 2 S2 (fully in-person with quizzes), the vast majority of students earned an A or B.
The hybrid structure of Year 1 yielded higher grades than the fully in-person structure of Year 2 S1. We suspect this is due to the administration of both in-person and remote exams simultaneously in Year 1. For example, one of the remote exams in Year 1 S1 had extra problems compared to the in-person version of the exam. The course coordinator added points to everyone's score, so many students exceeded 100%. In Year 2 S1, remote administration of midterm exams was not allowed by the university, which alleviated many grading issues. Alternatively, the hybrid structure may support students' learning better than requiring all students to participate fully in-person. Perhaps, the ability to participate from a chosen location allowed students to focus on learning more than in a large classroom with many other students around.
The quiz structure in Year 2 S2 drastically shifted the distribution of letter grades compared to all previous semesters. The fraction of students earning an A letter grade greatly increased compared to all implementations of the course with midterm exams. Students' performance on quizzes significantly exceeded performance on midterm exams in previous semesters. We suspect that improved performance compared to midterm exams may have resulted from the structures of the assessments. For examples, the quizzes had a narrower focus compared to midterms. The free response format of the quizzes allowed for partial credit, which was not available on midterms with mostly multiple choice questions. The grading emphasis for the quizzes was on explaining the process, which may have resulted in improved learning compared to the midterms that required students to reach correct answers.  Figure 2 shows the distribution of students' sense of belonging for each course. Distributions for both semesters in Year 1 and Year 2 S1 are relatively similar. Year 2 S2 has a larger portion of students reporting a strong sense of belonging in the course.

B. Sense of belonging shifts between semesters
Comparing courses with similar midterm exam structures, in Year 1, we suspected that students' sense of belonging in the course would be lower than that of Year 2 S1 because of hybrid course participation. With the hybrid structure, most students could only attend their Studio in-person once every three classes. Therefore, we expected that students would feel weaker sense of belonging compared to a semester that all students attended fully in-person. However, Year 2 S1 appears to show students experiencing a slightly weaker sense of belonging than in Year 1, directly contradicting our expectations.
In Year 2 S2, students' stronger sense of belonging may result due to a variety of reasons. For example, for the first semester since the start of the pandemic, most students took their introductory mechanics course fully in-person. Therefore, students had experience working in a similar environment to this E&M course and could have been more comfortable than students who changed modalities. On the other hand, perhaps the increased sense of belonging was due to the change in course structure to quizzes instead of the exams.
C. Sense of belonging relationship with course performance Figure 3 shows students' mean sense of belonging according to their earned overall letter grade in the course. Students who earned higher grades in each course tend to indicate stronger sense of belonging, similar to the find- ings of Edwards et al. [8]. In Year 2 S2, despite the larger proportion of students earning a higher grade (Fig. 1), students' mean sense of belonging for letter grades of A, B, and C were consistent with prior implementations.
With this, we cannot reliably disentangle the effect of course structure on students' sense of belonging. Students' sense of belonging was higher overall in Year 2 S2 (Fig. 2), but from the data presented here, we are unable to identify the specific reason for the increase in students' sense of belonging. Students could feel less test anxiety with the quiz structure, which could lead to better performance on quizzes and improved learning. Due to the frequency of quizzes, students may have had a better sense of how they were doing in the course and were motivated to continue to perform well. Perhaps the quizzes were much easier for students than multiple choice questions on exams and consistently earning higher grades led to improved sense of belonging in the course because grades may be viewed as an external signal of belonging.

D. Preliminary results from interviews
As a way to attempt to identify mechanisms that help and hinder students' sense belonging in the course, we interviewed 14 students who were actively taking the course in Year 2 S2. One of us (D. T. L.) interviewed student volunteers one-on-one. After soliciting for volunteers through a message on the learning management system and in-class announcements, students reached out to D. T. L. via email. Here, we present a few opinions that the students expressed comparing the quiz structure to midterm exams they heard about from peers or experienced themselves.
Students frequently mentioned that they discussed the course with their peers, which shaped their incoming perceptions about the course difficulty. One student mentioned that before taking the course they "knew the exams were tough and I had friends who said it was rig-orous and hard, so having the quizzes helped me out a lot." Another student mentioned that "quizzes are pretty nice. They're not as stressful as tests because I heard some bad things last semester. But the quizzes, I think, provide a less stressful time for you to put what you learn from the homework and put that test." The students who conveyed that they heard stories of previous semesters of the course from their peers tended to express negative attitudes toward the previous midterm course structure. However, students expressed less stress and better performance than they expected with the quizzes.
Interestingly, two interviewed students took the course in Year 1 S2 with the hybrid and midterm course structure and then retook the course in Year 2 S2 with the entirely in-person and quiz structure. Both students said their stress levels were greatly reduced due to the implementation of the quiz structure. One student indicated that "this year, it's been less stressful because I get more chances. I get seven quizzes and a final for that 60%, so it's been a lot less stressful. The quizzes are less stressful compared with the final usually. If I could have a say, I would recommend the quizzes. As I said, it was it's been a lot less stressful, and I feel like I can take more time to understand. The last time I took [the course]; I was trying to cram a lot before every exam." Our preliminary results from the interviews suggest that many student found the quiz structure to be less stressful than the midterm exam structure. As we analyze the interviews, we intend to pay attention to the reasons why students believe the stress was lessened. Perhaps, the higher frequency lowered the stakes of any one individual assessment. Or, the free response format allowed students a better chance to demonstrate what they did and did not understand. Or, the narrowed focus on a smaller number of topics made the workload more manageable for studying.

IV. DISCUSSION
This study found that a course structure with quizzes corresponded to higher overall letter grades and a stronger sense of belonging in the course than course structures with midterm exams. Additionally, on average, students who earned higher overall letter grades reported a stronger sense of belonging; these means, when segregated by letter grade, were relatively consistent across different course structures. Surprisingly, students' sense of belonging did not change drastically from a hybrid course structure to being entirely in-person when the midterm exam structure was in place.
One of the limitations of the data is a considerable selection effect. Students selected whether or not to participate in research studies using the survey and grade data presented here. Compared with overall class statistics, students who earned higher grades in the course elected to participate at much higher rates than those earning lower grades, with about two-thirds of all students participating in the research. Figure 1 does not accurately represent the actual distribution of grades in the course; for example, students earning letter grades of A are more represented than students earning Ds. The decision to participate was prompted at the beginning of the semester, so students who felt a weaker sense of belonging in physics courses to begin may have opted out of participation.
Another limitation is the timing of the administration of the survey: We only measured students' sense of belonging at the end of the course. At this point, most students have an idea of their letter grades in the class. Therefore, their expected grade may influence how they answer questions about their experiences in the course. Alternatively, a student may have earned a higher letter grade because of their heightened sense of belonging in the course. However, we are unable to comment on causality in this study. Consequently, we recommend that future work tracks students' sense of belonging multiple times throughout a term using various data sources and aims to determine whether and how students' grades may impact their sense of belonging in a course.
As we further analyze the interviews with students in Year 2 S2, we intend to explore the interactions between students' sense of belonging, the factors that help and hinder them, and students' self-reported identities and responsibilities with their experience in the course. For example, one student noted "one thing I have found helpful during my physics experience at [this university] is seeing people of color as professors and TAs. As a person of color, seeing someone who looks more like myself in positions of power is inspiring. I am sure other students feel the same, as it can often make students of color relate to their instructors deeper." These individual experiences may provide deeper insights into the specific structures that currently encourage and inhibit students' sense of belonging in their physics courses, physics as a whole, or their university community.
We also view this study as a comparison between different research-based instructional settings with similar populations of students. In our case, we cannot disentangle many of the various complexities that exist, even with just a simple change from midterm exams to quizzes. Students' attitudes and beliefs about their physics courses can be affected in many ways, and these may or may not directly impact students' learning and affective experiences. In future work, we aim to investigate other affective experiences in these courses through surveys and interviews and compare them to students' understanding of course material. We hope to provide an avenue for physics education researchers to understand students' experiences in different research-based instructional settings from student learning and student affect perspectives.

ACKNOWLEDGMENTS
We thank N. G. Holmes for assistance in the visual presentation of the results.