PERC 2019 Abstract Detail Page
Previous Page | New Search | Browse All
| Abstract Title: | Learning by Analyzing More Than Just Correct Answers |
|---|---|
| Abstract: | The great majority of quantitative work in PER is based on students correct answers (raw score) - for example when analyzing concept tests or classroom grades. However the distractors on concept tests often represent commonly held alternate conceptions of physics. Thus the particular distractors selected, and by extension students' textual responses, symbolic responses, and discussion posts contain additional information about their state of mind beyond 'not correct'. This session showcases some research analyzing responses to each distractor as well as analysis of student's free response writing. Analyzing it can reveal specific student misconceptions and other knowledge that could help instructors tune instruction to address specific misunderstandings. This session highlights the power of modern analytics to reveal new educationally relevant insight, but also deals with the new tools and their potential pitfalls. |
| Abstract Type: | Poster Symposium |
| Session Time: | Parallel Sessions Cluster I |
| Room: | Cascade C |
Author/Organizer Information | |
| Primary Contact: |
Dave Pritchard MIT |
| Co-Author(s) and Co-Presenter(s) |
Benjamin Van Dusen, John Stewart, T. Smith, A. Traxler, Nasrine Bendjilali, |
Symposium Specific Information | |
| Moderator: | Dave Pritchard |
| Presentation 1 Title: | The Second Dimension of the FCI is Mostly Medieval |
| Presentation 1 Authors: | Angel Perez Lemonche, John Stewart, Byron Drury, Rachel Henderson, Alex Shvonski, David Pritchard |
| Presentation 1 Abstract: | In order to measure students' physics beliefs prior to instruction, we applied two-dimensional Item Response Theory (2DIRT) to all 150 pre-instruction responses to the Force Concept Inventory (FCI) with N = 17000. Examination of Item Response Curves (fraaction selecting a response vs raw score) showed an absence of random guessing because students scoring below chance overall coalesced on only one or two distractors. One dimension of 2DIRT corresponded to Newtonian ability. The second dimension corresponds closely to the number of frequently selected responses whose response curves maximized at intermediate raw score, over a dozen in total. These responses embodied known commonsense physics ideas, most frequently the Medieval concept of impetus. The lowest Newtonian skill students selected a wider range of "wronger" responses. The ability to measure the detailed misconceptions of individual students or classes will allow development and application of instructional interventions for those specific misunderstandings. In general classes with intermediate FCI scores believe in impetus in one or more of its guises. |
| Presentation 2 Title: | Modernizing use of regression models in physics education research: a review of hierarchical linear modeling |
| Presentation 2 Authors: | Ben Van Dusen and Jayson Nissen |
| Presentation 2 Abstract: | Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (a.k.a., multi-level models) account for this hierarchical nested structure in the data. In this presentation, we outline the theoretical differences between how single-level and multi-level models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field's understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multi-level datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets. |
| Presentation 3 Title: | Using IRT to rank incorrect responses FMCE questions |
| Presentation 3 Authors: | Trevor I. Smith and Nasrine Bendjilali |
| Presentation 3 Abstract: | Research-based assessment instruments (RBAIs) have been standard tools for measuring student learning in physics for decades. The prevalence and power of using RBAIs hinge on the incorrect "distractors" being designed to correspond with students' intuitive ideas, and typical analyses yielding a single number that is straightforward to calculate and easy to understand. Unfortunately, by (implicitly) considering all incorrect responses to be equivalent, these analyses throw away much of the rich information that students provide when they select responses corresponding with particular intuitive ideas. We present analyses of more than 7,000 students' responses to the FMCE. Using item response theory (IRT), we show that students' likelihood of selecting particular responses may be correlated with their overall understanding of Newtonian mechanics. We use IRT correlation parameters to rank incorrect responses and develop a scoring method that values what students know rather than focusing primarily on what they don't yet know. |
| Presentation 4 Title: | Using Multidimensional Item Response Theory to Understand the FCI, the FMCE, and the CSEM |
| Presentation 4 Authors: | Cabot Zabriskie, Jie Yang, Seth DeVore, and John Stewart |
| Presentation 4 Abstract: | Constrained Multidimensional Item Response Theory (MIRT) is a powerful tool to understand the detailed structure of a multiple-choice instrument. A detailed model of the conceptual solution of the instrument developed by experts in the field can be mapped onto the MIRT model and the degree to which the expert solution models student thinking can be evaluated. Small, theoretically motivated, changes to the model are then explored to find an optimal model of student thinking. This process was applied to the FCI, FMCE, and CSEM. The structure of each instrument was dramatically different. This analysis suggested that the FCI was largely one-dimensional, while the FMCE contained multiple sub-facets. The CSEM models for two institutions were compared showing that the optimal models selected by MIRT were very similar but not identical across institutions. |
| Presentation 5 Title: | Network Analysis of Students Descriptions of Scientific Research |
| Presentation 5 Authors: | Adrienne Traxler, Carissa Myers, Jason Deibel, Meredith Rodgers |
| Presentation 5 Abstract: | This work analyzes students' talk about their expectations and experiences of undergraduate research. The Applying Scientific Knowledge (ASK) program at Wright State University takes science and math majors in their second year, builds a cohort through a shared research methods class, then places students with faculty-mentored research projects. Students' accounts of their experiences are examined using epistemic network analysis, which focuses on connections between different conceptual elements in their responses. Here we give preliminary results from two physics majors, one early and one late in the program. We describe the conceptual elements present and the network structures that emerge from how they link ideas in their responses. The long-term goal of the analysis is to compile trajectories of how students in various disciplines think about and experience scientific research. |
