home - login - register

PERC 2019 Abstract Detail Page

Previous Page  |  New Search  |  Browse All

Abstract Title: Learning by Analyzing More Than Just Correct Answers
Abstract: The great majority of quantitative work in PER is based on students correct answers (raw score) - for example when analyzing concept tests or classroom grades.   However the distractors on concept tests often represent commonly held alternate conceptions of physics.  Thus the particular distractors selected, and by extension students' textual responses, symbolic responses, and discussion posts contain additional information about their state of mind beyond 'not correct'.  This session showcases some  research analyzing responses to each distractor as well as analysis of student's free response writing.  Analyzing it can reveal specific student misconceptions and other knowledge that could help instructors tune instruction to address specific misunderstandings. This session highlights the power of modern analytics to reveal new educationally relevant insight, but also deals with the new tools and their potential pitfalls.
Abstract Type: Poster Symposium
Session Time: Parallel Sessions Cluster I
Room: Cascade C

Author/Organizer Information

Primary Contact: Dave Pritchard
MIT
Co-Author(s)
and Co-Presenter(s)
Benjamin Van Dusen, John Stewart, T. Smith, A. Traxler, Nasrine Bendjilali,

Symposium Specific Information

Moderator: Dave Pritchard
Presentation 1 Title: The Second Dimension of the FCI is Mostly Medieval
Presentation 1 Authors: Angel Perez Lemonche, John Stewart, Byron Drury, Rachel Henderson, Alex Shvonski, David Pritchard
Presentation 1 Abstract: In order to measure students' physics beliefs prior to instruction, we applied two-dimensional Item Response Theory (2DIRT) to all 150 pre-instruction responses to the Force Concept Inventory (FCI) with N = 17000. Examination of Item Response Curves (fraaction selecting a response vs raw score) showed an absence of random guessing because  students scoring below chance overall coalesced on only one or two distractors. One dimension of 2DIRT corresponded to Newtonian ability.  The second dimension corresponds closely to the number of frequently selected responses whose response curves maximized at intermediate raw score, over a dozen in total.  These responses embodied known commonsense physics ideas, most frequently the Medieval concept of impetus. The lowest Newtonian skill students selected a wider range of "wronger" responses. The ability to measure the detailed misconceptions of individual students or classes will allow development and application of instructional interventions for those specific misunderstandings.  In general classes with intermediate FCI scores believe in impetus in one or more of its guises.
Presentation 2 Title: Modernizing use of regression models in physics education research: a review of hierarchical linear modeling
Presentation 2 Authors: Ben Van Dusen and Jayson Nissen
Presentation 2 Abstract: Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (a.k.a., multi-level models) account for this hierarchical nested structure in the data. In this presentation, we outline the theoretical differences between how single-level and multi-level models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field's understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multi-level datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets.
Presentation 3 Title: Using IRT to rank incorrect responses FMCE questions
Presentation 3 Authors: Trevor I. Smith and Nasrine Bendjilali
Presentation 3 Abstract: Research-based assessment instruments (RBAIs) have been standard tools for measuring student learning in physics for decades. The prevalence and power of using RBAIs hinge on the incorrect "distractors" being designed to correspond with students' intuitive ideas, and typical analyses yielding a single number that is straightforward to calculate and easy to understand. Unfortunately, by (implicitly) considering all incorrect responses to be equivalent, these analyses throw away much of the rich information that students provide when they select responses corresponding with particular intuitive ideas. We present analyses of more than 7,000 students' responses to the FMCE. Using item response theory (IRT), we show that students' likelihood of selecting particular responses may be correlated with their overall understanding of Newtonian mechanics. We use IRT correlation parameters to rank incorrect responses and develop a scoring method that values what students know rather than focusing primarily on what they don't yet know.
Presentation 4 Title: Using Multidimensional Item Response Theory to Understand the FCI, the FMCE, and the CSEM
Presentation 4 Authors: Cabot Zabriskie, Jie Yang, Seth DeVore, and John Stewart
Presentation 4 Abstract: Constrained Multidimensional Item Response Theory (MIRT) is a powerful tool to understand the detailed structure of a multiple-choice instrument. A detailed model of the conceptual solution of the instrument developed by experts in the field can be mapped onto the MIRT model and the degree to which the expert solution models student thinking can be evaluated. Small, theoretically motivated, changes to the model are then explored to find an optimal model of student thinking. This process was applied to the FCI, FMCE, and CSEM. The structure of each instrument was dramatically different. This analysis suggested that the FCI was largely one-dimensional, while the FMCE contained multiple sub-facets. The CSEM models for two institutions were compared showing that the optimal models selected by MIRT were very similar but not identical across institutions.
Presentation 5 Title: Network Analysis of Students Descriptions of Scientific Research
Presentation 5 Authors: Adrienne Traxler, Carissa Myers, Jason Deibel, Meredith Rodgers
Presentation 5 Abstract: This work analyzes students' talk about their expectations and experiences of undergraduate research. The Applying Scientific Knowledge (ASK) program at Wright State University takes science and math majors in their second year, builds a cohort through a shared research methods class, then places students with faculty-mentored research projects. Students' accounts of their experiences are examined using epistemic network analysis, which focuses on connections between different conceptual elements in their responses. Here we give preliminary results from two physics majors, one early and one late in the program. We describe the conceptual elements present and the network structures that emerge from how they link ideas in their responses. The long-term goal of the analysis is to compile trajectories of how students in various disciplines think about and experience scientific research.