PERC 2024 Abstract Detail Page
Previous Page | New Search | Browse All
| Abstract Title: | Redesigning Legacy Conceptual Inventories: Using Evidence Centered Design to Develop Valid, Equitable, and Flexible Items in Kinematics and Dynamics |
|---|---|
| Abstract Type: | Talk Symposium |
| Abstract: | In this session, we reach out to the PER community to describe our emerging NSF-funded project, discuss our preliminary findings, and invite participants to give feedback and consider ways in which they might engage with the project. The overall goal of the project is to construct valid and equitable assessment instruments for kinematics and dynamics that are improvements over the existing legacy instruments. Constructed within Evidence Centered Design, the new assessment items will be grounded in learning and measurement theory, based on a construct-centered and community-based approach. The flexible instrument will include multiple scales measuring topics within kinematics and dynamics organized into subscales which instructors can utilize to build their own classroom assessment. The library of items will be extensively validated to ensure superior psychometric properties and instrumental fairness for women, underrepresented minority students, and first-generation college students. The session is in the format of a talk symposium with 3 invited presenters and a follow-up discussion. The first presenter will discuss an overview of the rationale, goals, and development strategy of the project. The second presenter will present initial results on pilot testing of 50 items with over 3000 students representing 3 universities. The discussion will include results of quantitative validation with productive examples of specific items. The third presenter will discuss the qualitative methods used, including interviews which prompt the student to think-aloud while solving problems, provide retrospective probes intended to fill in apparent gaps in stated thought processes, and elicit student opinions about the problems. We will share how the rich data are being used to revise and validate items and current lessons learned. Finally, we will prompt audience reflection and input on the project, the process, and ways they could productively participate in the project. |
| Footnote: | Funded by NSF IUSE award (#2235518) |
| Session Time: | Parallel Sessions Cluster 2 |
| Room: | Harbor I |
Author/Organizer Information | |
| Primary Contact: |
Andrew Heckler Ohio State University |
| Co-Author(s) and Co-Presenter(s) |
Rachel Henderson, Michigan State University John Stewart, West Virginia University Elaine Christman, West Virginia University |
Parallel Session Information | |
| Moderator: | Andrew Heckler, Ohio State University |
Symposium Specific Information | |
| Presentation 1 Title: | Using Evidence Centered Design to Develop Valid, Equitable, and Flexible Items for Conceptual Understanding of Kinematics and Dynamics in Introductory Physics |
| Presentation 1 Authors: | Rachel Henderson |
| Presentation 1 Abstract: | Commonly used legacy instruments, such as the Force Concept Inventory (FCI), have shown some serious flaws including substantial psychometric problems and demographic biases which make them inaccurate for some underrepresented student populations. Even with these flaws, these instruments have been vital to the development of Physics Education Research (PER) as a discipline; however, while the teaching of physics has evolved, the formative assessments researchers, instructors and, more broadly, physics departments use to evaluate their classrooms have not. In this talk, we will discuss the project goals and development strategy of our new NSF award (#2235518) intended to design a set of valid, fair, and flexible tool assessing conceptual understanding of kinematics and dynamics for introductory physics courses. Constructed within Evidence Centered Design, the new assessment items will be grounded in learning and measurement theory, based on a construct-centered and community-based approach. The flexible instrument, the Kinematics and Dynamics Assessment, will include multiple scales measuring foundational topics within kinematics and dynamics organized into subscales which instructors can utilize to build their own classroom assessment. The library of items will be extensively validated to ensure superior psychometric properties and instrumental fairness for women, underrepresented minority students, and first-generation college students. |
| Presentation 2 Title: | Redesigning Legacy Conceptual Inventories: Initial Results from the One-Dimensional Kinematics Subscale |
| Presentation 2 Authors: | John Stewart |
| Presentation 2 Abstract: | This project seeks to develop replacement instruments for the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE). The project seeks to identify and develop subscales representing different dimensions of a Newtonian Force Concept. The first scale to be developed measures a knowledge of one-dimensional kinematics. More than 50 items were piloted and tested at 3 large research universities in the Spring 2024 semester producing over 2000 student responses. This talk will discuss lessons learned in the validation process and present examples of both bad and good kinematics items. Results of quantitative validation will be presented. The validation process will be summarized and audience input on refining the process solicited. |
| Presentation 3 Title: | The Role of Cognitive Interviews in Redesigning Legacy Conceptual Inventories |
| Presentation 3 Authors: | Elaine Christman |
| Presentation 3 Abstract: | This project is underway to use evidence-centered design to create a replacement for the Force Concept Inventory (FCI) and Force and Motion Conceptual Evaluation (FMCE) comprising subscales developed with input from the physics education community. We have begun to collect qualitative data on the first of these subscales, covering concepts in one-dimensional kinematics, through three-step test interviews in which we ask students to think aloud while solving problems, give retrospective probes intended to fill in apparent gaps in observed behavior and stated thought processes, and then elicit student opinions about the problems. In this talk, we'll share how the rich data collected in these interviews are being used to revise and validate items, as well as what we've learned about best practices in writing items and developing figures based on the sometimes surprising ways students approach these problems. |
