PERC 2025 Abstract Detail Page
Previous Page | New Search | Browse All
| Abstract Title: | Assessing a combined human coding and natural language processing method to qualitative analysis in physics education research |
|---|---|
| Abstract Type: | Contributed Poster Presentation |
| Abstract: | As artificial intelligence becomes increasingly integrated into education research, it is critical to assess the capabilities and limitations of such tools in complex disciplinary contexts. This study explores the effectiveness of natural language processing (NLP) for topic identification using the \texttt{BERTopic} package to analyze undergraduate quantum mechanics syllabi. By comparing AI-generated topics with human-coded themes from a corpus of syllabi from 50 US institutions, we find that while NLP reliably identifies broad and structural elements of syllabi (e.g., policies, textbook references), it is limited in its ability to capture the nuanced and context-dependent content typical of upper-level physics instruction. The AI-derived topics were often underrepresented and misaligned with expert human interpretation, particularly in areas requiring deep disciplinary knowledge. Our findings support the use of NLP as a complementary tool for theme generation and cross-validation, but emphasize the continued necessity of expert human analysis for rigorous and meaningful educational research. |
| Session Time: | Poster Session A |
| Poster Number: | A-1 |
| Contributed Paper Record: | Contributed Paper Information |
| Contributed Paper Download: | Download Contributed Paper |
Author/Organizer Information | |
| Primary Contact: |
Alexis Buzzell University of Utah Salt Lake City, UT 84106 Phone: 4138340256 |
| Co-Author(s) and Co-Presenter(s) |
Tim Atherton Ramon Barthelemy |
Contributed Poster | |
| Contributed Poster: | Download the Contributed Poster |
