home - login - register

PERC 2022 Abstract Detail Page

Previous Page  |  New Search  |  Browse All

Abstract Title: Machine learning for automated content analysis: characteristics of training data impact reliability
Abstract Type: Contributed Poster Presentation
Abstract: Natural language processing (NLP) has the capacity to increase the scale and efficiency of content analysis in Physics Education Research. One promise of this approach is the possibility of implementing coding schemes on large data sets taken from diverse contexts. Applying NLP has two main challenges, however. First, a large initial human-coded data set is needed for training, though it is not immediately clear how much training data are needed. Second, if new data are taken from a different context from the training data, automated coding may be impacted in unpredictable ways. In this study, we investigate the conditions necessary to address these two challenges for a survey question that probes students' perspectives on the reliability of physics experimental results. We use neural networks in conjunction with Bag of Words embedding to perform automated coding of student responses for two binary codes, meaning each code is either present (positive) or absent (negative) in a response. We find that i) substantial agreement is consistently achieved for our data when the training set exceeds 600 responses, with 80-100 responses containing each code and ii) it is possible to perform automated coding using training data from a disparate context, but variation in code frequencies (outcome balances) across specific contexts can affect the reliability of coding. We offer suggestions for best practices in automated coding. Other smaller-scale investigations across a diverse range of coding scheme types and data contexts are needed to develop generalized principles.
Session Time: Poster Session 2
Poster Number: II-30
Contributed Paper Record: Contributed Paper Information
Contributed Paper Download: Download Contributed Paper

Author/Organizer Information

Primary Contact: Rebeckah K. Fussell
Cornell University
Ithaca, NY 14850
and Co-Presenter(s)
Ali Mazrui (he/him), Cornell University
N. G. Holmes (she/her), Cornell University

Contributed Poster

Contributed Poster: Download the Contributed Poster