PERC 2024 Abstract Detail Page
Previous Page | New Search | Browse All
| Abstract Title: | Analyzing the responses of GPT-4 to permutations of a standard mechanics problem |
|---|---|
| Abstract Type: | Contributed Poster Presentation |
| Abstract: | We will present an analysis of OpenAI's GPT-4 Large Language Model (LLM) responses to different versions of a basic physics problem that involves an object moving down an inclined plane. The permutations of the problem varied depending on the object involved, the verb used, and the characteristics of the incline, allowing for sliding, rolling, or scenarios that are not physically possible. The analysis of the LLM's responses focused on the problem's setup, the assumptions it made, and its problem-solving methods. When determining whether an object should roll or slide, its inconsistencies compared to the expert view differed from the inconsistencies a group of twenty introductory physics students showed. For example, for abstract objects like a basketball, it chose rolling motion much less commonly than for geometric objects like a solid sphere. While nearly half of the LLM's responses agreed overall with those of experts, it generally failed to articulate all the necessary assumptions and struggled to address contradictory scenarios properly. |
| Session Time: | Poster Session 2 |
| Poster Number: | B90 |
Author/Organizer Information | |
| Primary Contact: |
Ralf Widenhorn Portland State University Portland, OR 97201 Phone: 5037253898 |
| Co-Author(s) and Co-Presenter(s) |
Ryan Sissons, Portland State University Justin C. Dunlap, Portland State University |
