How traditional physics coursework limits problem-solving opportunities

A major goal of physics education is for students to develop problem-solving skills. To become expert problem-solvers, students need to deliberately practice those skills. In this analysis, we defined problem-solving skills as a set of 29 decisions that were previously identified as defining the problem-solving process of expert scientists. We quantified the amount of practice undergraduate physics students get at making each decision by coding the decisions required in assignments from introductory, intermediate, and advanced physics courses at a prestigious university. A research-focused capstone course was the only example that offered substantial practice at a large range of decisions. Problems assigned in the traditional coursework required only a few decisions and routinely removed potential opportunities for students to make other decisions. This analysis suggests that to better prepare undergraduates for solving problems in the real world, we must offer more opportunities for students to make and act on problem-solving decisions.


I. INTRODUCTION
STEM educators strive to teach problem-solving skills because scientists and engineers are often presented with complex problems that don't have definite solutions.The ability to solve such problems requires adaptive expertise [1], but training students to become adaptive experts has remained elusive.In a previous study, Price et al. [2] conducted cognitive task analysis interviews [3] to deconstruct the problemsolving process of skilled scientists and engineers.They identified a set of 29 decisions that are made at some point during the problem-solving process.These decisions are choices that need to be made, often based on limited information, between possible actions.Decisions include: what are relevant features of the problem?(D4 -important features), what information is needed to solve?(D13 -info needed), what calculations or data analysis are appropriate?(D16 -calculations), and how good is a potential solution?(D26 -reflect on solution) (see Table I for full list).
According to the principles of deliberate practice [4], for students to develop expertise at problem-solving, they need repeated practice and feedback on the individual sub-skills involved in problem-solving.Here we define those sub-skills as the 29 problem-solving decisions.Work by Burkholder et al. [5] found that chemical engineering students only begin to improve their performance on an assessment of these problem-solving decisions after they have taken a capstone research course or internship, where they presumably had to engage in many of the problem-solving decisions.Another study showed that students in introductory physics lab courses can learn critical thinking skills (closely related to problem-solving) when provided with "structured agency" by being instructed to make relevant decisions in their labs [6].These studies support the idea that problem-solving expertise requires practice making problem-solving decisions.
Are undergraduate students getting relevant practice at making problem-solving decisions?Physics and other STEM programs often include capstone or research experiences that allow students to make problem-solving decisions in authentic contexts, but these experiences are usually reserved for the end of college.Therefore, students' main opportunities to learn problem-solving skills occur in their more traditional courses.In traditional physics courses, students solve problems on problem-sets, in-class worksheets, and exams.However, these problems are unlikely to require the whole set of problem-solving decisions that define authentic problemsolving.This study set out to quantify what practice they do provide.Our research questions are: To what extent do homework problems give students opportunities to practice making the problem-solving decisions that define authentic problemsolving?How often do students need to make each decision, and how often is the decision-making opportunity taken away from students by the problem statement?We quantified the opportunities students were given to engage in each problemsolving decision by analyzing assignments given in intro, intermediate, and advanced physics courses.This work is related to extensive work on characterization of problems according to various features.Here, instead of classifying problems into categories, we analyze which decisions need to be made as a way of characterizing the process required to solve the problem.Decisions can be applied to other problem characterizations as well.For example, Chi and colleagues [7] found that novices and experts categorize problems differently: according to surface features vs. underlying concepts and solution approaches.This categorization relies on the solver's schema or mental model of the problem, which is reflected as making D4 (important features) and D5 (mental model) in the set of problem-solving decisions.The characterization of problems as well-defined (or well-structured) and ill-defined is particularly relevant [8].Problems that would be categorized as ill-defined require solvers to make D3 (goals), and likely D13 (what info need).Ill-defined problems are also described as having multiple possible solutions and solution paths [9], so would require solvers to make D8 (possible solutions) and D25-26 (reflect on approach, reflect on solution).Problem-solving transfer described by Mayer and Wittrock [10] depends on D7 (related problems).Other work defined a subset of ill-defined problems as complex problems, which are novel, complex, changing, and intransparent [11].Such complex problems would require most of the problem-solving decisions.Heller et al. [12] have presented "context rich" problems for physics instruction which call on students to make more decisions than in typical physics course problems, and they have outlined a 5 step solution strategy which involves making several of the 29 decisions we present here.Here we are extending previous work to examine exactly which decisions students are being called upon to make in the problems given in actual physics courses.

II. METHODS
We analyzed homework problems from physics courses that are part of the required mechanics sequence for physics majors at a prestigious university: introductory mechanics for freshman potential physics majors ("intro"), junior/seniorlevel intermediate statistical mechanics ("intermediate"), and a senior-level advanced Lagrangian mechanics ("advanced").As contrasts to the traditional problems used in these courses, we also analyzed the project from a research-based capstone laboratory course ("capstone"), and homework problems from another university's introductory mechanics course that were structured around a template designed to require problem-solving decisions ("template").The intro, intermediate, and template courses were taught in an active-learning style, while the advanced course was a traditional lecture.For consistency of comparison across courses, we analyzed homework problems only, not in-class work.For the capstone course, we analyzed the final research project.
Problem-solving decisions required to solve the homework problems were coded in an iterative process.The research team consisted of a senior undergraduate physics major, a non-physicist discipline-based education researcher with expertise in problem-solving decisions but less physics content knowledge, and a physics and DBER professor who also had expertise in problem-solving decisions.The student researcher solved each problem step-by-step, noting decisions made during the process (for the intermediate and advanced courses, these were solved in real-time while the researcher took the respective courses).For a subset of problems, the student researcher also cross-examined the instructor's solutions to identify any additional decisions that the instructor intended to be made.The research team then discussed the decisions taken when solving, how the other researchers would have approached the problem, and what phrasing in each problem prompted or removed decisions.We created a code book of definitions of each decision, with physics-problemspecific language and examples.These definitions were it- eratively updated after each discussion.The team discussed around 70% of the problems analyzed and reached a consensus on the decisions involved.The remainder (primarily from the intermediate and advanced physics courses) were coded just by the student researcher.We analyzed all homework problems from the intro, intermediate, and advanced courses, but only two problems from the template course because the template structure required the same decisions for all problems.To analyze the research project from the capstone course, we conducted a retrospective cognitive task analysis interview [3], in which the student researcher described their group's project, explained their problem-solving process, and presented their final report along with some of their analysis work.After the interview, the decisions involved were coded by the interviewer and verified by discussion.
We categorized decisions into two subsets: encountered or removed.Encountered decisions included prompted decisions in which students were explicitly instructed to make a decision as part of their solution; for example, "state any assumptions you need to make."Encountered also included unprompted decisions that students would be required to make on their own to successfully solve the problem.Removed decisions were explicitly made by the problem statement, preventing the student from needing to make that decision.Sometimes two or more decisions would be impossible to differentiate.For example, in the context of these problems D15 (plan) and D16 (calculations) were often the same decision because the planning (D15) required to solve these problems usually involved deciding which equations to use and which mathematical manipulations to apply (D16); in that case, we coded both decisions as required but noted they likely coincided.Figure 1 shows an example of our coding for an intro mechanics problem.We represented this data by calculating the percentage of problems in which each decision was en-FIG.2. Percentage of problems in which decisions were encountered (blue) or removed (yellow).All homework problems were analyzed from the traditional physics courses: Intro mechanics (n=26), intermediate statistical mechanics (n=41), and advanced Lagrangian mechanics (n=21).As contrasts, one project from the capstone research-based lab course was analyzed through retrospective think-aloud interview, and two "template" problems were analyzed from a problem-solving oriented intro course.countered or removed (see Fig. 2).This representation was used to quantify and compare the level of decision-making agency afforded to students for each set of coursework.If a decision was encountered in one part of a problem but removed in a different part of a problem, we counted the decision as being both encountered and removed.If a decision was encountered or removed more than once in a problem, we only counted it once.Of note, for any given problem, many decisions were not given any code -this means the decision was not prompted or explicitly removed and we did not have sufficient information to conclude how likely a student would be to choose to make that decision in solving the problem.Some decisions are easier to infer from a problem statement and student work than others, reflection decisions being particularly challenging unless they are explicitly prompted; this is a limitation of the analysis that future work conducting think aloud interviews should help clarify.

III. RESULTS
We found that the most commonly encountered decisions in undergraduate physics homework were D4 (features), D15 (plan), and D16 (calculations).Decisions D19 (results vs. predictions) and D26 (reflect on solution) were the mostly likely to be explicitly prompted.Meanwhile, the most commonly removed decisions were D10 (simplifications), D11 (decompose), and D15 (plan).When comparing across varying course levels, our data suggest that introductory physics courses eliminate decisions more frequently than intermediate and advanced courses, but none of these traditional courses require many problem-solving decisions.
Our data also reveals a gap in the range of available decisions between the traditional courses and the researchfocused capstone course, the latter offering a far wider diver- sity of decisions for students to make.In addition, we found that problems designed to follow a problem-solving decisions template indeed prompted many more decisions.On average, students only encountered 2.76 decisions per problem in the traditional courses and had 1.6 of decisions removed per problem.All three traditional courses were similar, with the intro course removing the most decisions on average per problem but also encountering the most (Table II).Interestingly, the intro course only required (unprompted) 2.1 decisions on average, which was less than than the intermediate (at 2.3), but included more prompted decisions, for more total encountered.As noted above, on any given problem, many decisions were not coded in any category -they were not explicitly prompted nor removed but we also did not believe making them was required to solve the problem.Although most "absent" decisions are not required for solving, some students could decide to put in extra effort and make them regardless.Some decisions that involve more reflection, like D7 (related problems) and D23-26 (reflect), were particularly difficult to identify based on the problem statement unless they were explicitly prompted or removed.Therefore, the amount of prac-tice with these decisions may be highly student-dependent.Other decisions, such as D1 (importance), D2 (fit), and D27 (implications) are unlikely to be relevant to most undergraduate level problem solving (these decisions were also not universally represented in the analysis of expert problem solving by Price et al. [2]).

IV. DISCUSSION
Our results show that the traditional physics courses (intro, intermediate, and advanced) we analyzed only provide students with practice at making a few key problem-solving decisions -D4 (features), D15 (plan), and D16 (calculations).Other decisions that are important for problem-solving in the real world are either not encountered or are eliminated by being explicitly made for the students.This is mostly due to the nature of the assigned coursework.In an attempt to guide students, curb difficulty, and/or ease grading, instructors may unintentionally remove decisions, reducing the amount of preparation students receive for scientific problemsolving.The courses we analyzed are standard requirements for undergraduate physics majors and span a range of difficulty levels, therefore we believe this work will likely generalize to other universities that have similar courses and assign textbook-style homework problems as the main problemsolving activity.Problems that specify assumptions or have a well-defined expected pathway for solving are particularly likely to remove decisions.However, it is possible to design problems and other learning experiences that involve more decisions, as demonstrated by the two contrasting examples analyzed (capstone and template).
The "template" problems that were specifically designed to prompt problem-solving decisions do indeed give students practice at making many more decisions, though still not the full set.In contrast, a research-focused capstone provided students with opportunities to make nearly every decision from the problem-solving decisions framework.That course was designed as a capstone experience to simulate an authentic research environment; requiring lots of decisions should be a design feature of a good capstone course.Other courses, not analyzed here, may also provide more decision practice, particularly if they involve a research element or include realworld problems that don't provide all necessary information up front.
Our analysis has some limitations because it is impossible to eliminate the subjectivity of a problem solver during the process of characterizing problem-solving decisions.We attempted to limit inconsistency by discussing a large portion of the problems and comparing student and instructor solutions.We are also in the process of validating our characterization through think-aloud interviews with multiple student solvers.We also stress that our analysis only coded for decisions that were clearly encountered or removed.Theoretically, students could put in a conscious effort to make additional decisions that were absent in our coding.Indeed, the student researcher occasionally did make decisions that we coded as "studentchosen" (not presented in this analysis) because they were not required to solve the problem but the student chose to do extra work (e.g., extra solution checking).Future work will involve think-aloud interviews to analyze which decisions multiple students make when solving these problems in real time.
Since the current curriculum for physics majors typically involves many traditional physics courses and very few research-focused courses (usually as capstone experiences), physics undergraduate students may not be receiving adequate preparation for solving real problems that they will encounter after they graduate.More research is needed to determine an optimal number or sequence of decisions for students to practice in an assignment, but we argue based on the principles of deliberate practice that students will need multiple opportunities to practice most of the decisions.Although it might not be feasible to have an introductory-level student practice all of the decisions in a single problem, they could be supported by gradually increasing the number of decisions required throughout a course.Our work shows that increasing course difficulty does not necessarily increase the number of opportunities students will have to make decisions, so problem-solving decisions need to be explicitly incorporated in the curriculum.
When designing a problem, it is important to consider both how many decisions are encountered and how many decisions are removed.Other research on problem-solving in physics has demonstrated that there can be unintended consequences of eliminating decisions about planning (D15) and representation (D17) [13,14].Our analysis of the problem-solvingtemplate problems demonstrates that it is possible to design problems for traditional courses that still provide students with opportunities to deliberately practice a reasonable portion of the problem-solving decisions.In cases where instructors feel students need more support in their problem-solving process, they can opt to prompt a decision rather than eliminate it for the students [15].Decision prompting has been demonstrated to be a viable approach to incorporate decisionmaking into introductory physics labs as well [6].When students are tasked with solving problems in their future careers (whether in physics or not), they will need to make nearly all the problem-solving decisions during their solution process.Therefore, undergraduate programs need to consciously give students more opportunities to practice making these decisions during their coursework.

TABLE I .
Problem-solving decisions characterized by Price et al.Note: numbers are for reference, not meant to imply a sequential order of decisions.

TABLE II .
Number of decisions encountered (Enc) or removed (Rem) per problem.