Relationship of Scientific Reasoning to Solving Different Physics Problem Types

In recent years researchers have compared scientific reasoning abilities of students in introductory physics courses with gains in conceptual learning. This research suggests students with more formal reasoning patterns are more proficient learners. However, little has been done to investigate how scientific reasoning abilities relate to ability to solve problems based on the application of simple algorithms and those which depend on conceptual understanding. In this pilot study we compare student scientific reasoning abilities, as measured by Lawson's Classroom Test of Scientific Reasoning, to student ability to correctly solve both problem types on a final examination. Results indicate that students with higher reasoning abilities perform equally well on both problem types while students of average and lower reasoning abilities struggle in solving problems that depend on conceptual understanding. This suggests that students with average and lower reasoning abilities may depend more readily on memorization of simple procedures to solve problems.


INTRODUCTION
As the nation works toward increasing the number of students graduating with degrees in the science, technology, engineering, and math (STEM) disciplines, educators continue to search for the best ways to attract and retain students in these majors.One approach is to understand how students engage in problem solving such that support mechanisms can be provided for those who struggle.Novice problem solvers tend to memorize problem types and then simply apply this learned set of solutions to problems with similar surface features.However, many physics problems are more complex requiring content knowledge coupled with the ability to critically think about what the problem is asking and to determine how to approach resolving it.These types of problems require a different level of reasoning which may not be fully developed in all students.Therefore, it is important to understand how reasoning abilities relate to student ability to solve different types of problems.
This pilot study assesses how students enrolled in an introductory physics course solve two different types of problems; algorithmic and conceptually-based.Students were ranked based on their reasoning abilities, as measured by Lawson"s Classroom Test of Scientific Reasoning (LCTSR) [1], and this data was compared with student ability to solve simple algorithmic problems, as well as more complex problems that involved conceptual understanding, on the final examination given in the lecture course.Results of student abilities in solving these two types of problems in high, average and low reasoners are discussed.

PRIOR LITERATURE
According to Inhelder and Piaget [2] and Shrager and Siegler [3], students naturally develop reasoning abilities as they progress through the stages of learning.Piaget [2] indicates that although basic reasoning skills begin around age 4 in the preoperational stage (ages 4-7); scientific reasoning abilities typically develop in adolescence.At this stage individuals begin to include more complex logic in their thinking and consider multiple variables in problem solving, understand physical and social phenomenon, and consider the perspective of "if/then" reasoning.
Shrager and Siegler, basing their research on Inhelder and Piaget"s work, determined that individuals develop reasoning abilities at different rates and at different ages as indicated in their strategy choice model [3].Simple problem-solving strategies can be learned at earlier ages, while more complex strategies may develop with practice and maturation as the individual grows into adulthood.Shrager and Siegler indicate [3] that one of the most advanced problemsolving strategies involves retrieval; that is, where an individual builds upon the knowledge and problemsolving skills he/she has previously learned, and these skills are later easily recalled, rather than depending on memorized procedures to be successful.
In recent years researchers have compared scientific reasoning abilities of students in introductory physics courses with gains in conceptual learning [4,5].This research suggests students with more formal reasoning patterns are more proficient learners.However, little has been done to investigate whether or not scientific reasoning abilities relate to one"s ability to solve different types of problems, including those based on the application of simple algorithms and those which depend on conceptual understanding.In the field of chemistry, however, Cracolice, Deming and Ehlert [6] compared student performance on four pairs of algorithmic and conceptually-based questions in a general chemistry course.They found that students of higher reasoning abilities performed equally well on both problem types, but students of lower reasoning abilities performed significantly better on the algorithmic problems compared to the conceptually-based problems.

Population and Setting
This study was conducted at the University of Cincinnati, a large, doctoral-granting university.Data was collected from students enrolled in the first semester Introductory Physics course during Spring Semester, 2013.There were 476 students enrolled in the course across 4 lecture sections taught by 4 professors and 30 recitation sections taught by different graduate teaching assistants.The course involved 3 hours of lecture and one hour of recitation per week.Lectures were traditional in nature with modest use of a personal response system.The recitation sessions used the Tutorials in Introductory Physics [7] and promoted conceptual understanding of the material covered in lecture.Of the 476 students enrolled in the course, 99 students were randomly selected across the four lecture sections to complete the LCTSR.Of this subset, 80 students also took the final examination.After initial review of the LCTSR scores, it was evident that two individuals did not take the test seriously, as they had the lowest reasoning scores (0 and 2) which were not congruent with their midterm and final exam performances.In addition, insufficient exam data was available for 3 individuals; therefore, 75 students were included in this study.

Tests and Scoring
Lawson Classroom Test of Scientific Reasoning [1] The LCTSR is a 24-question multiple choice test assessing six reasoning domains: conservation of mass and volume, proportional thinking, identification of and control of variables, probabilistic thinking, correlational thinking and hypothetico-deductive reasoning.
The test consists of 11 paired questions in which students are asked to respond to a question and subsequently choose the best reason for their response, along with 2 independent questions.Students must get both questions in each pair correct in order to receive a point toward their final score.
Therefore, the maximum score for the LCTSR is 13.
For the 75 students included in this study, they were split into three groups based on the LCTSR score distribution.The data did not provide an easy split of students into three equally-sized groups.That is, there were a large number of students who scored a 9 and a 12 (which was exactly where cutoffs would lie for 3 equally-sized groups), so determining which scores to include in each group could not be easily discerned.Therefore, we chose to define the groups where a natural split occurred in the scores and in the end groups consisted of 17 high reasoners (scores of 12 or 13), 34 average reasoners (scores of 9 to 11) and 24 low reasoners (scores of 4 to 8).

Final Examination
The final exam for this course was a common exam given on the same day/time for students in all 4 lecture sections.The exam consisted of 40 multiple choice problems with some being more quantitative in nature and others more conceptual.All problems were similar to those found at the end of the chapters in the text.

Problem Types
For the purposes of this study, exam questions were identified as algorithmic or conceptually-based.Algorithmic problems were identified as those which require minimal conceptual knowledge for a student to respond correctly.These questions entail solutions by which a student could easily have memorized a simple equation or a single quantitative calculation.Conceptually-based problems were identified as those which require deep conceptual understanding, where students must understand the ideas or theories behind the content in order to organize and perform a series of steps to correctly solve the problem; and in some instances provide reasoning for their response.Questions that fit neither category, such as rote recall of algorithmic solutions, were not used in this study.
The authors of this paper independently coded each problem on the exam according to its type and resolved any differences (all minor).In the end, it was determined that the exam contained 15 algorithmic problems (simple memorization/single calculation) and 22 conceptually-based problems (5 concept problems with a single calculation; 3 concept problems with multiple step calculations; 14 concept only problems); with 3 questions not being used in the analysis.To better understand how the problem types were determined, example questions that require the application of Newton"s Laws are provided below.The results of responses for these questions are shown in Fig. 3.
Algorithmic question: "A person weighing 0.70 kN rides in an elevator that has an upward acceleration of 1.5 m/s 2 .What is the magnitude of the force of the elevator floor on the person?"(Newton"s 2 nd Law) This problem was identified as algorithmic due to its simple nature.That is, students had solved several problems similar to this one in course assignments, and therefore could easily memorize the solution associated with this problem ("plug and chug").
Conceptually-based question: "A heavy weight is supported by two cables that exert tensions of magnitude T1 and T2.Which statement is correct?(a) We need the mass of the box in order to determine the correct answer." In this example, because the mass of the hanging object is not provided, students are not able to depend simply on a single equation to solve the problem, as was practiced prior to the exam in assigned homework.Rather, students need to apply their conceptual understanding of vectors first, before considering how Newton"s Law applies and making a determination about the relationship between the two tensions.

Results and Discussion
In this pilot study, the problem solving performances on a final exam were compared for 24 students with low LCTSR scores (low reasoners), 34 students with average LCTSR scores (average reasoners) and 17 students with high LCTSR scores (high reasoners).As expected, the overall average scores on the final exam show a difference in performance between students of different reasoning abilities; 43% (low reasoners), 51% (average reasoners) and 58% (high reasoners).
We further analyzed the problem-solving success by separately plotting the number of total correct responses for each student for the algorithmic problems (Fig. 1) as well as the total number of correct responses for the conceptually-based problems (Fig. 2).Overall, 53% of high reasoners have an algorithmic score located above the average (average score = 8 correct), whereas 41% of average reasoners and only 29% of low reasoners have a score located above the average.For student performance on the conceptual-based problems, 65% of high reasoners, 32% of average reasoners, and 17% of the low reasoners have a score located above the average (average score = 11 correct).This indicates that high reasoners have greater proficiency in solving both problem types as compared to average and low reasoners.Although this supports what was found by Cracolice, Deming and Ehlert [6], they used paired algorithmic and conceptually-based questions written around the same topic, rather than focus on an entire exam.Because our study occurred in a natural setting, we were not able to put together paired questions for all questions on the final exam.However, we were able to find three sets of questions that mirror the efforts of Cracolice, Deming and Ehlert [6].These question sets involve the application of Newton"s Laws, torque, and fluid mechanics, all of which are representative of the previously discussed problem characteristics: simple memorization, concept only, concept problems with a single calculation, and concept problems with multiple step calculations.See Figs.3-5.In each of these three examples, one-tailed ttests were conducted to determine significance of differences between algorithmic and conceptual scores within and between the reasoning groups.Standard error was used when calculating error bars.Figure 3 shows a pattern similar to that found by Cracolice, Deming and Ehlert [6].That is, high reasoners have similar average scores on both problem types (65% algorithmic, 62% conceptually-based, p=0.25), while there is a substantial difference in performance between problem types for average and low reasoners (56% and 67% algorithmic; 34% and 27% conceptually-based; p=0.06 and p=0.05, respectively).Figure 3 also indicates there is a significant difference in performance on conceptual questions between high and low reasoners (p=0.00) as well as high and average reasoners (p=0.02).
This pattern is not as evident in Figs. 4 and 5.For these question pairs the high reasoners did not perform equally well on both question types as shown in Fig. 4 (82% algorithmic, 68% conceptually-based, p=0.25) and Fig. 5 (88% algorithmic, 60% conceptually-based, p=0.12).
Average and low reasoners performed similarly.Figure 4 shows the gap is more pronounced, however, for both average and low reasoners (74% and 71% algorithmic; 38% and 44% conceptually-based; p=0.05 and p=0.12, respectively).
A significant difference in performance between groups on algorithmic questions are seen in Fig. 5 between high and low reasoners (p=0.01), and high and average reasoners (p=0.04).As the authors were not the instructors for this course, we can"t be certain as to the level these topics were covered in the course and if that might have impacted this outcome.The information shown in Figs. 3 through 5, coupled with the observed gap in total performance on the final exam between reasoning ability groups, suggests a relationship exists between scientific reasoning abilities and problem solving performance.In general, high reasoners outperform average and low reasoners on both problem types, high reasoners perform more similarly on both problem types compared to average and low reasoners; and both average and low reasoners perform better on algorithmic over conceptually-based problems.This implies that students with more formal reasoning patterns may have developed the ability to construct and apply more complex concepts in their problem solving, which is logical in light of the work of Coletta and Phillips [4,5].In addition, these findings support the notion that students with lower reasoning abilities depend more heavily on memorization to be successful in solving problems.

Future Research
Although these findings support what was reported by Cracolice, Deming and Ehlert [6], we recognize several limitations within our pilot, including a small sample size, the use of multiple choice questions and a limited number of paired questions available for our analysis.In the upcoming year, we will repeat this study on a larger scale and make it more robust by constructing specific paired, free-response algorithmic and conceptually-based problems to include on exams.

FIGURE 1 .
FIGURE 1. Tally of the total number of correct responses by student for algorithmic problems.

FIGURE 2 .
FIGURE 2. Tally of the total number of correct responses by student for conceptually-based problems.

FIGURE 3 .
FIGURE 3. Comparison of scores on closely related final exam problems using ideas from fluid dynamics.

FIGURE 4 .
FIGURE 4. Comparison of scores on problems that involved the application of Newton"s Second Law.

FIGURE 5 .
FIGURE 5. Comparison of scores on problems related to the concept of torque on the final exam.