Failure to Engage: Examining the Impact of Metacognitive Interventions on Persistent Intuitive Reasoning Approaches

The performance of introductory students on similar tasks used to assess their understanding of a particular physics topic can vary widely; conceptual and reasoning competence demonstrated on one task is often not exhibited on another, closely related task. Indeed, performance is often poor on tasks that strongly elicit students' intuitive ideas. Previously, we developed a paired-question methodology to disentangle reasoning approaches from conceptual understanding and used the dual process heuristic-analytic theory of reasoning to account for observed inconsistencies in student reasoning. It has been argued that metacognition may foster the productive engagement of the analytic process during reasoning. In this study, we examined the impact on student reasoning patterns of three metacognitive interventions that varied significantly in both focus and scaffolding. Our findings suggest that, even for students with a robust conceptual understanding, incorrect intuitive reasoning persists and these interventions do not appear to engage the analytic process more productively.


INTRODUCTION
This study examines a particularly puzzling occurrence in introductory physics courses: conceptual and reasoning competence demonstrated by students on a specific task often fails to be exhibited on other, closely related tasks.Indeed, prior studies have shown that many students who provide erroneous responses on specific types of questions do possess the formal knowledge and skills necessary to arrive at a correct answer.However, instead of applying the appropriate knowledge and skills, some students tend to rely on intuitive ideas that lead to erroneous conclusions [1][2][3][4].Work reported in this study is a part of a larger effort that aims to probe student intuitive reasoning patterns at a deeper level and to identify circumstances under which tendencies to apply intuitive vs. formal reasoning approaches are enhanced or suppressed.The identification of such circumstances may provide additional insights into the development of effective instructional strategies.
In order to account for observed student intuitive reasoning approaches in a mechanistic fashion, we have applied the dual-process heuristic-analytic theory of reasoning proposed by Evans [5,6].(See Fig. 1.)The theory suggests that two processes are involved in most cognitive tasks: heuristic and analytic.The heuristic process is quick, subconscious, and automatic; when a reasoner is presented with an unfamiliar situation, the heuristic process immediately suggests a "first impression" mental model of the presented situation based on prior experience, relevance, and contextual cues.This model then becomes available for scrutiny by the analytic process, which is slow, deliberate, and rule-based.If a reasoner feels confident in the first impression mental model, the analytic process is often bypassed, thus yielding a response solely based on heuristics.The engagement of the analytic process, however, does not always result in a rigorous and logical evaluation of heuristicbased mental models due to reasoning biases.For example, if reasoners believe that an answer is correct, they do not tend to search for alternatives; instead, they tend to apply reasoning that supports their highly anticipated initial conclusions (confirmation bias).It has been argued that individuals' metacognitive abilities are responsible for the regulation of the thinking process through the productive engagement of the analytic process [7].In this study, we aim to gauge the resilience of student intuitive reasoning to different modes of metacognitive intervention.
For the purpose of this study, student thinking consistent with the first available response suggested by the heuristic process is referred to as intuitive reasoning.
Such reasoning may not be based exclusively on everyday knowledge and experiences; in fact, it may utilize a variety of strategies and ideas ubiquitous in physics instruction if they emerge before any others.

METHODOLOGY
In order to investigate student reasoning patterns, it is imperative to develop methodologies that allow for the disentanglement of student conceptual understanding and reasoning abilities.As such, we have been developing sequences of questions featuring one or two "screening questions" and a "target question" [8].The screening questions probe whether a given student possesses the requisite knowledge and skills necessary to arrive at a correct response.A target question features an analogous situation that calls for the application of the same knowledge and skills, and that, at the same time, may elicit intuitive rather than formal responses.
In this study, a sequence of two questions involving forces and Newton's laws was used as the basis for three different metacognitive interventions.The sequence was included on an exam in an introductory calculus-based physics course, and it was followed by one of two versions of an optional ungraded "metacognitive question" designed to engage student metacognition.In addition, the sequence was used in group interviews intended to foster spontaneous, socially mediated metacognition [9].
Exams.On version I, the metacognitive question was directly related to the Forces sequence and presented students with an opportunity to reflect on their own reasoning and to consider alternative solutions.It was hoped that the inclusion of such a question would engage the analytic process more productively and thereby increase the likelihood of revisions of student reasoning approaches on the target question, shifting them from intuitive to formal.On version II, the metacognitive question asked students to reflect on their understanding of the material presented on the entire exam and to predict their exam scores.The question was not designed to draw attention to specific reasoning approaches and thus was not anticipated to lead to revisions of student responses to the Forces sequence.Comparison of the students' performance on the Forces sequence included in both exam versions allowed us to probe the responsiveness of student reasoning approaches to a directed (not spontaneous) metacognitive intervention.

Group interviews.
As described above, collaborative group work may also be classified as a metacognitive intervention, and we have conducted and analyzed multiple group interviews.In this article, however, we use a single interview to illustrate how an analysis of the reasoning approaches employed by two students as they worked collaboratively on the Forces sequence allowed us to gain further insight into the persistence of intuitive thinking.

QUESTION SEQUENCES: OVERVIEW
Forces sequence.On the screening question, students considered Box A at rest on a rough surface.They were told that a horizontal 30 N force is applied to the box, as shown in Fig. 2(a), but the box remains at rest.
Students were asked to compare the magnitudes of the applied and frictional forces.The target question involved an analogous situation concerning box A and box B of equal mass.Students were told that the coefficient of static friction between box A and the floor is 0.4; the coefficient of static friction between box B and a different floor is 0.6.(See Fig. 2(b).)A horizontal 30 N force is applied to each box, and both boxes remain at rest.Students were asked to compare the magnitude of the frictional force exerted on box A to that exerted on box B.
Follow-up metacognitive question.All students received the Forces sequence above on their course exam.
The follow-up metacognitive question, however, differed depending on the exam version.Version I included a focused metacognitive question that directed students' attention to their reasoning on the Forces sequence.The question asked students (1) to predict what answer people who applied intuitive thinking to the target question would give, and (2) to reflect on whether the students themselves applied intuitive reasoning or formal knowledge.Version II contained a metacognitive question that asked students to provide an honest estimate of the number of points (0-100) that they thought they would get on this test.It is important to note that, at this point in the investigation, it was not our goal to probe students' abilities to categorize specific lines of reasoning as intuitive or formal.Instead, the inclusion of the focused metacognitive question was intended to provide students with opportunities to analyze alternative lines of reasoning and to reflect on their own solutions.

RESULTS
Exams.On the screening question, most students were able to apply Newton's second law correctly in order to recognize that the magnitude of the friction force must be equal to that of the applied force since the block remains at rest.(See Table 1.)The percentage of correct responses on the target question, however, decreased significantly, regardless of the exam version.More than 20% of the students who were able to apply Newton's second law correctly on the screening question failed to apply an analogous line of reasoning on the target question in order to recognize that the magnitudes of both frictional forces must be equal, or (f s ) A = (f s ) B .Instead, the students invoked the mathematical relationship between the coefficient of static friction and the (maximum) frictional force (f s,max = µ s N) and argued incorrectly that since µ A < µ B and The direct application of the heuristic-analytic theory suggests that the inclusion of the extraneous information on the target question (i.e., (µ S ) A < (µ S ) B ) prompted the development of a first impression mental model based on the relative values of µ.Although the mathematical relationship between f and µ is irrelevant to the task, this expression is both appropriate for and required by many physics problems in introductory mechanics.In the context of the target question, an incorrect interpretation of this relationship provides ready confirmation of an intuitive and highly anticipated answer.
As such, many students abandoned their lines of reasoning based on Newton's second law.
Group interviews.The analysis of video and audio from small groups of students working through math and physics problems or labs has played a key role in much of the work on metacognition and socially mediated metacognition [9][10][11].Indeed, as part of our larger effort to probe inconsistencies in student reasoning and the role of metacognition in minimizing such inconsistencies, we have also been examining the lines of reasoning given by pairs of students working through the Forces sequence during video-and audiotaped interviews.In these interviews, each student was given a written copy of the question sequence, and students were asked to collaborate as they worked through the questions.Livescribe smart pens were used to capture the in-the-moment writing of each student such that it was synched with the audio from their discussions.The interviewer did not intervene as the students completed the question sequence.
Below, we share an illustrative transcript from one interview, in which two students, identified here as Alex and Billie, worked through the Forces sequence.On the screening question, Billie argued that the frictional force could be equal to or greater than the 30 N applied force.Billie: Equal or greater than, right?Alex: No, it's going to be equal to because if the force of friction is greater than, it would be moving backwards, but it has to be equal.It can't be … Billie: It's like the normal force in the horizontal direction almost.Alex: So, here's a free body diagram.You have to have this 30 N this way.Billie: Yeah.Alex: And then 30 N that way.If you sum it upsum 'em up, the F net has to equal zero if it's not going to move.So, 30 minus 30 is equal to mass times acceleration.Billie: If friction is equal to ! --.Alex: Zero equals ten times a, a is equal to zero.It's not moving.If F net equals zero --Billie: What if there's a really big !? Why can't frictional force be greater than --Alex: Because ... Billie: the force of --Alex: Because the block remains at rest.If, if it were initially moving and then the frictional force is greater than, it would bring it to a stop, but since it doesn't move at all, it's going to be equal to.Billie: Okay.
In the discussion above, Billie suggested that a large coefficient of static friction should lead to a large frictional force.Alex attempted to convince her that the net force on the block must be zero, so the magnitude of the frictional force must be equal to that of the applied force.After a bit more discussion, Billie, perhaps reluctantly, agreed with Alex's application of Newton's second law and indicated that this line of reasoning made sense to her.Neither Alex nor Billie explicitly attempted to reconcile the apparent discrepancy between these two approaches.
As the group began to work on the target question, Billie and Alex quickly noted the values of the coefficients of static friction in both cases, and then focused on using these values to determine the magnitudes of both frictional forces.Alex: So the force of friction is equal to normal force times that.So let's do it for A. The force of friction is equal to 9.8 times 10 times .4for A.
And then force of friction is equal to 9.8 times 10 times .6.So the force of friction for B is going to be greater.On the target question, Alex, who had argued so strongly on behalf of analyzing the screening question using Newton's second law, apparently abandoned this line of reasoning in favor of the more equation-based reasoning articulated earlier by Billie.In terms of the heuristic-analytic theory, the transcript suggests that the nature of Alex's first impression mental model was impacted by the inclusion of specific information about the values of the two different coefficients of static friction in the question prompt.He immediately began to apply, incorrectly, equation-based reasoning that relates the frictional forces to their corresponding coefficients of friction.No attempt was made by the group to check for consistency between their responses to the target and screening questions.Indeed, this suggests that either the analytic process was bypassed entirely or confirmation bias with respect to the first impression mental model strongly impaired the functioning of the analytic process.This phenomenon is consistent with findings reported by Singh on a similar problem involving friction [12].As illustrated by this interview, even unscaffolded collaborative group work may not be sufficient to foster the level and quality of socially mediated metacognition needed to engage the students' analytic processes appropriately for the target question.

DISCUSSION
The quantitative results from the first two interventions revealed no intervention-dependent differences in the (poor) performance on the target question of those students who demonstrated a solid understanding of the relevant concepts on the screening question.Given our hypothesis that the version I intervention would be more likely to affect student reasoning than that on version II, it is quite plausible that neither intervention impacted student reasoning.While it is not possible to "see" in-themoment reasoning from written student responses, the transcript from the student group participating in a third intervention reveals an abrupt and unquestioned shift from formal reasoning (using Newton's second law) to incorrect intuitive reasoning (a higher coefficient of friction always implies a larger frictional force) on the target question.Indeed, Alex, who argued against that incorrect line of reasoning on the screening question embraced it immediately and fully on the target question, despite the inconsistency in approach.
Even with opportunities for socially mediated metacognition, these incorrect "firstavailable" reasoning approaches may persist on the target question, suggesting student failure to engage the analytic process in a productive manner.

CONCLUSIONS
The findings from our study serve to highlight the persistence of the kind of incorrect intuitive reasoning approaches often employed by introductory physics students.We have presented data from three different metacognitive interventions, which varied significantly in terms of focus and scaffolding, and we have observed similar reasoning patterns in all three cases.Moreover, our results suggest that students who find confirmation of their intuitive ideas in misinterpreted formal relationships tend to be particularly unlikely to question their first impression answers.

FIGURE 1 .
FIGURE 1. Diagram illustrating reasoning process based on heuristic-analytic theory.

FIGURE 2 .
FIGURE 2. Diagrams illustrating the experimental setup for (a) the screening question and (b) the target question of the Forces sequence.

TABLE 1 .
Results from Forces sequence of questions.