Effects Of Belief Bias On Student Reasoning From Data Tables

This study explores both students’ ability to make valid inferences from data tables and the effect of students’ prior beliefs on that ability. Over 300 introductory physics students participated in one of two experimental conditions. In both conditions, the data filling the tables was identical; however, in the first condition, the tables were presented within a familiar physical context, a method well known to elicit incorrect prior beliefs in many students. In the second, the tables were presented in a generic context. We found that, while most students were able to draw valid conclusion from simple, generic data sets, they were significantly more likely to draw invalid conclusions in the familiar physical context. Closer analysis revealed that, when provided with a physical context, students tended to look at the data less, relying in part on their prior knowledge to draw their conclusions. Interestingly, students in the physical context condition indicated a higher confidence in their responses, despite their lower accuracy.


INTRODUCTION
As educators work to equip students with the skills necessary to succeed outside the classroom, there has been an increasing emphasis on studying and improving students' scientific reasoning abilities [1].One of the skills often singled out for study is that of interpreting experimental data.A number of studies have investigated students' ability to make valid inferences from data, and perhaps unsurprisingly, it was found that students frequently failed to distinguish between actual data and their own expectations or theories of how the data should behave.
This tendency manifested itself throughout all stages of experimentation.For example, in designing experiments, prior beliefs seemed to narrow students' focus; students often concentrated on the variables they already believed to have the most significant effects.Working with 10-14 year old students, Penner and Klahr witnessed this narrowness of focus as students experimented with sinking objects [2].Given a range of variables such as the size, shape, and mass of the sinking objects, students tended to focus on only a few variables, not testing the others, and limiting the number of comparisons made overall.
When studying middle school students, Kuhn saw students following a similar trend [3].From a variety of studies, Kuhn and Dean summarized student reasoning from data as "a dynamic process of theoryevidence coordination" [4].More specifically, students often base their conclusions on not only the available data but also on their own repertoire of inference strategies and causal beliefs, and the students' common failure to distinguish between evidence and their own theories frequently led them to incorrect conclusions.In a related study of eighth-grade students, Park and Pak found that, even when physically shown controlled experiments, students were inclined to use prior beliefs as a source of information on which to base their conclusions [5].And, as might be expected, students tended to draw conclusions that aligned with their prior beliefs.
Unfortunately, these issues have been demonstrated to persist into college and beyond.In a study comprised both of university students and K-8 teachers taking introductory physics, Boudreaux, Shaffer, Heron, and McDermott saw prior beliefs inappropriately affecting students' design and interpretation of experiments [6].This is consistent with the findings of Kuhn, Katz and Dean; at all levels, failure to separate theory from data led students to design confounded comparisons, inconsistently interpret results, and incorrectly predict outcomes [7].
In this study, we investigate in greater detail students' ability to interpret data and more explicitly demonstrate the role of prior belief.We hypothesize that, if students use prior beliefs to guide data evaluation, they should have more success evaluating data that agrees with their prior belief.Conversely, they should struggle with data that contradicts those beliefs.As a secondary goal, we also explore the strategies that students use to evaluate data.Based on the work done by Kuhn and collaborators, it is reasonable to expect that students might naturally choose different strategies depending on their prior beliefs about the data presented to them.
To this end, we designed this study with the previously unused method of manipulating the strength and type of prior beliefs for a given data set to determine how this affects student reasoning.Specifically, we presented various kinds of data tables to over 300 introductory physics students and asked them to draw conclusions from the tables.One group of students was presented with a familiar physical context for their data, about which they commonly held correct or incorrect beliefs.The other group was presented with the same data but with a generic context, which had no explicit physical interpretation.

METHODS
319 university students participated in this study.All were enrolled in the second semester of an algebrabased, introductory physics course at Ohio State University, a large public research university.About eighty percent of the students were biological or health science majors, and slightly over half were female.
As part of the course, students were expected either to participate in this research or, if they did not want to be a part of the study, to complete an alternate assignment.Credit was given for participation, not correctness, and was equivalent to the credit given for a single homework assignment.
Participants completed the task in individual carrels in a quiet testing room.Each participant was assigned a variety of computer or paper-based tasks lasting no more than 55 minutes.The task given for this project took students five to ten minutes to complete, and was in pencil and paper format.
To test their ability to interpret data, we had students evaluate data tables.To construct each table, we used one of two separate data sets -one of which had a simple monotonic relation between one variable and the numerical outcome, and the other had a more complex non-monotonic relation between a number of variables and the outcome.Each data set was then presented in either a familiar physical context or a generic context.This produced a 2 (simple vs. complex data set) × 2 (physical vs. generic context) design which was run entirely between subjects.Students were randomly assigned to one of the four conditions.A one-way ANOVA showed no significant differences between students' final course grades in each of the four conditions, F(3)=1.375,p=0.250.
The two physical contexts used in this study were chosen because, in prior pilot testing and interviews, we found that students held a mix of both correct and incorrect beliefs about the causal relations between the variables and the outcome for each context.Using the generic condition as a baseline, this allowed us to evaluate whether students performed better on average when the data aligned with their typical prior beliefs and worse when they contradicted.
The data sets presented to participants are shown in Figure 1a, along with the setup for their corresponding physical contexts.In the generic context conditions, the names of the variables were changed (e.g. to x, y, and z), no figures were included, and the only setup was given in the form: "A student conducted an experiment to determine what factors affect a result R. The student's data is recorded in the table below."This is shown in Fig. 1b.For all conditions, the students were asked to use only the data in the table to determine how each variable affected the outcome.Finally, they were asked to rank their confidence that their conclusions agreed with the table on a scale from 1-5.

Comparison of Scores
The most notable result from this study was that students made completely valid conclusions less often in a familiar physical context compared to a generic context.For the tables with the simple data, only 30% of students given the physical (projectile motion) context came to completely valid conclusions compared to 56% of the students who evaluated the same data in the generic context, t(172)=3.505,p=0.001, d=0.56.
Unfortunately, the complex data appeared to be too difficult for this population of students to successfully interpret.A common difficulty was the non-monotonic nature of the variable r.Altogether, only 7% of students given the physical (electric field) context drew correct conclusions about every variable.Again, students given a generic context did better, but only slightly; 13% were correct overall, and the difference was not significant, t(143)=1.045,p=0.298, d=0.200.
Looking at each variable individually also suggested that, as predicted, students came to correct conclusions more often in the physical context about variables that behaved physically as they expected.They were correct less often in the physical context about variables whose behavior did not conform to general prior belief about the physical relations.
This pattern can be seen in Fig. 2. For example, in the complex data set, students reasoned correctly more often in the physical context about the angle, θ, and their explanations showed they had a correct physical, intuitive understanding of the effect of changing the angle.They reasoned correctly less often about r and typically had an incorrect physical intuition about how electric field depends on r.Similarly, in the simple data set, students often held misconceptions about the initial velocity v o , horizontal velocity v xo , and mass m.When given the physical context, they drew valid conclusions about these variables less frequently.
The only variable that does not follow this pattern is v y0 from the simple data set.In general, students believed that increasing the vertical velocity of a projectile increases flight time.Based on this, one might expect students in the physical context condition to perform better than those in the generic context condition.However, in general, students also seemed to be more familiar with the effects of initial speed, v o .This appeared to distract them from thinking about v y0 .In fact, over twenty-five percent of students in the physical context condition ignored v y0 and did not draw any conclusion about it whatsoever.On the other hand, in the generic context condition, less than ten percent of students failed to draw a conclusion about the variable.

Comparison of Strategies
There were also several differences in the way students approached the data tables based on context.When given a physical context and despite explicit instructions to only consider data from the table, students appeared to draw on their prior beliefs when coming to a conclusion.This tendency was most noticeable when reviewing students' written responses.For example, one student, who had been given the complex data and physical context, wrote, "smaller r 2 & θ produce stronger electric field because in that configuration, the field lines add up instead of having cancellation effect."Another student wrote about the simple data, saying, "As velocity increases, flight time increases.As the angle the object is shot at increases, the flight time increases."Interestingly, no data on launch angle was given in the table.Moreover, as the data is presented, the initial speed is not directly related to flight time.
Other students simply implied that they already knew the correct answer.One student decided she didn't need to see any more data to be sure of her conclusions because her "conclusions are true."They were not.Yet another student briefly justified his correct conclusions: "duh."Tellingly, none of the four students mentioned here cited data from the tables.
Students also drew on outside information by writing down equations.Slightly over 20% of student who were given the electric charge context wrote down an equation, most frequently the equation for electric field due to a point charge.Just under 10% of students given the projectile motion context cited an equation.
Aside from explicitly mentioning external information, students also differed in the way they cited data from the tables.When evaluating the generic context data, students tended to cite data more frequently and thoroughly.The proportion of students citing data (frequency) and making more than one comparison per variable (thoroughness) are shown in Fig. 3. Two-way ANOVA tests showed a significant main effect of context both on frequency, F(1)=4.171,p=0.042, d=0.26, and thoroughness, F(1)=10.684,p=0.001, d=0.52.There was a significant main effect of data set with more students citing data when given the simple data set than the complex data set, F(1)=8.57,p=0.004, d=0.37.There were no significant interaction effects.Students' confidence in their conclusions was also affected by context.Students ranked, on a five-point scale, how confident they were that their conclusions agreed with the data.Those given a physical context tended to express higher confidence than those given a generic context.For the complex data set, the average confidence rating in the physical context condition was 3.76 versus 3.15 in the generic context condition, a significant effect (Wilcoxon rank sum W= 2924.5, Z=2.525 p=0.012).For the simple data set, the averages were slightly higher: 3.93 vs. 3.69 in the physical and generic contexts respectively, a marginal difference, W=4576, Z=1.676, p=0.094.The smaller difference is possibly due to the simple data set being easier to interpret, raising the confidence of students in the generic condition.
Finally, students in all groups appeared to make a similar number of control-of-variable mistakes.Of those students who directly cited data, a two-way ANOVA test did not show any significant differences between conditions, F(3)=0.656,p=0.580.Just over twenty percent of students who cited data made control-of-variable mistakes.

CONCLUSION
Overall, we found that many students are able to draw valid conclusions from simple, generic data sets.However, by using the novel, explicit manipulation of the context and thus the prior beliefs that students brought to a specific data set, this study provides direct evidence that the accuracy of student inferences from data tables strongly depends on the type and strength of their prior beliefs.Compared to a generic context, students performed better when given a familiar physical context in which the data conformed to their prior belief.They performed worse when their prior beliefs contradicted the data.This played out even at the level of individual variables.
Furthermore, the generic and physical contexts also elicited different reasoning strategies.When given a familiar physical context, students cited the given data less often and less thoroughly, frequently referencing external ideas as a supplement.They also often appeared to use the given data to illustrate their prior beliefs, rather than evaluating the data separately.And, perhaps because they already believed the results, they indicated more confidence in their final conclusions.

FIGURE 1a .
FIGURE 1a.Complex and simple data tables given within a physical context.

FIGURE 3 .
FIGURE 3. The top graph shows the proportion of students who directly cited the given data in their responses.The bottom graph shows, of the students who cited data, the proportion who made more than one comparison per variable.