Adapting Transformative Experience Surveys to Undergraduate Physics

Transformative experience (TE) is a theoretical construct intended to capture the extent to which science concepts learned in the classroom shape students' everyday meaning-making and engagement with science outside the classroom. One tool available to assess the depth and prevalence of TE is with surveys. We have been adapting existing surveys for use in various undergraduate physics courses at two different institutions, including algebra-based introductory physics courses and physical science courses for pre-service elementary teachers. We describe our efforts to modify existing surveys for use across different courses and content areas and describe our initial findings concerning the depth and prevalence of TE. From survey data, large differences can be detected in both the depth of students' overall engagement and the degree to which that engagement falls off when students are not in the classroom or working on required assignments.


INTRODUCTION
The physics education research community has long been interested in investigating and promoting interactive engagement classrooms in which students are intellectually and socially engaged with physics content [1].The focus of our work-while certainly related to in-class engagement-focuses instead on the extent to which physics courses can foster student engagement with science outside of the classroom.More precisely, we are not solely interested in whether students can be shown to think and act more like scientists when prompted by instruction or diagnostic instruments; we are interested in the extent to which students become actively engaged in thinking about and acting upon scientific ideas in their everyday lives, even when unprompted.
To conceptualize out-of-classroom engagement we draw on a construct called transformative experience (TE), which is intended to capture "experiences in which students actively use concepts to see and experience their everyday world in meaningful, new ways" [2].To be clear, TE refers to the discrete occurences themselves (e.g., a student walks outside expecting to find a crescent moon above the treeline and is pleased to see it is); rather than referring to more general changes to one's beliefs or attitudes toward science (e.g., a student now believes that science is related to her everyday world) or changes to one's science identity (e.g., a student now sees herself as a kind of person who enjoys thinking about science in everyday life).As defined by Pugh et al [3], TEs involve a synthesis of three elements into a single experience: (i) the motivated use of scientific concepts, (ii) an expansion of perception in how the everyday world is seen, and (iii) experiential value making such experiences personally meaningful.
We are interested in the extent to which students have such experiences as a result of physics instruction and investigate this phenomena using multiple methods, including survey development, case study, and video ethnography.In this paper, we report on our work to adapt existing surveys [3,4] to undegraduate science courses.We describe how we have adapted these surveys and report on differences found among different undergraduate science courses.Our findings highlight that there are not only differences in overall TE measures across courses/content, but differences in the degree to which engagement decreases with decreasing proximity to class and class assignments.

SURVEY STRUCTURE
The TE survey that was adapted for our research consists of thirty-one statements to which students respond on a four-point Likert scale.Consistent with the TE construct, most of the items ask students to consider statements about behaviors (e.g., talking or noticing) rather than statements about beliefs or dispositions.Due to the fact that TE is also intended to capture experiences that are meaningful and new students, some survey statements necessarily concern non-behaviors such as situational interest (e.g., "I am interested when I hear things about…") and perceptions about their learning (e.g., "The ideas we learned about… changed the way I see,").
The survey aims to measure TE with respect to specific content, so survey statements consist of stems that can be modified.Statements specify physics content and some reference specific physical contexts.Table 1 shows a list of the stems that are used.For example, in the force and motion survey, statement 2 reads, "I think about the laws of force and motion, when I see objects that are falling, colliding, or balanced."In the optics survey, statement 2 reads, "I think about the rules for optics and the nature of light, when I see things like light bulbs, cameras, or glasses.There are several structural aspects of the survey relevant to analysis.First, each survey statement targets a specific aspect of the TE construct-either motivated use (1-13), expansion of perception (14-18), or experiential value (19-31).Second, collections of statements refer to a specific behavior in different contexts (e.g, statements 1, 3, and 5 describe "talking" as occuring during class, outside of class, and just for fun.).We leverage this structural feature of the survey to compare how engagement decreases as the behavior becomes increasingly removed from the classroom.

CONTEXT FOR RESEARCH
We have adminstered the TE survey in a small number of undergraduate physics and physical science courses.Four are reported in this paper (see Table 2).All the courses have small student-instructor ratios (32:1 or less).The first two courses cover traditional introductory physics topics in a reform "studio-like" setting and are taken by a variety of different majors.The second two courses cover fewer topics in a more open inquiry environment [5] and are typically taken by students planning to enter into K-8 teaching certification programs.All of these courses can be characterized by a healthy abundance of interactive engagement.The two courses for future teachers involve the significant development of students' ideas overtime in a responsive teaching setting [6].The physics courses use methods such as Peer Instruction [7] and collaborative problem solving [8], to improve student learning of scientific concepts, but in a less responsive environment with less time per topic.In each course, surveys were adminstered shortly after the surveyed content had been covered (typically within a week).The specific content that was surveyed was selected to reflect content that was a core focus of that class.In each case the content was taught over a significant period of time (five weeks or more).In making the surveys, phrases describing the physics content and contexts were made to be as general as possible (to allow for future comparisons), but consultations were made with instructors to make sure that such descriptions would be understandable to students and related to the actual content of the course.
Due to practical constraints arising from instruction, the surveys in the two physics courses were adminstered via paper and pencil, while the surveys used in the two physical science courses were adminstered online.Written comments embedded in the online version of the survey, in addition to interviews, have been used to help establish validity.

SURVEY RESULTS
Figure 1 provides overall snapshots of how students responsed to the TE survey in each course.Some general findings can be summarized as follows: • All of the courses showed more overall agreement than disagrement, but this occured to widely varying degrees.Specifically, NSCI 321 had the highest percentage of agreement (89% of the total); while PHYS 2021 measured 53%.• The two inquiry-oriented courses for future teachers measured higher levels of agreement (at 89% and 77%) than the two interactiveengagement physics courses (at 63% and 53%).• In terms of individual questions, statements 1, 4, 8, and 16 ranked among the easiest to agree to, concerning behaviors of talking, thinking, using, and noticing in class.• The most difficult statements to agree to were 5, 11, 15, and 18, which concerned experiences that were either sought out (e.g., actively "looking for" examples), or engaged in spontaneously (e.g., "just for fun").

Engagement Decreases Out of Class
While it is generally true that students in all classes were less likely to agree to statements about behaviors occuring outside of class than behaviors occuring inside of class, the degree to which that agreement trends downward differs across the courses.
For example, statements 8, 9, and 10 are each about using knowledge-"during class", "outside of school", and "even when I don't have to."In NSCI 321, reported levels of using knowledge is high and fairly level across the three settings (91%, 95%, and 82%, respectively).In contrast, in PHYS 2011, reports of using knowledge starts off high but falls off rapidly (94%, 49%, and 28%, respectively).Using the Mann-Whitney U-test to investigate this "decrease in engagement", one finds that the decreases for both classes are significant (p < 0.01).However, the effect size, calculated as != !/!, is relatively small in NSCI 321 (r = 0.2) and relatively large for PHYS 2021 (r = 0.6).
The trend that NSCI 321 shows much smaller effect sizes continues to bear out when similar analyses are made for statements concerning talking and noticing examples (see Table 3).

Discussion
Our initial attempts to adapt TE surveys to undergraduate physics and physical science courses are promising.Not only does the survey seem capable of making discriminations using overall measures of student agreement/disagreement, we find that much of the differences we observe across courses arise from particular patterns that potentially signify transformative experiences.That is, in each of the courses described in this paper, students show high levels of agreement to statements concerning in class engagement; so that what distinguishes them, for the most part, is the amount of engagement with content outside of class.Combined with qualitative analyses of online comments submitted from students, interviews, and insight drawn from local instructor knowledge (all not described in this paper), we are confident that the TE survey provides useful measures of undergraduate student engagement with science content that are relevant to the having of transformative experiences.
Our goals in continuing to develop and refine these surveys are to eventually (i) provide an easy-to-use tool for physics instructor's to use in assessing their own courses and (ii) provide physics education researchers with a new tool that is capable of identifying interesting cases for further study.Future work to broaden the scope of the survey work, in particular, includes adminstering TE surveys in traditional lecture courses that use little to no interactive engagement methods and also in preservice teacher courses that use commerciallyavailable curriculum such as Physics by Inquiry [9] and Physics and Everyday Thinking [10].This and other data would help us to better achieve the two goals stated above-understanding how to make TE surveys applicable and valid for a broader range of courses, and using such data to help identify factors that foster TE.
Based solely on the survey data presented here, in which course content, student populations, and teaching methods vary, it is impractical to identify any specific factors as fostering transformative experiences.Research to answer questions concerning the degree to which various individual factors, instructional methods, and even the content itself contribute to students' having transformative experiences is ongoing, both in the broader studies of transformative experiences, as well as, our own work.

1 .
During class, I talk about… 2. I think about…, when I see… 3 Outside of class, I talk about… 4.During class, I think about… 5.I talk about… just for the fun of it 6.Outside of class, I think about… 7. I find myself thinking about… in everyday life.8.During class, I use the knowledge I've learned about … 9. Outside of school, I use the knowledge I've learned about… 10.I use the stuff I've learned about…even when I don't have to.11.I look for chances to use my knowledge of…in my everyday life 12.During class, I see things in terms of the laws I've learned about… 13.When I am working on a class assignment about… I tend to think of them in terms of… 14.If I see a really interesting situation (either in real life, in a magazine, or on TV), then I think about it in terms of… 15.I can't help but see situations in terms of … 16.During class, I notice examples of… 17.I notice examples outside of class of… 18.I look for examples outside of class of… 19.Learning about…is useful for my future studies or work.20.Knowledge of…helps me to better understand the world around me. 21.Knowledge of… is useful in my current, everyday life.22.I find that knowledge of…makes my current, out-of-school experience more meaningful and interesting.23.Knowledge of…makes learning physics much more interesting.24.In class, I find it interesting to learn about… 25.I think…is an interesting topic 26.I find it interesting in class when we talk about… in terms of… 27.I'm interested when I hear things about…outside of school 28.I find it exciting to think outside of school about… 29.The ideas we learned changed the way I see… 30.I think about…differently now that I have learned about… 31.I pay more attention to…now.

FIGURE 1 .
FIGURE 1.The four bar graphs show student responses to individual survey statements.Each bar represents 100% (of the responses to a question) and the bar's location is positoned vertically to indicate the percentage agreement (above the axis) and the percentage disagreement (below the axis).Dark are used to indicate the fraction of strong agreement or strong disagreement.For example, NSCI 321 shows bars mostly above the axis, indicating a strong preference toward agreement; while PHYS 2021 shows a mix of bars above and below the axis, indicating a mix of agreement and disagreement.

TABLE 1 .
TE Survey Statement showing the "Stems"

TABLE 2 .
Courses and Content Areas Surveyed

TABLE 3 .
Effect sizes representing engagement "fall off"