Understanding Educational Transformation: Findings from a Survey of Past Participants of the Physics and Astronomy New Faculty Workshop

We surveyed past participants of the Physics and Astronomy New Faculty Workshop to learn more about their experiences. Survey questions were based on areas that emerged as salient from a longitudinal three year study of 15 workshop participants. Questions included: current practice, experiences with student resistance to reform, perceptions of available curricular and support resources, perceptions of local climate, and demographics of both faculty and their students. In this paper we report on an initial analysis of the survey data.


INTRODUCTION
We are engaged in a multi-project effort to better understand sustained and effective educational transformation, in higher education STEM courses, from the traditional lecture-based approach to one based on findings, strategies and curriculum founded in education research.From previous work [1], we know that the Physics and Astronomy New Faculty Workshop (NFW) has been highly impactful in terms of raising awareness of and motivation to use researchbased instructional strategies (RBIS), but does not necessarily lead to effective or sustained implementation.Therefore, we embarked on a study of the experiences of workshop attendees in order to work toward a more complete, and effective, model for change.
In the first phase of this project we selected 15 NFW participants from the summer 2010 NFW who expressed a particularly strong interest in incorporating what they had learned in the workshop into their teaching.By focusing on these motivated faculty we hoped to be able to identify particularly strong barriers for all faculty as well as focusing on systemic (as opposed to individual) barriers to reform.
These 15 faculty were interviewed pre and post each teaching semester of the study for a total of up to 5 semesters.Analysis of these interviews highlighted potentially interesting areas which we wanted to probe further, either by documenting the extent to which the experiences of these 15 faculty were common among the general population of NFW participants, or by following up with additional probes to test hypothesis.

METHODS
The research team worked together to identify main areas of interest and develop a survey which could be given to the full population of past NFW participants.The broad topics we decided to probe include: demographics of the faculty (race, gender, international background), self-report of teaching practice, issues related to student pushback, characteristics of local colleagues/department, perceptions of PER resources, course characteristics, integration of ideas learned in NFW into upper level courses, and correlations between each of these categories.
In the early summer of 2013 all past NFW participants (n=1306 1996-2012) were invited to participate in the web survey.Up to three reminder emails were sent about one week apart for faculty who had not yet responded.This resulted in 526 useable responses (including 34 partial responses) for a response rate of 40%.
In this paper we report an initial analysis of some of the findings.A more complex and complete analysis of the findings and interactions between variables will be reported at a later date.

FINDINGS
We asked faculty to report their estimated percent of class time spent in lecture.For simplicity of analysis faculty were then binned based on their response.Demographics of survey respondents broken down by percentage of class time reported lecturing are reported below in Table 1.We note that in our sample 27% of respondents overall were women compared to 24% of the participants of the workshop itself.This is an indication that our respondents were at least not largely unrepresentive of the larger population.Our data indicates several interesting findings.First, few (16%) of faculty report a completely lecture dominated traditional approach, which is encouraging.Secondly, we note that, as we found previously [1], women report a greater willingness to move away from lecture than men.Finally, we note that faculty who are non-native English speakers make up a significant fraction of the respondents (31% overall) and that they show a bi-polar pattern with respect to their use of lecture.We included questions about international status (of which native language is an indicator) because, although a significant number of physics faculty were not born in the USA, little is known about how this group interacts with PER.Our data indicates that international status appears to have an impact.However, the exact nature of that impact is currently unclear and will require further study.Finally, we note that many of the names in the group that reported the least lecturing are familiar to us, indicating that this group is heavily populated by faculty who identify as members of the PER community.

Perceptions of Available PER Resources: Room for Improvement
We asked several questions designed to assess faculty perceptions of PER resources.Results are presented in Tables 2 and 3 below.
We note that while a majority of faculty generally have positive perceptions of available resources it is not an overwhelming majority (approximately ¼ are reporting the need for improvements) and very few report strong positive perceptions.Additionally, it is interesting to note that there are disparities between the group that reported primarily using lecture (81-100% of class time) and the group that reports rarely lecturing (0-20% of class time) with the low lecture group having more positive perceptions than the high lecture group.This indicates that the availability of PER resources may be a particular barrier for faculty entering into the realm of more research-based approaches, or for faculty who may be more hesitant to embrace interactive methods.Of particular interest is the extent to which faculty report difficulty interpreting the results of assessment instruments.Having the ability to effectively measure outcomes seems essential to promoting reform yet nearly half off all faculty report difficulties in this area.This points to an area where the PER community should consider expanded efforts, i.e. providing faculty with more guidance and support to use research based assessment instruments.

Student Resistance to Research Based Methods: A Mixed Bag
From our interviews we noticed that several faculty had significant difficulties with student resistance while others were praised by their students for their use of research based techniques.It was not obvious to us from our interviews what led to the different reactions.Therefore, we asked several questions designed to assess faculty perceptions of and reactions to student complaints about non-traditional instruction.Results are presented in Tables 4 and 5  We note that the extent to which faculty report difficulties with student resistance varied by reported use of lecture as shown in Table 6 below.We found no differences in reported student resistance among native vs. non native English speakers, or among those who reported colleagues valuing or not valuing research based approaches, and marginal (but not significant, p=0.16) differences between male and female faculty.We also found no relationship between time since attending the NFW and reported levels of student complaints.On a positive note, the majority of faculty report little difficulty with student resistance and half report feeling their use of research-based strategies increased their student evaluation scores.However, reports of student resistance become more significant as faculty decrease their use of lecturing.This indicates that student resistance may not be a barrier to lite PER use but may become a barrier with more robust reforms.Certainly, more investigation is needed, but it does appear that reports of student resistance increase when lecture is reduced to below 40% of class time and continue to increase with decreasing lecture time.
We also note that we were surprised to find no other predictors of student resistance beyond extent of lecturing.This could potentially be an artifact of our survey design in that we relied on self-reports and did not have access to actual student evaluation data.Therefore we consider this area to still merit further study.

Grassroots Dissemination
One of the core ideas behind many current strategies to create change in teaching practices is that new teaching ideas will spread directly from person to person and also, within a given person new teaching ideas will spread from one class to another.Thus, we asked several survey questions designed to better understand how ideas spread.Some of these results are presented in Tables 7 and 8.The spread of NFW ideas to upper level courses (Table 7) is an encouraging result.The NFW was primarily aimed at introducing faculty to reforms in introductory courses, yet 77% of faculty who teach upper level courses have tried to implement ideas from the workshop into those courses.We asked faculty who tried techniques in the upper level courses, but do not currently use them what led to this discontinuation.We also asked faculty which ideas from the NFW they implemented.We intend to report an analysis of that data in a future paper.Table 8 presents data related to the spread of NFW ideas from attendees to their colleagues.Again, these results are encouraging.Nearly all faculty report discussing ideas from the workshop with their colleagues and many report changes in their colleague's teaching as a result.This indicates that efforts such as the NFW have the potential to have an impact beyond the actual participants.It is also interesting and encouraging to note that faculty who have more fully embraced interactive techniques report greater levels of impact on their colleagues.
The gender difference in reported impact of faculty on their colleagues is significant (chi squared = 4.18, p=0.04).Given that women are over represented in the low lecture groups which generally report higher levels of impact on colleagues than the high lecture groups, it would have been expected that women would also report higher impact levels than men, making this result likely even more significant.
A more detailed analysis indicates that the level of reported impact on colleagues increases for men as they decrease their use of lecture.However, women in all categories of amount of lecture use report the same levels of impact.There are multiple possible explanations and implications for this finding which we hope to pursue in more depth in future analysis.

IMPLICATIONS
The survey results are encouraging in many ways.Faculty who attended the NFW report generally moving away from pure lecture-based teaching approaches.Additionally, they report significant efforts to transfer what they learned in the workshop to their upper level courses and are influencing their colleagues.We are seeing the impact of the efforts of the Physics Education Research community on mainstream faculty.
However, our survey results also point to ways in which reform efforts could be improved.We offer the following recommendations.

Improve the availability of curricular resources:
Faculty report current resources are helpful but there is clearly room for improvement.2. Provide more guidance to faculty about how to use the results of research-based assessments: Faculty often do not know how to interpret the results of PER-based assessments, such as the Force Concept Inventory.More widespread use of such instruments could have a significant impact on teaching practices.3. Conduct more research to understand the nature of student resistance to innovations: One common reason that faculty give for not using PER-based instructional strategies is that they are concerned about student resistance.The results presented here suggest that, in practice, student resistance is not a big problem as long as lecture time is kept above 40% of total class time.However, more research is needed to confirm this result as well as to better understand the nature of student resistance to courses with limited lecture.4. Continue recent efforts to support integration of PER into upper level courses: Faculty are interested in and trying to apply PER to these courses.This is a very promising result and the PER community should continue its efforts in this area.

Find ways to leverage the informal dissemination
that occurs between colleagues: Faculty do discuss ideas with and can influence their peers.Yet, we know very little about how and why this happens and what can be done to make it happen more.

TABLE 6 .
Percent of faculty agreeing with statement that students complained when they integrated interactive techniques, broken down by reported extent of lecture use.When I have tried to integrate interactive techniques, such as those presented at the NFW, many students complained

TABLE 7 .
Faculty reports of integration of research based strategies in upper level courses.I have integrated ideas presented in the New Faculty Workshop into my upper level physics course.

TABLE 8 .
Reported extent of influence on colleagues.