Changing the culture: Documenting shifts in a department’s norms around data use

,


I. INTRODUCTION
When facing a need for change, academic leaders tend to adopt simplistic approaches to change (e.g., adopting a single approach or strategy), often ignoring the change process in between and the needed support to enact the change [1]. They also tend to ignore the local organizational culture and the role of culture in shaping the change process and the effectiveness of outcomes [1]. Changes in a complex environment are more likely to be effective and sustained when accompanied by organizational cultural change, the sense of changing underlying shared assumptions [2,3]. However, such cultural changes are challenging and time-consuming [2]. Therefore, change leaders require continued support and resources to help enact such change. At a time where greater accountability influences institutional practices (e.g., funding), one aspect of cultural change that has received increasing attention is the use of assessment (culture of assessment) [4]. However, there is limited evidence on how institutions organize and establish cultures of assessment in higher education [5]. Moreover, while PER has invested in building assessment tools, it still remains unclear how these tools get woven into (if at all) collectivist approaches to change.
The APS developed the Effective Practices for Physics Programs (EP3) Guide, which is a collection of knowledge, experience, and proven good practices focused on helping physics faculty improve aspects of their undergraduate programs (e.g., recruitment and retention, department culture) [6]. The community engagement part of the EP3 Initiative is the Departmental Action Leadership Institute (DALI) [7]. DALI was launched to offer practical guidance to interested faculty change leaders in pursuing change efforts in their local departments. Two physics faculty are nominated from applying departments for help in responding to challenges and opportunities to improve their departments. DALI supports them in creating and leading a local team with multiple stakeholders following the Departmental Action Team (DAT) model [8]. The change leaders receive continued support within DALI in their pursuit of local change efforts. The EP3 research project focuses on understanding and documenting how change is enacted and sustained through collective, team-based efforts in local departments. The project also focuses on documenting how the emerging microculture within such team-based efforts is situated within the dominant departmental culture and how elements of this microculture spill over into other departmental areas. In this paper, we investigate the use of data and collective sensemaking within the emerging microculture of a DAT. In particular, we focus on the following research question: How is data sensemaking in a DAT situated relative to the department's dominant culture around the use of data, and what does it look like in practice?

II. METHODS
As a part of our larger study, we launched case study investigations of the participating programs in DALI from the first and second cohorts. These case studies include the collection of interview data from the two change leaders from each participating program at three points in time over a year. We follow them as they launch a local DAT and document how their DAT collaboration evolves. The DAT includes five physics faculty members (including the two change agents), one education faculty member, three students, and one alumni. Additionally, we collect interview data from DAT members and record field notes and artifacts from observations of their DAT meetings. This paper focuses on one of the first four case studies from cohort 1, Maple College drawing on the first three interviews conducted with the two change agents.
Maple College is a public, primarily undergraduate College. About 60% of Maple College students are Pell Grant eligible, about 20% are classified as 'first generation and low-income,' and about 35% are students of color. The two change leaders (DALI participants) represent the physics program which, along with chemistry and earth sciences, make up the Physical Sciences Department. The physics program is currently navigating a challenging financial environment that includes a reduced college budget, which threatens the sustainability of the physics program. With program review upcoming, the change leaders (Morgan and Misha) applied to DALI hoping to receive support in their efforts to increase student enrollment and identify ways to assess the effectiveness of past curriculum changes.
Morgan is an associate professor and the assistant chair of the department. Morgan has been in the department for more than sixteen years and has led many of the department's past curriculum changes. Misha is also an associate professor and has been in the department for about seven years. The following results draw on the three interviews conducted with each change agent.
We used thematic analysis, a qualitative method for identifying, analyzing, and reporting patterns in the data [10]. The analysis followed the six phases of familiarizing with data, generalizing initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the report [9]. More specifically, we first identified transcript segments relevant to the change leaders' recollections on the dominant culture around the use of data in the physics program. This theme heavily draws on the first and second interviews from both change leaders. This theme focuses on any program-wide efforts (individual and/or collective) that includes records of using data from student assessment and the extent to which data informed decision making or changes in the program. Next, we identified transcript segments relevant to the change leaders' hopes for using data in the DAT and later actions/behaviors that indicate the use of data to inform decision making in the DAT (emerging culture around the use of data). We also focused on the ways that change leaders described data sensemaking taking place in the DAT and the ways in which it led to collective decision-making (if any).

III. RESULTS
A. Dominant culture around the use of data The main theme encompassing the physics program's dominant culture around the use of data is the lack of collective discussion and reflection on the available data. Change leaders from Maple College reported that past student assessment has been conducted as an individual faculty member-led effort. Both change leaders emphasize that there has been a lack of collective focus and discussion about student assessment results. More specifically, individual faculty members collect data on student learning in single courses and/or teacher evaluations -however this data is never discussed collectively.
In the past ten years, the physics program at Maple College has gone through multiple curriculum changes. One of these efforts included changing all introductory physics courses from a traditional lecture-lab format to an active-learning, workshop model that integrated lecture and lab. Morgan was heavily involved in these efforts. She described that while assessment data was collected to assess the effectiveness of the curriculum changes, none of these results were ever discussed at a collective-level among the faculty members. Moreover, Misha explained that while assessment is being collected, there has never been a collective, department-or programlevel discussion in which faculty engaged in sensemaking around data, as shown below: "We don't do anything like this [collective discussion on assessment] in the department. Like we never had a focus group. I mean, let's just put it this way, all the efforts in my department are from individuals. [...] There have been some department-wide reorganization efforts but there's never been like a comprehensive look and data collection with an overall overarching theme in mind at all. There's always like assessment done, it's always like your feedback, usually teacher feedback just goes to the teacher and you read about your own feedback and you take what you can from it and and you are the kind of person who believes in growth and you do some some changes to your teaching techniques, but if you are the person who never looked at it, you never look at it. So the data that have been collected in my department are like teaching evaluations and also have peer to peer class observation." -Misha As Morgan described, past department-level decisions were solely made based on people's opinions and were rarely tied to any form of formal evidence. The approach to the issues that the department faced was messy and ad hoc, as shown below: "I think, in the past we've just kind of been like 'this is a good idea, let's try it' without even really trying to figure out what the problem is. We'd say, 'the problem is recruitment, okay, so let's talk to the high schools. Let's try to get in touch with guidance counselors.' Without really thinking about the other ways that we could recruit or really thinking about what are the most appropriate high schools to go to. So I think we've just kind of been like, let's try it and see if it sticks." -Morgan Morgan continued by emphasizing the lack of use of evidence in these sort of department level approach to resolving faced challenges: "Assessment is actually very tricky and unfortunately, no one is that interested in assessment, except me. But I've been trying to get them more interested in assessment, because we don't assess our upper level physics courses really at all and it's hard because they're so small. [...] I guess typically someone would bring an idea and then people would either say 'oh yeah that's a good idea' or 'oh no we don't like that idea', and if people say it's a good idea, then that person will work on it. [...] So I'm hoping that the data will encourage people to sort of have a more like communal responsibility for things and that we can start dividing up some of this work, because I feel like it tends to fall on me as the like head of the program and then maybe if I say this needs to get done by next week, then she'll do it, but otherwise not." -Morgan In the above quotation, Morgan expressed her hope that members of the department will develop a communal responsibility for collecting and interpreting assessment data through the DAT's change efforts. This thought shows her intention to influence a cultural shift around the use of data towards more communal engagement. As shown later in her interview, Morgan's reflections and dissatisfaction with the dominant culture around the use of data and her intentionality for cultural change are tied to her role as the assistant head of the department and as she shifts into becoming the head in the next year.

B. Emerging culture around the use of data
We identified four main themes encompassing the department's emerging culture around the use of data through the development of the physics DAT. First, we identified a shift in mindset for the two change leaders' towards valuing data as it serves to better understand the root of the problem at hand. Next, we identified that within the DAT, change leaders enact collecting brainstorming to decide on data collection tools and processes. Moreover, DAT members engage in collective data sensemaking. Finally, we identified that data informs the DAT's decision making and leads to a revision of implicit assumptions. In the following paragraphs, we present each theme in more detail.
1. Shift in mindset: data as a means to understanding the root of the problem Both Morgan and Misha described taking a slow pace to first understand the nature of the challenges based on evidence as one of the most important things they took from their participation in DALI.
"I guess I was really interested in the whole approach, sort of the information gathering and the like 'let's not frantically just change things but let's try to take a more measured approach and really try to get to the bottom of what the problems are.'" -Morgan As opposed to the dominant culture around the use of data in the department, the change leaders from the DAT are adopting the idea that data is not only used as an assessment of student learning but also as means to understand the root causes of wider problems at the department-level.

Collective brainstorming on data collection tools and processes
Due to the change leaders' participation in DALI, assessment has become an integral part of their DAT's change effort. Moreover, we identified a shift in the type of assessment; whereas it is more typical for members of the department to collect course evaluations and research-based student learning assessments (e.g., FCI), the DAT has developed its own assessment tools to assess the effectiveness of their program. Their DAT worked together for almost a full academic year to develop assessment tools and collect and interpret data to decide on the DAT's action items. In particular, their DAT collected student survey and focus group data to understand the reasons that non-physics majors did not select physics as their major or minor and to get a deeper insight from physics majors about their experiences and satisfaction with the physics program. Their DAT collectively decided that the student DAT members should take a lead in the data collection process, following the logic that since the student participants are more likely to share their most honest responses when power dynamics are minimized. In her quotation below, Morgan described the process the DAT went through around data collection: "The students [on the DAT] have been doing a lot of the work, because they did the focus group, so we brainstormed questions as a DAT at one of the meetings and then they took them and sort of winnowed them down and gave them what they thought would be the best questions. Then Misha and I met with them [DAT students] and we sort of fine tune to those even more, and then we met with them separately, a separate time with our alumni member just sort of go over the ground rules for like what they were supposed to be doing so try to make them comfortable with leading the focus group, and then they did that." -Morgan The DAT divided the labor across two subteams to work on the survey and focus group protocol development and analysis -each included faculty members and students. Misha specifically described how they worked on developing the survey questions, stating that she first read the survey development literature that DALI provided, and then worked with one education faculty member and a student team member to develop an initial draft of the survey. They then submitted the survey to DALI facilitators for feedback and worked on a plan to analyze the data.

Collective data sensemaking
The collected survey and student focus group data informed the action items that the DAT would work on next. Morgan and Misha explained that the DAT reviewed the data collectively (transcripts, survey responses) finding emerging themes. In the quotation below, Morgan described what the process of data interpretation looked like: "So we all read through that [transcripts from focus group] the entire DAT, and then we kind of picked out what we thought were the main themes from that. And the students who ran the focus group were there as well and they sort of gave their input so that was really kind of an informal thing. The survey, mainly Misha has been looking at the data and we've again, we've just been looking at As shown in Morgan's description above, the DAT was working together to identify the emerging themes in student focus groups as well as the open ended survey responses. The DAT students who led the student focus group contributed their input from this data collection process. Misha described below the data sensemaking process in the DAT: "So what Morgan and I did is we took this information [resulting themes] that we got from the survey and the focus group. And then we picked the five topics [from the EP3 guide]. And then we took it to the DAT and we said, listen, this is the data that we got, and these are the five topics we picked from the EP3 guide, which we think best matches the concerns that the students raised in all the data. And everybody agreed. I mean, it wasn't that hard to see. All students talked about was careers, so we need to talk about careers. So it was not even ambiguous, it was obvious that we need to talk about careers. And then we had a whole bunch of students complain about 'oh how some courses are hard and some are easy, and the math background is not enough, and then like we don't know what we're doing with our math, and why do we have to do this, can we do this instead of that.' So there was obviously another thing which said 'yeah we need to talk about curriculum.'" -Misha

Data informs DAT's decision making and leads to revised implicit assumptions
One of the emerging themes from the DATs collective sensemaking was the focus on the program's curriculum, an aspect that the DAT had not previously considered. Morgan described how the discussion on the state of curriculum that came up earlier in the DAT but that it did not continue since everyone considered it to be satisfactory and not in need of revision. However, the student data contradicted DAT members' initial opinions. In light of this outcome, the DAT reconsidered and redirected its focus. Morgan's quotation below describes this process: "I would say that the focus group really made us think more about our curriculum. Because we kind of thought that this DAT wouldn't focus much on the curriculum. We thought our curriculum was in at least a decent place. And we weren't really too interested in tinkering much with it. But when we listened to the focus group students, we really got the idea that they don't see a real direction to the curriculum, they don't understand how the curriculum builds. They do find the curriculum a bit inflexible, and they feel that that makes it difficult to graduate in four years, especially if they're transfer students or other non-traditional students. They've talked a lot about feeling like the math that they learned in the math classes, they don't understand how it translates into the physics classes." -Morgan The above quotation shows the extent to which the DAT, although largely unfamiliar with discussing assessment results at a department level, valued and respected the outcomes of assessment by redirecting their focus on designing a flexible curriculum to meet students' needs. In particular, it is evident that the DAT is open and willing to reconsider and revise their assumptions in the light of evidence.

IV. DISCUSSION & CONCLUSIONS
Within the physics program at Maple College, there is a dominant departmental assessment culture in which data collection is individual-led and there is a general pattern of insularity with respect to data interpretation. This case study shows us that within such as a dominant culture, a collective, team-based approach can create an alternative microculture which has the potential to spread into the broader departmental sphere. Through their participation in DALI, Morgan and Misha developed a new understanding of the purposes for which data could be used. As we saw earlier, data can be used to understand in depth the problem at hand by taking a more measured approach to change. Moreover, we found that within this new emerging microculture, team members collectively work to develop assessment tools, identify the best approaches to data collection, and collectively discuss and interpret the data.
Regarding data sensemaking, we saw that some aspects of this process are initially organized by individuals, but the ongoing sensemaking takes place at a collective level. We also found evidence that effective and meaningful data sensemaking at the collective level can lead to reconsidering and revising previous assumptions. Finally, we found that team members' roles in the department can be instrumental in helping to widen the influence of the emerging microculture within the broader department. In this case, Morgan, as she is shifting into the department head's role, emphasized her intentions on leading efforts to develop a communal responsibility on the use of data at the department. This foreshadows possible opportunities for broader cultural change.
Moving forward with our study, we will analyze and report on how additional case studies from both cohorts approach the use of data within their DATs and how these microcultures are situated within different dominant cultures at local departments. We will use additional interview data from team members and fieldnotes to make claims about the collective data sensemaking process within DATs and how interpersonal interactions influence and are influenced by data sensemaking.

ACKNOWLEDGMENTS
This work is partially supported by the NSF #1821372 and the APS Innovation Fund. The authors thank the study participants for their continuous collaboration and the EP3 research team members, Fatima Abdurrahman and Joel Corbo, who provided feedback on this paper.