You pick my brain and I’ll pick yours: a pilot case study of research-practice partnership in an informal physics space

Research-Practice Partnership (RPP) is recognized as a valuable tool for generating actionable research and improving educational practices by involving practitioners and researchers in both research and the implementation of research ﬁndings. However, creating and maintaining such partnerships is challenging; it requires time and ﬁnancial resources, and a team with diverse expertise and high commitment. Despite their wide recognition in educational ﬁelds, RPPs are under-studied in physics education research and literature in informal physics programs is sparse. In this paper, we present a case study of RPP between a physics education researcher and an informal physics practitioner. We describe the development and maintenance of this partnership and use qualitative and ethnographic methods to investigate practitioner-driven concerns about self-evaluation practices and attendance in the program under study, an informal conference for physics teachers. We found evidence that surveys and ethnographic interviews provided complementary data: surveys allowed the practitioner to learn about their audience’s motivations, needs, and perception of the program’s value. Ethnographic interviews were particularly useful as a means to elicit actionable ideas for how conference organizers might better support teacher attendance. In addition, participating in the RPP provided the practitioner with skills that they can carry forward into their future assessment work and yielded actionable insights beneﬁcial to both practitioner and audience. Our experience suggests that RPPs in informal physics spaces face many of the same challenges as those conducted in formal education settings, and that best practices documented for formal education RPPs will also be of beneﬁt to them.


I. INTRODUCTION
University physics departments, national labs, and research institutes invite a wide variety of participants to learn through their informal physics (IP) activities, also called physics outreach. Hence, those environments provide rich contexts for researching IP learning [1]. However, conducting education research in those spaces requires time and financial resources, supportive institutions, and teams with diverse expertise. Research-practice partnership (RPP) is a promising tool that allows researchers and practitioners in a broad spectrum of educational and social programs to co-create projects that answer research questions while providing practitioners with tools they can apply in their daily work (e.g., [2][3][4][5][6]).
While educational policy increasingly emphasizes evidence-based practice [7], teaching is in fact often not guided by research evidence [8,9], and some researchers remain skeptical of the way research is used in formulating policy [10]. Researchers, practitioners, funders, and policymakers recognize RPP as a valuable tool for bridging this gap by involving educators in research and the implementation of research findings [5,11,12]. Questions remain, however, about how to conduct research useful to practitioners, to involve practitioners in performing research, and how to best apply RPP findings to educational practice [11][12][13]. Despite their wide recognition in educational fields [11,14], RPPs are under-studied in physics education research (PER) (see [15,16] for curriculum-centered partnerships). While several have investigated RPPs in informal science spaces [1,[17][18][19][20], few have taken place in informal physics environments [21], where learning goals are situated in the discipline's distinct content, practices, and culture and in the particular difficulties students may experience in that discipline [10].
In this paper, we describe a case study of RPP between a physics education (PE) researcher from Physics Education Research Lab at Michigan State University and the teacher's conference lead facilitator at the Kavli Institute for Theoretical Physics (KITP) at the University of California, Santa Barbara. This study is qualitative and exploratory and seeks to answer two research questions: 1) how is a research-practice partnership designed, developed, and maintained over time in the context of one particular IP program? 2) What can survey and ethnographic research reveal about the program under study? This project gives us a lens to examine inherent challenges of RPP and facilitate the development of such partnerships by exploring each side's languages, cultural norms, incentives, and institutional strategies [22] .

II. METHODS
This partnership between PE researcher and IP practitioner is designed as a case study to collect information about the inner workings of the IP program under study and explore the limitations and challenges of such collaborations.
Program Under Study-Once a year since 1999, KITP has held a one-day conference for high school physics teachers from across the United States. This conference is decidedly an IP program designed for teachers to learn about current physics research through direct interactions with researchers. It is not pedagogical training. The content is driven by research trends and not by educational content standards. There is no fee to participate and financial support for travel is provided. KITP's informal programs are run by a collection of administrators and physics faculty. The teachers' conference is always coupled with one of the KITP's research workshops, with the theme varying each year. 4-5 physicists give talks on their research, each talk consisting of 40 minutes of lecture plus a 20-minute discussion period. Informal interactions outside the talks, both among teachers and between teachers and physicists, are also prioritized, with 2.5 hours devoted to lunch and coffee breaks. In addition to the scientific coordinator and speakers, KITP's director, outreach director, and some postdoctoral fellows attend breaks in order to interact informally with the teachers. These break-time interactions are an opportunity for physicists and teachers to continue conversations about the science content of the lectures, for teachers to ask questions about any physics topic, and for physicists and teachers to learn about each other, in particular about each other's observations and roles in students' transitions from secondary to post-secondary physics studies.
Design of the Partnership-The RPP includes a team of one PE postdoctoral researcher (researcher), the conference lead facilitator (practitioner), and an undergraduate researcher (UR). The partnership originated from the practitioner's participation as an interview subject in an IP landscape study [23] On a site visit to KITP in summer 2021, the researcher and practitioner initiated conversations around the teachers' conference and how strengths of and concerns about that program could make a fruitful partnership. As suggested by the RPP literature [11,22], the design of the research focused on problems of practice; challenges, needs, and questions that the practitioner finds pressing and important. KITP had extensive program evaluation data (e.g., 20 years of attendance records and eight years of participant surveys). Participant numbers had declined persistently since 2013. The practitioner was concerned that this indicated a gap in her knowledge about the potential audience. Her goal was to build her own capacity in research methods as a means to 1) better understand reasons for declining enrollment, and 2) improve self-evaluation and assessment practices.
The partners established a draft plan for the study, taking Coburn et al.'s [22] RPP definition as their guiding principle: 'a long-term mutualistic collaboration between partners that is intentionally organized to investigate problems of practice and solutions for improving program outcomes'. The proposal was pitched to KITP leadership as an opportunity for self-evaluation and improvement, staff capacity-building, and sharing findings with similar institutions [24]. The KITP director received periodic updates on project development. Each team member brought unique soft and hard skills to the project, which often made it immediately clear who should take on a responsibility [25]; e.g., the researcher's project management skills and the practitioner's institutional knowl- Teachers' emotional reaction to the conference and sense of its personal value "This excites me to learn theoretical physics. Makes me feel like a student again, which I miss."

Teachers' Interactions
Interactions of the teachers with either other teachers or program personnel "[I] also networked with other people about educational resources or teaching physics!"

Motivation and Attitudes
The reasons teachers attend or seek to participate in the program events, activities, environment "I find it is valuable that my students see me as a life-long learner."

Teachers' Attendance
The number of teachers who attend the events/program, and whether there is evidence that they return "I have been to almost all of the conferences."

Teachers' Needs & Challenges
Resources and other help that the teachers request in order to facilitate their participation in the conference "Since attendance is driven, to an extent, by the stipend, please work to keep the stipend aligned to cover travel costs." edge [26]. The research team developed cross-organization collaboration strategies by establishing norms of communication and weekly meetings. Standardized practices were established for document and task management [24]. Data Collection & Analysis-We analyzed practitionerdesigned program evaluation surveys and administrative data: annual program evaluation surveys had been conducted by institute staff prior to this project and were available from 2013 & 2015-2021; the team obtained IRB permission to repurpose these for academic research. Most survey questions were the same in all years, but sometimes questions were revised, added, or removed to better address evolving concerns of practice. The 2018 & 2021 surveys were conducted online; all others were given on paper after the last talk of the day. In addition to qualitative analysis, the practitioner used the existing survey questions with categorical answers and administrative data to investigate audience composition (type of school and subjects taught) and the appropriateness of the technical level of talks. Those questions were only asked in 2019, 2020, and 2022. The members co-designed the survey protocol, maintaining its general structure while adding questions that address research questions. Two interview protocols were codesigned, one for teachers and another for speakers. The UR conducted the interviews and collected observation data by taking field-notes.
Qualitative Analysis (QA)-QA was performed on four years of survey data (2015, 2016, 2018, 2022), a total of 147 responses, using MAXQDA [27]. These years were selected to ensure that our sample would include: 1) wide time points; 2) conferences with diverse themes; and 3) data predicted to best address our research questions. The initial theme of the analysis was adopted from 'audience', one of six themes identified in the Organization Theory (OT) framework for IP programs contextualized by Izadi et. al. [28][29][30][31]. Our analysis started with initial coding of the survey data by a single coder, using the 'audience' theme subcategories (satisfaction, interactions, motivation & attitudes, attendance, needs, and impact on the audience) [28,29]. The research team held multiple rounds of discussion to explore emergent ideas. After preliminary coding, codes were refined to better fit the overarching theme of 'participant teachers', the specific audience of the conference (see Table I). A single coded segment constitutes of a sentence or sentences in one response that meets the definition of a code. Codes can overlap and each response can contain multiple codes. A second UR was engaged for an inter-coder reliability (ICR) check. The second UR re-coded 10% of the 2015, 2016, and 2018 surveys. Among these, each code was present at approximately the frequency with which it occurred in the entire three-year sample. The MAXQDA Intercoder Agreement function [27] was used for testing consistency between coders, taking 90% match of a single segment to be an agreement. We found 82% agreement between the two coders.

III. FINDINGS
In this case study, we developed a partnership between a PE researcher and an IP practitioner (program for high school teachers) and explored its viability, utility, efficacy, and limitations; emphasizing ethnographic data collection and analysis. Here we present QA of the surveys as well as illustrative interview quotes developing selected themes. Our findings suggest that such partnership is a valuable way to understand issues of practice and provide actionable research.
Program-specific Findings-Archived administrative data and categorical-variable survey questions were used to observe attendance trends and participant composition. high because teachers were informed in advance about the research study and how past years' feedback had been used for program improvements. 2019, 2020, and 2022 surveys reveal that 73% of teachers who participate in the conference teach at public high schools, 15% at private high schools, 7% at community colleges, and the remainder at other types of institutions. 90-95% of participants teach some type of physical science; the remaining teach math, statistics, or computer science. About 80% teach more than one subject. In 2019 and 2020, in response to speaker concerns, the survey asked teachers to rate the technical level at which talks were given. 78% rated the level "just right", 8% "too high", and 1% "too low". 14% said that the level was just right for some talks and too high for others. Among those who said the level was too high, several expressed qualifying sentiments like, "The material is always over my head, but that's good, it excites me and inspires me to do more research on my own." Qualitative Analysis-The six sub-codes and their overlaps are shown in Figure 1. Each circle represents one theme (e.g., 'needs and challenges'), its size directly proportional to the number of code counts (in parentheses). The lines between two themes show how interconnected they are: line weight is directly proportional to the number of times the codes cooccurred (overlapped). 'impact on teachers' and 'teachers' satisfaction & engagement' were the most commonly encountered codes, with 215 & 131 instances respectively, and 'satisfaction' overlapped heavily with 'interactions.' We explored the coded segments containing overlaps between 'interactions' and 'satisfaction' and found responses expressing excitement about the "quality interactions with experts", "discussions with speakers", and discussions with "other teachers". One teacher wrote: "I am very happy that I made the long trip across the US to attend the conference. The talks were excellent and meeting great teachers from around the country was fun... I don't get to talk to other physics teachers very often." One teacher interviewee addressed the connection between 'interactions' and 'satisfaction': "[T]he important part... is the coffee break and the time to have the more informal conversations for either follow-up questions [about] the talk or the accidental connections that get made that bring about some really cool work or projects." The coders found repeated reference to how the teachers' participation affected or might affect their students. The word 'students' was mentioned in 59% of the teachers' surveys, with 46% of statements referring to students also containing 'impact' codes and 26% containing 'motivation & attitude' codes. Teachers mentioned both content and affective dimensions of impacts on students. One teacher wrote, "Several of my students have chosen to major in science because they were interested in some of the concepts presented here." The 'needs & challenges' code did not occur in any 2015-2016 surveys; however, it occurred in 20 in 2018 and 11 in 2022. All instances were in response to a single question asked in 2018 & 2022: "Please share any additional comments or suggestions." Most 2018 'needs & challenges' referred to a feature unique to that year's conference. Traditionally, speakers focus almost exclusively on sharing content through lecture and discussion. In 2018, however, each speaker included a hands-on demonstration. Surveys revealed teachers' mixed views of that tactic. A number said they welcomed content they could take back to their classrooms, one writing, "[Demos] provide us with another tool to reach our students,"). Others felt that the demos were not aligned with the conference's unique value of presenting current academic research; one referring back to it in 2022: "[Four] years ago the speakers tried to teach us how to teach, instead of explaining their research... Not what they, or you, do best. We teach; they need to inspire by explaining to us what they are discovering." Since few survey questions yielded information about 'Attendance', we asked interviewees "What do you think would incentivize more teachers to attend?" They said that teachers need support from school administrators in order to attend and suggested how KITP might help them obtain it: "One thing might be some sort of letter of support... where it says that KITP has a long history of staging these one day teachers conferences... it's a single topic [conference] with experts in the field, that it's a unique professional development opportunity that typically a teacher can't get in another way. And what I think a letter like that [would] lend some teeth to a request by a teacher to have a couple of days of personal or professional development time [off]... where a teacher doesn't have to tap into his or her personal days or sick days..., but that it's recognized as something that is beneficial to the teacher and ultimately to the students." The one speaker's interview suggested that the conference is both an opportunity for teachers to learn about physics and for physicists to learn about teachers: "[The teachers] said that they come to sort of get a flavor of modern developments in physics. And as one person put it, they want a hook that they can then use for their students to keep them excited. So if you tell them about some open problems which are being solved right now, I can see how students can be more interested in that than reading historical books about things that were discovered 100 years ago... I think the teachers are looking for these changes, trying to understand them, trying to make sure that when they communicate to their students that they correctly describe what happened." While the main focus of our work was on the teacher participants, interviews and observations presented an opportunity to learn about physicists' ideas of public engagement. During their talks, the coordinator and two speakers explicitly mentioned the necessity of public understanding of science in order to advance basic science. The physicist's description of teachers' motivation and needs was consistent with those expressed by teachers themselves; the physicists overwhelmingly understand the level at which to communicate with teachers. Furthermore, multiple presenters expressed, without prompting, appreciation and respect for teachers.

IV. CHALLENGES & LIMITATIONS
Both the senior researcher and the practitioner had higherpriority commitments in their daily jobs, which limited the time they could devote to the project. Despite this, we adhered to a schedule of routine meetings and maintained amiable relationships. Communicating openly allowed us to address uncertainties and re-assess the scope and direction of the work when needed.
Re-purposing program evaluation into research data presented challenges. Like many practitioners, the organizer had used field observations and results from one year's assessment to inform changes in the next; e.g., feedback from one year's speakers led her to add the question, "If a physicist asked you about their target audience for this conference, what would you tell them?". This fluid approach suits practitioners but yields a flawed research design, with data not standardized from year to year [4]. We would have liked to include demographic data in this work (gender, race, ethnicity), but institutional concerns about collateral effects on participants' willingness to provide such data trumped this desire.
The findings could not fully answer the practitioner's originating questions about declining attendance. Surveys and interviews shed light on why teachers do participate, and on the resource limitations they must overcome to do so, but they cannot shed light on additional needs or challenges of teachers who do not register. We recognize that IP programs vary widely in their content and structure, and acknowledge that our evaluation findings are program-specific.

V. DISCUSSION & FUTURE DIRECTIONS
In this paper, we outlined the design of our partnership and how we established practices to maintain a long-term, mutually beneficial partnership. This required researcher and practitioner to understand each other's distinct needs and goals. The team's lack of overlap in expertise and agendas alleviated some natural tension. Following the best practices for RPPs [24], partners were able to view the lack of overlap as grounds for sharing authority [21], dividing labor and working efficiently. We note that creating and maintaining longterm partnerships can have high initial barriers and require high effort and commitment.
We found evidence that complementary survey and ethnographic methodology allows practitioners to learn about their audience motivations and perceptions of their program's value. It is evident that participating physicists and teachers interact in an atmosphere of mutual respect. We intend to further investigate the relationships between the two groups and how they view each other's roles during the conference.
Although the case study could not reveal with certainty why potential audience members choose not to participate, insights gained about teachers who do participate gave several areas for action: 1) understanding how the conference benefits teachers and students will let KITP publicize the event in ways that convey the unique benefits of participation; 2) addressing the needs and challenges of participant (e.g., continuing financial support) may facilitate repeat participation; and 3) making evidence-based improvements to assessment practices will reveal how KITP can better serve its participants. The interview question, "What do you think would incentivize more teachers to attend this conference?" elicited actionable ideas and will be included in future surveys. In addition, the 90% response rate to 2022 surveys suggests that telling teachers how KITP has used survey data for program improvements incentivizes teachers to share feedback; this will become an annual practice.
The practitioner reports that participating in academic research was a capacity-building experience that will have lasting impact on evaluation and program-organizing efforts, both within and beyond the program under study [11]. Prior to the project, the annual quantitative data reporting did not include formal QA, under-utilizing valuable survey data. Learning QA research methods is allowing her to rigorously draw broad themes from surveys and interviews, better articulating the informal activities' impacts and seeing areas where many participants identify needs. This mitigates evaluator bias toward acting on a few attention-getting responses, be it confirmation bias or being reactive to felt criticism. Furthermore, the paper-writing process served as reflective practice for both parties. While others have identified reflective practice as a benefit of RPPs [32][33][34], involving practitioners in writing and presenting research findings for an academic audience may itself be a fruitful area for exploration [1].
We note that the findings presented here were centered around one practitioner concern, teachers' attendance. Interviews revealed that teachers need support from school administrators in order to participate in the event, and they believe that ISE practitioners can provide help getting administrative buy-in. The COVID-19 pandemic may have lasting effects on views about the costs, benefits, and equity issues of professional travel [35,36], and attendance is a serious issue that requires attention.