Making expert cognitive processes visible: planning and preliminary analysis in theoretical physics research

Many of the activities and cognitive processes that physicists use while solving problems are “invisible” to students, which can hinder their acquisition of important expert-like skills. Whereas the detailed calculations performed by researchers are often published in journals and textbooks, other activities such as those undertaken while planning how to approach a problem are rarely discussed in published research. Hence, these activities are especially hidden from students. To better understand how physicists solve problems in their professional research, we leveraged the framework of cognitive task analysis to conduct semi-structured interviews with theoretical physicists ( N = 11 ). Here we elucidate the role of planning and preliminary analysis in theorists’ work. Theorists described using a variety of activities in order to decide if their project was doable while also generating possible solution paths. These actions included doing preliminary calculations, reﬂecting on previous knowledge, gaining intuition and understanding by studying prior work, and reproducing previous results. We found that theorists typically did not pursue projects unless they had a clear idea of what the outcome of their project would be, or at least knew that they would be able to make progress on the problem. Thus, this preliminary design and analysis phase was highly important for theorists despite being largely hidden from students. We conclude by suggesting potential ways to incorporate our ﬁndings into the classroom to give students more numerous opportunities to engage in these expert-like practices.


I. INTRODUCTION
When non-expert physicists see photos of Einstein writing on a blackboard or watch an episode of The Big Bang Theory, they are certain to see a jumble of mathematical expressions that tell them little aside from the fact that physics involves a lot of math. Introductory physics students struggling to perform an integral in class similarly understand that math and physics are inextricably linked. Yet based on these experiences, one might believe that "doing physics" predominantly consists of doing difficult mathematical calculations. Although technical math prowess is an important skill for physicists, experts use math in diverse and nuanced ways that go beyond mere calculation [1,2]. Unfortunately, little of what expert physicists do is obvious to novice physicists.
Identifying expert problem-solving practices so that students can more efficiently engage in them is important for a variety of reasons. Teaching students to become skillful problem solvers has been the subject of research in physics education for many years [3][4][5][6], and cognitive apprenticeship theory emphasizes that students learn to become better problem solvers by performing expert-like problem-solving practices themselves [7,8]. Developing these skills is especially important for undergraduate students aspiring to become STEM professionals, as those are the skills that they will utilize on a daily basis in their future careers [9]. However, little research has aimed to characterize how physicists solve real-world problems related to their research, and has typically focused on engineers [10], multiple STEM fields [11], or physics graduate students [12]. Few have focused exclusively on physics researchers [13][14][15]. Moreover, prior studies have provided little information about the context in which the physicists were performing the problem-solving activities, therefore yielding limited insight into how and why physicists made the decisions that they made.
The analysis presented here seeks to address this gap, and is a subset of a larger project investigating theoretical physicists' problem solving in real-world contexts [16]. We conceptualize theorists' "problem solving" in such naturalistic settings similarly to Price et al. [11], who characterize problem solving as encompassing diverse activities such as generating research questions, doing calculations, and interpreting results. Lastly, we chose to study theorists for several reasons. Since much of the content in traditional undergraduate physics courses is centered on theory and does not have an experimental component, data collected from theorists could provide valuable insights for improving classroom practices. Elucidating hidden aspects of theorists' problemsolving practices will allow teachers to better leverage expert approaches for supporting student learning.
In this paper, we sought to answer the following questions: • What role does preliminary planning play in theorists' research processes? • What are some of the preliminary planning activities that theorists do before executing their main research calculations?

II. BACKGROUND
Many physics education research studies on expert-like problem solving practices focus on how experts (typically faculty and graduate students) solve introductory-level problems [5]. This research contrasts the ways that expert and novice physicists solve problems, and has found differences between the knowledge structures and procedures that undergraduate students and faculty use [17][18][19][20]. However, work specifically examining authentic problem-solving processes of physicists in real-world scientific contexts is comparatively limited [12-14, 16, 21], and many other investigations of expert problem solving in realistic science environments have either focused on specific disciplines outside of physics or on STEM professionals as a whole [10,11,22,23].
Among studies on expert problem solving in real-world contexts, several have identified a preliminary design and analysis phase of scientists' research processes. In one of the earliest self-reflections on problem-solving techniques, Polya's second step of a general four step problem solving plan was to devise a plan and choose an appropriate strategy for the problem [24]. Park et al. performed retrospective interviews with three experimentalists and three theorists and identified five primary processes that the physicists reported using in their authentic research [13]. These included generating hypotheses and designing research, which involved predicting possible solutions to their problems. Similarly, Price et al. identified a set of 29 primary decision points that experts use across disciplines to solve problems [11]. Among these were several that involved planning to solve the problem: Based on experience, what are potential solutions? Is the problem solvable, and is it worth pursuing?
We expand on previous studies of expert problem-solving by focusing specifically on theoretical physicists and examining the role of math in their preliminary problem solving activities. Although voluminous research in PER has examined the ways that students use and interpret math while solving physics problems [1,[25][26][27][28][29][30][31], we wish to explore some of the ways that math use manifests itself in this largely unseen preliminary planning phase of theorists' research.

III. METHOD
Knowledge about problem solving may be elicited from experts in a variety of ways, including interviews, selfreports, or direct observation. We opted to conduct interviews with theorists because they offer an efficient way of gathering data on long-term research projects. However, interviews may not offer the same level of richness that well-designed observational studies provide, and interviewees may not be able to fully and accurately recall, or may not be aware of, important processes in their research.
We designed semi-structured interview protocols based on the Critical Decision Method of task analysis [32] and Applied Cognitive Task Analysis [33]. Interviews were broken into three sections: the Task Diagram, Knowledge Elicitation, and General Questions. We began by asking interviewees to describe the main stages of a particular research project that had recently been completed, which we transcribed in a shared document (visible to both interviewer and interviewee) called the Task Diagram. We then asked the interviewee to go back through the tasks enumerated in the Task Diagram and describe them in more depth. In this knowledge elicitation stage, we asked in-depth probe questions to capture the cognitive processes used by the theorists at various stages. Examples of questions that often related to the planning stage of theorists' research included: What information made you decide to go forward with this project? How did you know it was doable and significant? What were some of the specific mathematical steps that you took during this project?
Although we interviewed N = 11 theoretical physics faculty, this analysis primarily focuses on five theorists from three institutions. We opted to analyze a subset of our interview population due to time and space considerations. These five theorists were chosen because they represented a diverse set of identities and sub-field specializations (see Table I). Although physicists are often classified as "experimentalists" or "theorists," there is no strict division between these groups, particularly with regard to computational research. Thus, to avoid making judgments on who qualifies as a "theorist," our selection criteria for interview subjects was that they selfidentified as a theoretical physicist. Of the subjects discussed in this analysis, N = 2 identified as male and N = 3 identified as female. Two identified as White, one as Latinx, and two identified as being from India.
Interviews typically lasted between 1 to 1.5 hours. We coded interview transcripts using Dedoose software [34] with the broad goal of identifying when and how theorists made important decisions. Therefore, in the first phase of analysis we coded interviews using a set of three process codes: Actions, Cues, and Sensemaking. From these process codes, we generated refined process diagrams using the software Lucidchart [35]. We first analyzed the process diagrams by grouping parts of each theorist's diagram into sub-processes (idea generation, preliminary design and analysis, executing research, and drawing conclusions) based on the common goals of each phase. A detailed discussion of these codes is presented in [16]. This process allowed us to gain insight into the first research question regarding the role that preliminary planning plays in theorists' research. In order to answer our second research question, we examined the specific actions that theorists took while planning their research and generated categories describing common activities.

A. Roles of theorists' planning processes
In each of the 11 interviews we conducted, theorists engaged in numerous preliminary planning activities after they  had chosen a potential new research project. These did not involve detailed calculations and did not necessarily result in products that theorists included in their published work. Across the 5 interviews that we analyzed closely for this paper, activities that theorists undertook while doing preliminary work in their research project served a number of roles (see Figure 1). For instance, theorists performed actions related to gathering and evaluating resources (human, computational, etc.) for their project. These included talking to graduate students to make sure that they would be willing to work on a prospective project, finding collaborators to supplement their own expertise, or deciding whether their groups had access to sufficient computational power. In this paper we focus on theorists' preliminary analysis (highlighted yellow in Figure 1) to determine whether a project is doable and to generate possible solution paths. Determining whether a project was doable included theorists judging whether they believed they could make progress on their question and if their research abilities would allow them to get useful results. Thus, generating possible solution paths was tightly related to determining project feasibility, as this process gave researchers confidence in their ability to make progress. Dr. Bahl expressed that one of the first thoughts she has after identifying a potential project idea is whether she can actually do it. She said, "So first is probably like getting the idea, right, which is kind of hard sometimes. And it's really hard to... one has to evaluate if it's doable." Hence, before ever beginning new calculations, Dr. Bahl goes through a process of evaluating whether her idea is feasible. In fact, each of the theorists expressed the sentiment that they were either fairly sure about the outcome of their research question prior to beginning detailed calculations, or at least were confident that their chosen direction would produce useful results. As Dr. Agarwal stated, "I think almost every project, we have a hunch of what's going to happen." Moreover, we observed that theorists had different thresholds for determining when to go ahead with detailed project calculations. For instance, Dr. Dunn stated, "In general, I would say that I spend a lot of time refining the, part of the early phases of assessing whether a project is doable is both these back of the envelope estimates... but also refining the question. I try to minimize the time I waste doing detailed calculations until I can see the path from beginning to end." This quote shows the importance that Dr. Dunn's preliminary work plays in his research process, as he utilizes this time to generate and test potential solutions to his research questions.

B. Specific planning activities
Preliminary calculations -As indicated by Dr. Dunn's quote in the previous section, performing preliminary calculations helped theorists figure out how to approach a problem and gain confidence that it was doable. We classified theorists' actions as preliminary calculations when the theorist indicated that they were taking mathematical or computational steps that were not highly detailed and could be done quickly, but yielded insight into their problem. Of the five interviews analyzed, two theorists described activities in this category.
While describing how she typically begins her projects, Dr. Costa recalled, "So usually, when I start a project, I kind of imagine what kind of scenarios I'm gonna get. So without, you know, I might not have done the simulation, I might not have written the equation, but I kind of have a feeling for what to expect, like different scenarios." Then, in order to quickly go through those scenarios to figure out which ones worked and which ones did not, she wrote simulations: "That's my favorite way of solving any problem like really quickly is by just simulating the process." Thus, Dr. Costa was able to identify viable solutions to her problem without spending time on detailed analytical calculations.
Similarly, Dr. Dunn undertook a series of calculations that he described as "basic algebra" that helped him identify po-

Preliminary
Design & Analysis  Figure 1: Aspects of theorists' research processes analyzed in this paper, highlighted in yellow. We focus on activities performed by theorists while generating possible solutions and deciding if a project is doable, which take place during a preliminary design and analysis phase.  tentially viable solutions (see Table II). In one excerpt, he describes using observations about energy to rule out certain mechanisms that could be driving magnetic field growth in large-scale astronomical objects. He says, "So first, follow the energy. That's the sort of very fundamental thing, if there's not enough energy in the things that I identify, there's no point in doing any detailed work. So you have to first assess very basic order of magnitude questions, possibilities, you know, rule out, so common sense calculations, which are non trivial, but save a lot of work." Ultimately, as described in the previous section, Dr. Dunn uses these types of calculations to get to a point where he "can see the path from beginning to end." These preliminary estimates therefore provided Dr. Dunn with essential insights into his problem and convinced him that his approach to the problem was viable.
Reflecting on prior knowledge -Whereas Dr. Dunn was unsure of how to approach his problem and therefore performed series of preliminary calculations to help him narrow down a solution space, other theorists described immediately "knowing" how to reasonably approach their problem. This stemmed from a familiarity with the problems and prior knowledge about the subject. When theorists utilized their existing expertise to determine how to approach a problem, we classified these excerpts as reflecting on prior knowledge. We observed instances of this in three of the five interviews.
While evaluating whether her project would be doable Dr. Agarwal said that she believed it would be "cut and dry" and that "the calculation itself seemed fairly easy" based on what she already knew about the alternate model of Newtonian gravity. She elaborated, stating that "The actual calculation itself was based on textbook physics. It's really just divergence and curl of a vector field. So the actual work itself was, you know, fairly textbook." Being highly familiar with what she perceived to be "textbook" physics allowed her to determine that making progress on the project would be "straightforward enough" to be worth pursuing.
Similarly, Dr. Costa described her existing knowledge of a particular algorithm applicable to her problem as a primary reason for her confidence in pursuing it. She recalled, "The technique that I was using I was fairly familiar with. So, like, the minute the question was given to me, I knew what method I wanted to do for it because I was familiar with that method from my bachelor's and my master's." Thus, Dr. Costa identified a potential problem and immediately identified a method for analyzing it. This, along with her initial simulations, helped convince her to pursue the project in detail.
Gaining intuition and understanding by studying -Three of the five theorists discussed in-depth studying, reading, and talking to other physicists as strategies to better judge whether they could make progress on their project. This was best exemplified by Dr. Erdogan, whose project involved work on turbulence, which is a longstanding problem in fluid mechanics. However, unlike other theorists interviewed, fluid mechanics was not Dr. Erdogan's area of expertise. He recalled, "So it took me a while to learn about it. I gave a few talks, uh, I even taught a course as a way to learn the subject. And then I wrote a book on the subject. Uh, so in these books, I, uh, in these books and courses, I was not talking about any new research." He also convinced a collaborator in the math department to help him learn the subject.
In-depth study of fluids allowed him to make headway into understanding turbulence after learning about a conserved quantity in fluids that had been shown to be topological in nature. "So that immediately piqued my attention because this is something I understand better than most physicists and certainly better than most fluid mechanics people. So I thought this is the place for me to start because here is something that I know better than almost everybody in that field." Despite over a year of reading, giving talks, and teaching a class, this quote emphasizes that this was all part of a preliminary phase of Dr. Erdogan's research. It was not until he had the proper background that he found a "place for me to start." Reproducing previous results -Lastly, one of the five theorists described reproducing previous results as an essential checkpoint for determining whether they are confident that their project is doable. Although only discussed by Dr. Bahl, we include it here as a standalone category due to the emphasis that she placed on its importance. Other researchers among the group of 11 theoretical physicists also discussed reproducing the results of other papers, but we did not perform a detailed analysis of those interviews for this analysis.
Dr. Bahl indicated that before performing any original calculations, her group meticulously recreated the results from the paper on which her new research was based. She said, "The first thing is always more or less reproducing previous results. So we have to be convinced that we understand the problem. And in order to know that we understand the problem and that we know we have the skills to do it, to do whatever new thing we want to do is, we should be able to reproduce other people's results." She sought to make the paper's results "like yours, like if you had done it." Thus, a major part of convincing herself that she had the ability to generalize previous work was being able to do that work herself.

V. DISCUSSION AND CONCLUSIONS
Despite physics' reputation for being a mathematically intensive subject, we observed that theorists did not perform lengthy calculations during the highly important process of planning their approach to a problem. Notably, Dr. Dunn regards this phase as the more difficult part of his problemsolving process than the actual calculations: "so there's the technical aspects of solving the equations, which is itself sort of essential, but I think what I notice is more challenging for people is... making these approximations and stuff becomes the more, the most difficult thing." Surely this revelation would be surprising to many students, since this crucial part of theorists' problem solving process does not appear in any published work and is largely hidden from their view.
Having illustrated several activities that theorists do while planning their problem-solving approach, we may begin leveraging these expert practices to support student learning. Doing so will afford students the opportunity to engage in these expert-like practices and improve their problem-solving skills. This is especially pertinent for upper-division undergraduates and graduate students who are seeking to become professional physicists themselves. Integration of more professional practice into upper level curricula would naturally contribute to these students' socialization into the field, both by improving their problem-solving skills and demonstrating more effectively what it means to "do physics." Moreover, it is likely easier to implement practices such as the preliminary design of a research project authentically into upper division courses via long-term projects. Long-term projects based open-ended questions could provide opportunities for students to engage in most or all of the preliminary design and analysis activities that we detailed here. Still, appropriately tailored curricular changes could benefit aspiring physicists at any level, even if those changes manifest themselves differently in lower level undergraduate classrooms. Whether undergraduate physics majors ultimately decide to pursue a career in STEM or not, giving these students exposure to the ways that make professional physicists' problem-solving skills useful and different would be beneficial.
Although proposing a detailed curriculum redesign is outside the scope of this discussion, we suggest several ideas to help instructors begin thinking about strategies encouraging students to engage in these expert-like practices. For instance, integration of Fermi problems and explicit messaging as to their utility could be one avenue of exploration. Similar exercises allowing students to perform preliminary predictions utilizing simulations would also mirror theorists' practice. Explicitly calling out these common practices may help students to develop these expert-like skills more quickly and make them a part of their regular problem-solving process.
We thank the theorists who participated in this study. This work is supported by NSF Award DGE-1846321 and REU-1757477.