A qualitative analysis of concept maps through the Research Experiences for Undergraduates (REU) programs.

Learning physics in any context, including undergraduate research experiences, requires learning its concepts and the relational structure between those new concepts with what students already know. We use concept maps, a knowledge elicitation method, for assessing mentees’ and mentors’ knowledge structures during Research Experience for Undergraduates programs. The study looked at maps from seven mentor-mentee pairs to understand how mentors and mentees use specific knowledge and strategies during the development of their concept maps. A qualitative analysis of the maps showed mentors and mentees differed in their ways of organizing and displaying their knowledge in terms of structure, scale, language, and use of conceptual and procedural knowledge. For instance, mentees used more procedural knowledge. It is perhaps due to their perception of finishing their Research Experiences for Undergraduates (REU) projects and the fact that they may have only limited and superficial knowledge of specific topics. However, mentors maps were smaller but more significant in using more comprehensive conceptual knowledge and connecting their maps to the broader scientific context.


I. INTRODUCTION
Students' conceptual understanding and their ability to apply knowledge in problem-solving in a classroom setting have been studied extensively over the last three decades [1][2][3][4][5][6][7]. Moreover, in order to characterize students' scientific content knowledge in different learning environments, some literature have explored students' development in different researchrelated skills associated with undergraduate research programs [8][9][10][11][12][13][14][15]. Sadler et al. [10] reviewed 53 studies of scientific research apprenticeship experiences and concluded that all the studies documented some positive gains in participant understanding of science content. However, they highlighted the need for future studies to explore how participants learn scientific knowledge in order to maximize potential gains.
Learning physics and thinking like a physicist require learning the concepts, terms, and relational structure of physics. In order to move forward on the path from novice to expert, undergraduate students are required to develop their understanding and knowledge to achieve an accurate scientific mental model. Early descriptions outlined mental models as internal representations which are not directly observable [16]. Craik discussed that mental models are a smallscale model that people have in their minds [16]. Later, authors argued that mental models are knowledge structures and reasoning mechanisms that exist in either individuals' working memory [17][18][19] or long-term memory [20,21]. Scientific mental models are able to be developed which help researchers identify students' current knowledge, their knowledge structures, and their learning process [22][23][24]. Considering that mental models are internal cognitive structures, we asked our participants to visually represent their mental model of their Research Experiences for Undergraduates (REU) project. The concept map is a tool to reflect and assess students' learning, and understanding [25,26]. Concept maps, which began in the 1970s by Novak and colleagues, are a way to represent meaningful relationships between concepts [27][28][29][30]. They are a diagrammatic method of looking at the structure of these scientific mental models with a particular focus on which knowledge is used. Concept maps are composed of nodes labeled with concepts and links that connect them. In this paper, we discuss the characteristics of mentees' REU project maps and their paired mentors' maps. We examine how they put their knowledge together and link their ideas throughout the development of their concept maps process. The key idea of the map has characterized the type of knowledge in their maps. The research questions that guide this study are as follows: • What types of knowledge are represented in mentees' research project concept maps? • How did mentees and their mentors organize their knowledge around their research projects? • What differences exist between mentors' and mentees' concept maps?

II. BACKGROUND
Undergraduate research experiences have many benefits for undergraduate students such as clarifying their career goals [31][32][33], facilitating their research-based skill development [31,[34][35][36], learning a wide variety of content knowledge [35,37], and improving their critical thinking skills [38]. However, assessing these outcomes is usually challenging due to the complex nature and size of undergraduate research programs [8,39].
The existing studies around assessing outcomes of undergraduate research experiences are mainly derived from a combination of online self-reported surveys, such as the Undergraduate Research Student Self-Assessment (URSSA) [40] and the Survey of Undergraduate Research Experiences (SURE) [31], and semi-structured interviews and focus groups [32,33]. The primary goal of much of this assessment of undergraduate research is to measure students' broad progress to assess outcomes of research experiences prioritized by funding agencies [9,41]. However, to measure more detailed outcomes of the undergraduate research programs, especially related to the conceptual understanding and structure of scientific knowledge, these self-assessment tools may be insufficient. For instance, the URSSA self-assessment survey contains four main clusters of questions that ask students about their perception of their learning gains through questions about how much they have gained in terms of skills (e.g., writing scientific reports or papers), thinking and working like a scientist (e.g., problem-solving in general), personal gains (e.g., ability to work independently), and attitudes as a researcher (e.g., feel a part of a scientific community). The URSSA measures cognitive skills and affective learning gains, but it is limited to documenting students' progress and distinguishing the challenges and benefits of the programs. In addition, conceptual assessments such as the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE) test students understanding of Newtonian forces and motion and are not necessarily adapted to the varying content of a research experience.
As we mentioned in section I, learning physics requires learning its concepts and the relational structure between those concepts. These structural relationships impact how we teach and learn physics. Under the family of methods known as Cognitive Task Analysis (CTA) [42], which is a knowledge elicitation method, we select concept mapping to capture the knowledge and cognitive processes. Concept map are represented by meaningful relationships between concepts in the form of propositions that are linked with directional arrows, and labeled by words [43][44][45]. This tool reflects and assesses students' learning, and understanding [25,26] by encouraging them to use meaningful knowledge and develop the relationships between the concepts [26,46].

III. METHODOLOGY
The current research, including interview transcripts and diagrammatic representation of the linkages among the different pieces of knowledge, examines the quality of partici-pants' maps and how mentees and mentors put their knowledge together and link their ideas throughout the development of their concept maps in the context of REU program. To recruit students, we requested the REU coordinators who host their REU program remotely to forward our invitation email to their REU students. When seven students volunteered to participate in our study, we contacted their paired mentors. Participation incentives were offered in the form of $20 Amazon gift cards for each interview. Data was gathered through weekly semi-structured interviews with mentees (N =1 female and N =6 males) and seven paired male mentors from six REU programs in the summer of 2020. Before participants constructed their own concept maps, they were given the concept mapping tutorial. The concept mapping tutorial included a verbal explanation of what is concept map and mental model is and what a concept map is used for. Next, the interviewers showed participants two different concept maps. Then, we asked participants to brainstorm concepts related to their project and place those concepts in circles, then connect the concepts with arrows and place the linking words above lines drawn between the concepts which define the relationship between the connected concepts.
We collected concept maps from mentees during weeks four and seven and from mentors in week five. Concept maps were created in a Google Slides presentation that was editable and viewable by both the interviewer and interviewee. Participants were asked to use the drawing tools within Google Slides to create their maps. We recorded their explanations as they drew the diagrams. Mentees developed their first concept maps in week four of the REU program, and in week seven of the program they refined the old concept maps by adding more nodes and connections. Since this paper was limited in size and scope, we only focused on mentees' concept maps from week seven and their paired mentors' concept maps from week five. Our analysis involved both the structure and the types of knowledge present in the concept maps. The structural analysis focused on the ways of connections, number of nodes, and the scale of knowledge that was depicted. Identifying conceptual knowledge and procedural knowledge was based on participants' descriptions of nodes and the links connected to those nodes. The results section explains more about the distinction between procedural and conceptual knowledge. In addition to our findings on the structural differences between mentors' and mentees' maps, we presented a case study about Eli and his paired mentor, Dr. E (both pseudonyms), in this paper to illustrate specific themes from the broader analysis.

IV. RESULTS
Our analysis looks at seven paired mentor-mentee maps to understand how mentors and mentees use specific knowledge and strategies during the development of their concept maps.

A. Categorizing knowledge types
In this study, concept maps were created as a way to assess mentors' and mentees' knowledge structures and types.
These maps illustrated the various ideas in their research projects, which we divided into two types of knowledge: procedural and conceptual. Generally, conceptual knowledge is a block of principled ideas or a mental model of how something works. In the present context, we effectively assign the code to the conceptual and procedural knowledge based on previous literature reviews [47,48]. After, characterized knowledge, multiple categories emerged from our data based on both scale of the knowledge and characteristic. Next, after several iterative cycles of analysis and refinement, the coding scheme was judged stable by two researchers. We organized conceptual knowledge into the three major categories: Fields, topics, and quantities. "Field" described subfields of Science, such as "Optics", "Cosmology", "Acoustics", and "Chemistry." "Topics" described big scientific knowledge in the REU projects. For instance, we observed the topics "Energy", "Atomic structures", "Topological Constraint Theory", and "Sound and pressure resonance." The last category of conceptual knowledge is "quantities," which defines not only as a scientific concept but also a measurable physical property such as "Entropy", and "damping rate." Sometimes interviewees had labeled the edges between that node and other nodes with descriptive words (e.g., explains, describes, deals with,...), which help us to understand, to what degree interviewees, can describe, and explain why those combinations of theories and methods work. It is likely variable how strong the links are; some are perhaps very strong links, while others may be more akin to guesses. Most of these linking words were borrowed from the everyday language. One mentee said, "We have this idea of 'selection', what it means, what is a signal? What is its background? How do we do things to do that. [We understand] the 'detector' and definitions. So, we have a 'particle detector'; we have to understand how these things work. We do these things to do the selection that improves the background signal, which is we make discovery." The text demonstrate the student's attempt to link "signal" to "detector." On the other hand, procedural knowledge is the series of steps or actions taken to approach the problem-solving process or communicate about the results of their project (e.g., conference presentation or publication). In the context of our study, participants used procedural knowledge in two ways: Explicit and implicit. Explicit procedures included actions such as "predict", "compare", "determine structure", "vary parameter", "evaluate the system", "observe","formulate", "count", "analyze", "calculate", "modify", "measure", "expand results". These procedural words were usually involved directly in observation, calculation, and simulation. One mentee said, "Then that results in measured cross sections for a certain event which will be compared with the predictions for experimental results." The implicit procedures were tools, such as programming, simulation, and data analysis tools (e.g., ROOT, C++ or MAT-LAB). These implicit procedural knowledge nodes in the concept maps were nouns, but they imply particular actions as part of a procedure. The verbal description in the transcripts further clarified their use in a process. In addition, this included outcomes for reporting their results to the larger community. For instance, one participant said, "We are using the Monte Carlo to find a prediction for cross-section probability...we are using those predictions plus specifications of the detector." One mentor said, "Then we prepare talk...here we find a model. Now we take this [concept box], a paper and share with colleagues in UK, Greece and Penn state." Analysing the maps showed that mentees tended to use more procedural knowledge than conceptual knowledge (PK=64% and CK=32% (total 137 nodes)). However, mentors used more conceptual knowledge with fewer nodes in their maps (PK= 40% and CK=57% (total 66 nodes)). Overall, for 7% of the nodes, we could not label as procedural or conceptual due to the complex nature of their projects.

B. Structural analysis of the maps
The seven mentors' and seven mentees' maps were compared pairwise to distinguish the differences in each mentor and mentee map. A quick observation across all the maps indicates that all interviewees tended to be more hierarchical (arrows connect nodes in a top-down dendrogram fashion) than network (where one or more nodes have many edges pointing to it. Connectivity or Structure By looking across all the maps, we noticed that mentees' maps had high connectivity expressed by a more significant number of links and nodes 1, part a). That indicates that mentees took many different steps and used different knowledge to achieve their project goals. Mentees spent between 10 to 50 minutes building their maps during weeks four and seven. However, mentors' maps are more abstract, had fewer connections, no labels on the links, and were usually completed in less time (5-20 minutes). Mentors were usually quiet while creating their concept maps, perhaps because they were focused on comprehending a project instead of giving the interviewer a detailed description of each connection (See Figure 1, part b)).
Scale These two sets of maps from mentors and mentees also differ in terms of scale, language, and how these affect the types of explanations provided by mentors and mentees. In the context of our study, scale is defined as how the mentors were focused on how their projects fit within a larger scientific endeavor, while the mentees saw their projects from the vantage point of a person carrying out the detailed steps in a project. For instance, one mentor described the concept of Big Bang theory as a big idea behind their project, while the mentee talked about different particles such as Helium-4 and how to refine the old analysis code as the main step in his project. Another interesting finding to note is that overall mentees used more time to develop their maps in weeks four and seven, compared to mentors who only worked on their maps in week five. Mentees tended to write ideas as they came to their mind in a chain of knowledge. In contrast, mentors were interested in developing their maps in a way to embrace a cohesive plan for their project.

C. One paired mentor-mentee case study
Eli is a physics major who switched from a computer science major after taking a programming class. His REU project was about characterizing the efficiency of a detector and learning about details of nuclear reaction simulations and refining them. Eli's map shows a detailed description of the internal process of his REU project and an application of the wide variety of mathematical tools and simulations that he used. Eli's and Dr. E's concept maps depicted in Figures 2 show a clear hierarchy, linking words and connections. However, the linking words are terse. The first impression is that Eli's concept map is richer, and he used many more concepts than the concept map created by Dr. E.
The first aspect of their maps is related to procedural and conceptual knowledge. Eli elaborated his REU project as a series of procedural steps to refine and understand how simulation works in addition to conceptual knowledge that he included in his maps. He focused more on procedural explanations and task completion for his project.Eli felt more comfortable talking about different procedural steps in his project. He said, "The way we do it is, within ROOT, we can plot the number of particles, ...and compare it to the particles that we actually like, put it into the simulation, because we can put in 100 but only 80 will make it to the detector. We're also trying to figure out how to relate that to the location of the particle when it goes in and as the angle. Then, a lot of this just understanding of the simulation deals with programming... Probably the big picture is the simulation and what we seem to get out of it...I went through the code and kind of seeing what broke when I added it here and then trying to fix that." On the other hand, Dr. E's map provides a cohesive picture of the project including a limited number of nodes. He omitted some procedural steps, resulting in a less connected map. The qualitative analysis reveals that Eli used the smaller knowledge scale compare to Dr. E. Eli stated that the big goal of the REU project is learning the simulation. He explained, "Simulation describes the efficiency [which is] described by using our ROOT analysis because we like plots." However, Dr. E's map is shown more general approach to represent the  REU project. He began the concept map by explaining the broad topic of "Electromagnetism". Dr. E placed the comprehensive concept "E&M" on top and then continued that, "In the context of the REU project, what we are doing is simulating the trajectories and the behavior of particle." He explained that ionizing radiation consists of particles able to detach electrons from them, and the continued that "interface of charged particle which is moving into the material, creating ionization and [which] is the subject of nuclear physics." After analyzing the transcript, we noticed Eli progressed slowly and spent approximately eleven minutes on week four and ten minutes on week seven to create his map. He did not complete the links' labels until week seven. Dr. E only spent approximately five minutes on developing his map and explained it to interviewers without writing labels on the links.

V. DISCUSSION AND CONCLUSION
Here, an attempt is made to investigate to what extent the concept map can be used as an undergraduate research assessment tool. The qualitative analysis of the maps showed mentors' and mentees' had different ways of organizing and displaying their knowledge. Research suggests that developing concept maps helps students to understand concepts better [49] and to think "inside and outside the box" [50]. The mentees' concept maps help reveal the interplay between ideas and how to apply those ideas to achieve their project goals. They had ten weeks to learn some new knowledge (not necessarily with rich connections), and use procedures to finish their REU project and achieve their project goals. Concept maps drawn by mentors were smaller but more significant in using more comprehensive knowledge effectively (e.g., big picture science ideas, major theories, or fields) due to their greater grasp of knowledge and connecting it to the broader scientific context. Evidence from previous studies indicates that expert maps are usually associated with more detailed knowledge and a well-connected structure [51][52][53][54].
It was beyond the scope of this paper, but we observed mentees' maps grow between week four and week seven in terms of using more conceptual knowledge in their refined maps. This may be because students learn about their project better after three more weeks and also understand by practicing self-reflection on their own progress. During week seven, they added new nodes and links and sometimes predicted their future steps in the project. An interpretation of the above results is that concept mapping is a flexible method for eliciting and creating a cognitive model for all sorts of REU projects. As an assessment tool, concept maps allow us to document mentors' and mentees' knowledge representation, including conceptual and procedural ways they were thinking and talking about their project. In concluding this analysis, it is important to acknowledge that this paper aims to provide a potential starting point for developing a concept map as an assessment tool to measure students' conceptual understanding during the undergraduate research programs. Using concept maps as a new assessment tool raises several new research opportunities for PER community. However, a more systematic approach is needed to collect accurate data and elicit knowledge systematically.
We acknowledge the number of limitations in this study. First, we conducted a short training on developing a concept map with few examples; more systematic training is needed for future data collection. There were technical limitations in our data collection because Google slides are not the easiest or quickest way to develop concept maps, but they were convenient and widely accessible tool for participants. Other tools are easier for drawing but require specialized software on the participant's computer. Additionally, concept maps are representations with nodes and links that rely on an individual's description and understanding, however, it was difficult to get participants to include their knowledge in the written concept map during the online interviews. To avoid this, the interviewer can strengthen the interviews by asking interviewees to describe each concept and relationship clearly.