Assessing physics quantitative literacy in algebra-based physics: lessons learned

Physics quantitative literacy (PQL)—applying familiar mathematics in novel ways in the context of physics— is ubiquitous across physics classrooms. The Physics Inventory for Quantitative Literacy, or PIQL, is a recently published reasoning inventory that can be used to assess PQL from calculus-based introductory physics through upper division courses (White Brahmia et al. 2021). There remains a need, however, for assessment of quantitative reasoning at the algebra-based level which includes not only algebra-based college courses but also pre-college physics courses. We present recent work adapting the PIQL to an algebra-based context towards developing the GERQN—the Generalized Equation-based Reasoning inventory for Quantities and Negativity. We report lessons learned from our efforts to adapt items from the calculus-based PIQL to the algebra-based GERQN, and provide examples of how items were revised to be within students proximal zone. We also report on our experience translating the GERQN into Flemish as part of a larger, on-going research project, and what we learned about language accessibility for native and non-native English speakers alike for developing assessment items, curricular materials, and when speaking with students.


I. INTRODUCTION
Quantitative literacy (QL)-the interconnected skills, attitudes, and habits of mind that support the sophisticated use of familiar mathematics to describe and understand the world [1, 2]-is a central learning objective across STEM courses. Given its ubiquitous and nuanced mathematical nature, introductory physics is well-positioned to fill an educational niche of improving QL. An inventory recently developed and published by S. White Brahmia, A. Olsho, T. Smith, A. Boudreaux, P Eaton, and C. Zimmerman-the Physics Inventory for Quantitative Literacy, or the PIQL (pronounced "pickle")-has been shown to effectively measure physics quantitative literacy (that is, quantitative literacy in the context of physics) for calculus-based introductory physics courses [3]. However, there remains a need for an instrument to measure PQL that is appropriate for algebrabased physics courses.
Algebra-based physics courses are required for a wide variety of STEM majors, and are much more common than calculus-based courses in high schools. By developing an algebra-based inventory, we widen the population of instructors for whom the inventory is applicable and useful. As high school physics teachers are agents of change in their classrooms and the wider STEM community, making PQL assessment materials available to them includes an essential group of instructors in moving the needle on QL for all students. We are therefore in the process of developing an algebrabased version of the PIQL called the Generalized Equationbased Reasoning inventory for Quantity and Negativity, or the GERQN (pronouced "gherkin"), to address this need.
The purpose of this paper is to share some early insights as we explore features of physics quantitative reasoning that are ubiquitous, regardless of the mathematical preparation of the learners. We will begin with an example of how we adjusted PIQL items to be better suited for the GERQN population, and then share two lessons we learned during the process of developing the GERQN: (I) It is well known that reasoning about the rate of change of a quantity is challenging, and learners commonly conflate it with the quantity itself [4][5][6][7]. In this paper we contribute to this body of work by describing how we adjusted assessment items to be within algebra-based students' zone of proximal development in graphical contexts, and offer a reflection on what kinds of resources about rates of change we think may be accessible to this population for instructors to build on. (II) Language we use in assessment can be filtering students for the wrong reasons [8][9][10][11]. Here we describe two key lessons we learned from translating the GERQN into Flemish as part of a larger project and on on-going collaboration with the researchers at Katholieke Universiteit Leuven (KU Leuven).

II. BACKGROUND
Algebraic reasoning underpins both calculus-based and algebra-based introductory physics curricula. Reasoning about proportion, sense-making around symbols and representations, and making sense of quantities are common themes is the body of research on mathematical reasoning in introductory physics [12][13][14][15][16][17][18][19]. The conceptual blend of procedural resources and conceptual reasoning is foundational to mathematical sense-making in physics contexts [20,21]. Resources are pieces of knowledge that are activated when students are engaged in making sense of a particular context [22][23][24]. Therefore, discriminating between which resources are within the zone of proximal development for each population is essential to the development of the GERQN. Through interviews with physics faculty and a middle school mathematics education researcher, we cyclically revised and developed items for the GERQN to probe resources that are foundational to "calculus-like" reasoning at higher levels.
The GERQN development is a part of collaborative comparison study with researchers at KU Leuven in Belgium that explores similarities in, and differences between, US and Belgian students' PQL. As a part of this study, we worked with researchers in Belgium to translate the GERQN into Flemish and gained insights about the clarity of our question statements for both native and non-native English speakers. In this paper, we aim to share some lessons learned during the GERQN development process from both a mathematical and linguistic perspective, and provide suggestions for researchers and instructors to consider when drafting assessment items, creating activities and speaking to students.

III. DEVELOPMENT AND VALIDATION
Our procedure for inventory and item development was conducted at a large, R1 university in the Pacific Northwest, and follows the procedure used in the development of the PIQL [3,25]. We designed the GERQN by modifying PIQL items and developing new items as needed based on the following assumed differences in the population of test-takers: (1) lower average grade level, and (2) fewer years of prior physics and mathematics courses. These items underwent expert validation with a panel of 6 experts (physics faculty that teach in the introductory physics sequence) and with a middle school mathematics education research expert. The items underwent a cycle of revisions based on expert comments and pilot testing. We then conducted student validation in 12 individual, think-aloud interviews with students from the algebrabased introductory sequence. The items were also administered in the same courses over three quarters (N = 1808). An added dimension of this process involved a collaboration with a researcher and masters students enrolled in KU Leuven's physics teacher training program. These researchers translated the GERQN in its current version into Flemish and performed 10 individual, think-aloud interviews with Belgian (a) The internal energy of a system can be increased by doing positive work on the system or by heating it, and it can be decreased by cooling the system or if the system does work. Which of the following equations represent(s) this relationship (U is the internal energy of the system, Q is positive when energy flows into the system, and W is positive when positive work is done on the system)? Choose all that apply. a The amount of money in your wallet changes by ∆M when you receive or spend money. The value of ∆M is greater than zero when you receive money, and the value of ∆M is less than zero when you spend money.
Let S represent the money exchanged for services. The value of S is greater than zero if you are paid for a service, and The value of S is less than zero if you pay for a service.
Let G represent the money exchanged for goods. The value of G is greater than zero if you sell something and The value of G is less than zero when you buy something.
Which of the following equations represent(s) this relationship? Choose all that apply. algebra-based physics students in Flemish. The analysis of both sets of student interviews is still on-going. However, early results from our initial expert validation, large-N administration, and translation into Flemish have generated knowledge about bringing expert PQL into the zone of proximal development of the students we teach and assess.

A. Item Development and Modification
The majority of the 18 items on the GERQN are the same as those on the PIQL. For the GERQN, we rejected PIQL items that required calculus, an understanding of vector quantities, or involved physics contexts that are not taught in the algebra sequence. These items were either modified or removed.
The items "Internal Energy" and "Money" are an example of the ways PIQL items were modified for use on the GERQN (Fig. 1). The PIQL version of the item, "Internal Energy," asks students to choose the best model to represent how work and heat are related to an object's internal energy. The PIQL item was originally developed to probe student reason-ing around symbolizing (specifically the implied meanings of "+" and "-") and use of variables. The internal energy context was validated with calculus-based students, and the most popular incorrect answer for calculus-based students is to choose A and F. Student interviews reveal this is because A and F align with definitions of the first law of thermodynamics from prior instruction. However, work, heat, and internal energy are less likely to be familiar terms for algebra-based physics students, in particular for those in pre-college classrooms.
The version "Money" was developed for the GERQN where the context was adjusted to more familiar quantities: the relationship between the change in the amount of money in a person's wallet and how much they earn or spend on goods and services. Early administrations of the GERQN suggest that the most common incorrect answer with algebrabased students is to choose all of the answer options (A, B, C, and D). Student interviews suggest that choosing A, B, C, and D involves reasoning that the variables represent only positive values and that a negative value would need to be indicated by an additional symbol outside the variable. Using this reasoning, all four equations collectively represent the possible scenarios that could occur.
This example illustrates how we made modifications to the PIQL items to adjust context and language to be more appropriate for GERQN audiences. In the following sections we will address specific lessons learned from: (I) adjusting items that measure reasoning about rates of change, and (II) adjusting language for non-native speakers based on our experience from translating the GERQN.

B. Reasoning about Rates of Change
Meredith and Redish summarize the mathematical reasoning foundational to algebra-based physics [26]. However, the line between what mathematical reasoning is required for calculus-based physics and algebra-based physics is not entirely clear; calculus-based physics tends to heavily rely on algebra-like reasoning with a small dose of derivatives and integrals. The items Spherical Bottle [27] and Cylindrical Bottle illustrate how we approach distinguishing between conceptual, mathematical reasoning skills for these two populations (Fig. 2).
Spherical bottle requires students to reason about a nonlinear rate of change between the height of the water and the volume of water in the bottle-the correct graph includes not only a curve, but a curve with a change in concavity. Student difficulties with changing rates of change have been well documented in both the mathematics and physics education communities, and reasoning about changing rates of change have been shown to be an essential resource for students entering calculus [6,16,[28][29][30][31]. The majority of students in an algebra-based physics class may not yet have accessible resources about continuously changing rates of change. To adjust the target zone, we developed the Cylindrical Bottle for the GERQN, which asks the same question but with piecewise linear graphs. Cylindrical Bottle includes only one instance where the rate of change changes-at the bottle neckwhich is intended to measure how students may be reasoning about changing rates of change but within their zone of proximal development.
Preliminary results on the GERQN show that 68% of algebra-based students chose the correct answer on Cylindri-The graph at right represents how fast two children are growing vs time. The children are named Alex and Jordan, and their growth is measured starting on their 10th birthday when they are both the same height. Which of the following choices best describes how much the children have grown in the year shown? a. Alex and Jordan have grown the same amount. b. Alex has grown more than Jordan. c. Jordan has grown more than Alex. d. The graph does not provide enough information to compare how much the two children have grown.
FIG. 4. The GERQN item "Growth," where students are asked to compare the growth of two children using a graph of growth rate. The correct answer is B, as Alex has a higher growth rate during the entire year.
cal Bottle, compared to 72% of calculus-based students who chose the correct answer on Spherical Bottle (N = 225 for algebra-based students and N = 240 for calculus-based students, see Fig. 3). For calculus-based students, the most common incorrect answer is E (13% of students), where the concavity is reversed. On the GERQN version, the most common incorrect answer among algebra-based students is B (15% of students), where the rate of change in the bottle neck is slower than the rate of change in the bottle's body. While we do not suggest that we make quantitative statements by comparing across populations, the similarity in difficulty on the item and parallel structure of the most common incorrect answer suggests that the cylindrical adaptation is likely appropriate for the algebra-based sequence, and in their zone. Some items, however, did not need to be adjusted despite appearing to some experts as though they may require calculus. The item we will refer to as "Growth", shown in Fig. 4, illustrates the balance between reasoning about rates of change and requiring calculus procedures to solve a problem. Growth was originally developed for the PIQL based on an item from the Precalculus Concept Assesment and prior work in PER that use same or similarly shaped graphs, but in the context of two cars driving [27,32,33] . The context was modified for the GERQN and remains isomorphic to the original PIQL item. Much has been written in prior research about the intersection of these curves through both a dual process lens and other theoretical frameworks [27,32,33], and our data support the current understanding that the intersection is a tempting answer choice: 45% of calculus-based students and 55% of algebra-based students choose A.
However, what we find novel and most interesting about this question in our current work is a debate amongst physics experts as to whether or not the item requires calculus.We suggest that this item does not require calculus because one can reason that Alex grows at a faster rate than Jordan the entire time. However in the faculty focus group, a discussion emerged about whether this item required calculus knowl-Original Revised "Which one of the following best describes..." "Which of the following choices best describes..." "Which of the following best represents..." "Select the choice below that best describes..." "Select the single best choice below." "Which of the following expressions helps "Which of the following expressions can you use to find out..." figure out..." "Which of the following is an expression for" edge when faculty members expressed solving the problem "required area under the curve" reasoning. Some experts in the focus group found it challenging to reason without using calculus even though the calculus-free reasoning is more straightforward-Alex is always growing faster. After discussion between the researchers conducting the interview and the participants, unanimous agreement was reached that the item could be solved without calculus, but one faculty member added that their students would be unlikely to reason that way. We suggest that experts are so entrenched in a calculus lens of physics that for some instructors it requires more effort to reason about rates of change in any other way.

C. Translation into Flemish
During the translation process, we gained valuable insights into how to make the GERQN less cognitively demanding to read for non-native English speakers: (1) the nuances involved in translating specific mathematical terms such as "rate," and (2) the care required in establishing clear, consistent grammatical structures for question stems.
The first lesson we offer relates to the translation of the word "rate," which we learned from our Belgian collaborators has no direct translation in Flemish. Instead, it is translated as "speed." This made us wonder about what kinds of mathematical terms we may assume are ubiquitous across languages, and for what other languages this may be commonplace. After a quick investigation into other languages, we learned Spanish also uses the equivalent of "speed" to refer to rate of change in mathematics. For the purposes of the GERQN, this made little difference as we had already removed all instances of the word "rate" as part of lowering the reading comprehension. However, we include this note here as it may provide insight for other instructors as they con-sider writing questions and otherwise communicating with students that are non-native English speakers.
We also implemented changes across the entire inventory to ensure clear, consistent question statements. On the PIQL, question prompts in subtle ways, including statements such as "Select the best statement below..," "Choose the correct expression...," and "Which of the following helps figure out..." (See Table I). In our discussions with Belgian researchers, it was pointed out that these varying statements unnecessarily increased the workload to translate, and some words were challenging to translate exactly with the same implied meaning. In particular, many of the statements obscure the subject in question. For example, "Which of the following..." and the phrase "helps figure out" were both challenging to translate due to the nuanced grammatical structure. To address this issue, we chose to change all the question prompts to one of two uniform statements: "Which of the following choices best describes..." or "Which of the following expressions can you use to find out...", as shown in Table I. These prompts were reported to be more easily understood by Belgian students, and we expect are also more easily understood by native English speaking and other non-native English speaking students alike.

IV. RECOMMENDATIONS FOR INSTRUCTION
In developing the GERQN, we identified some key features of algebraic reasoning and item writing that may be useful to instructors. We interpret the results from Cylindrical Bottle to mean that graphical representations that include curvature may be out of scope for algebra students. However, piece-wise constant slope functions are in bounds, and can be used in assessment. The idea that the intersection in Fig. 4 indicates the same height was prevalent in algebra-based students' responses as they are most likely to select "both children grow the same amount," consistent with prior research [32,33]. We suggest the resource of recognizing sameness could be leveraged in future learning about changing rates of change. We also note that an over-reliance on sophisticated mathematics can make it difficult to recognize language-based (rather than procedure-based) quantitative reasoning resources that many students can access and use to reason about rates and accumulation.
In our translation process, we identified some useful ways to consistently and clearly pose a question that was easily translatable into other languages. Employing simple grammatical structures and taking care when using specific mathematical terms such as "rate" were identified as useful first steps to ensuring language was clear for native and non-native English speakers alike.
[1] P. W. Thompson, Quantitative reasoning and mathematical modeling, in New perspectives and directions for collabora-tive research in mathematical education, Vol. 1, edited by