A Case Study of a School District Assessment System and its Correlation with Student Performance in Physical Sciences

This preliminary study explored the relationship among several school-level accountability measures and student physics and chemistry performance in New York City public schools. The NYC Department of Education prepared a three-year Quality Review and annual Progress Report for each high school in the district, assigning grades to assess overall school quality. Both scores were based upon a number of factors about the school including student progress in all subject areas, college and career readiness, attendance, and graduation rates. These scores presented consequential validity in the form of reputation and school desirability for students and their families. Concurrently, the New York State Education Department published annual student performance data on high-stakes Regents examinations in several content areas, including the physical sciences (physics, chemistry). Statistical relationships among these variables were explored to determine whether Quality Review and Progress Report scores and sub-scores were correlated to student performance in the physical sciences. Data showed that physics performance was not related to overall school quality but positively correlated to college and career readiness; chemistry was related to all of these measures. From a policy perspective, such school assessment measures have implications for the value placed upon physics education in urban secondary schools.


INTRODUCTION
Recent calls for science education reform have emphasized the need for a more educated workforce to meet the needs of an increasingly technological society [1].Legislation has simultaneously called for greater rigor and accountability in the nation's public schools to achieve this goal [2].Accountability systems in U.S. education have often been viewed with skepticism, with no one method having broad support among educators, administrators, and policy makers.However, there has been consensus that such systems should promote positive change in schools and classrooms.
This study aims to examine two of the school assessments in place in New York City Public Schools, known as Quality Reviews and Progress Reports.These measures will be compared with student standardized test performance in physics and chemistry in an effort to establish their predictive value for student success in the physical sciences.The researchers hypothesized that high performing school reports would correlate to high student achievement in the physical sciences, as evidence by scores on New York State Regents Exams.
The research questions addressed in this preliminary study include the following:

BACKGROUND
School assessment efforts have received considerable attention in educational research, particularly since the passage of the No Child Left Behind Act in 2001 [3].In an attempt to improve schools through stricter accountability measures, high stakes testing has been implemented throughout grades K-12 in several subject areas including science (though it is only required to be tested once in grades [9][10][11][12].Although the law succeeded in highlighting disparities in educational achievement for traditionally underserved students, there have been negative consequences, with some studies reporting that science has suffered from a disproportionate emphasis on reading and mathematics [4,5].Another reported consequence has been the decrease in rigor in science curricula to accommodate standards-based testing preparation [6].Students in New York State must pass one Science Regents Exam to graduate with a Regents Diploma, usually Living Environment, where the raw passing score happens to be considerably lower than raw scores required to pass Physics and Chemistry Regents Exams [7].
The elective nature of Physics and Chemistry in U.S. secondary education has a history dating back to late nineteenth century [8], and its impact is still seen today.For traditionally underserved students, access to and participation in Physics and Chemistry coursework has been particularly low [9,10].This is also true in New York City, yet the State has provided data on school-wide student performance in these sciences in the form of standardized tests.This makes the district a particularly rich context for exploring issues related to physical science performance and school quality.

RESEARCH DESIGN Context
The data for this study were publicly available from the New York City Department of Education, the largest system of schools in the U.S., serving approximately 1.1 million students.The racial composition of the student body during 2013-14 was 15.3% Asian, 28.3% Black, 40.2% Hispanic, 14.5% White, and 1.3% other; 78.9% of these students were eligible for free or reduced lunch [11].This school district has long been the largest and one of the most diverse high needs educational settings in the U.S. The district's location in New York State makes it part of a systemic standardized testing system that has been in place since the 1865 [12].Regents Exams are given to high school students upon completion of science coursework in Physics, Chemistry, Earth Science, and Living Environment.Students must earn a scaled score of 65% to pass an exam and 85% to earn the "mastery" designation.Students must take three science courses including Living Environment and one physical science course (Physics, Chemistry, or Earth Science), as well as one Regents Science Exam, to graduate with a Regents diploma [13].Consequently, these high stakes exams are a consistent concern for students, teachers, and administrators.
School assessments and Regents scores were analyzed for all diploma granting high schools in the city (grades 9-12), which enrolled 240,000 students.During the 2013-14 academic year, these students matriculated in various science courses as indicated in Table I [14].The column "other sciences" represents students who have primarily enrolled in Earth Science, which typically has higher enrollment than Physics and Chemistry combined.

Description of School Evaluations
New York City Public Schools employed three different measures to assess school quality: Progress Reports (annual), Quality Reviews (every three years), and the New York State School Identifications.This study analyzed the first two assessments.
The Progress Report has been issued annually each fall with the purpose of "encouraging principals and teachers to accelerate academic achievement toward the goal of career and college readiness for all students" (p.1, [15]).The letter grade was based on the numeric scores (0-100) assigned with the report (A through F) and measured the school's contribution to student learning in five areas: 1) Student Progress; 2) Student Performance; 3) School Environment; 4) College and Career Readiness; and 5) Closing the Achievement Gap.The results were reportedly controlled for student characteristics (such as poverty and English language proficiency) to eliminate confounding variables.
The Quality Review was based upon site visits by educators who assessed how well the school was structured to support student learning.The Quality Review was evaluated by a four-point scale (Well Developed, Proficient, Developing, and Underdeveloped) with 20 items of five sub-domains.One of the metrics for the Quality Review measured the efforts of the school to "consistently engage the school community and use data to set and track suitably high goals for accelerating student learning" (p.2, [15]).

Methodology
Three years of scores (2010-12) from the Progress Reports and Quality Reviews were categorized as discrete variables in SPSS.The numeric scores (0-100) of Progress Reports were used for the analysis (mean = 60.8, std.dev.= 13.2, minimum = 19.6,maximum = 98.7).Quality Reviews were evaluated by a four-point scale with 20 items of five subdomains.Categorical principle component analysis was used to create the composite scores of Quality Reviews (mean = -0.04,std.dev.= 1.01, minimum = -1.48,maximum = 2.15).These two continuous variables were correlated to several measures, including Regents scores in Physics, Chemistry, Living Environment, and Earth Science.
A Pearson correlation matrix was generated to view significant relationships.Since the Quality Reviews were performed every three years, the threeyear average of Regents scores for individual schools was used in the analysis.Data were only included for schools that had both Progress Reports and Quality Reviews (n=108).

FINDINGS
The data in Table II show correlations between school-wide Regents Exam passing rates in the sciences and Progress Report and Quality Review scores.As expected, Living Environment and Chemistry scores correlated with positive school assessments.However, Physics did not correlate to positive school reviews.This indicates that a school in New York City may be considered excellent by district established metrics, yet still had significant numbers of students performing poorly on the Physics Regents Exam.Physics was the only science for which this pattern was evident in both the threeyear average and individual years.Earth Science scores were consistently mixed when measuring school quality, suggesting further study is necessary to resolve the disparity.
To explore the issue of Physics performance and school reviews in greater depth, disaggregate correlations were calculated for the sub-scores of the Progress Reports.Physics performance was not correlated with school scores in Student Progress, Student Performance, School Environment, and Closing the Achievement Gap.However, Physics performance was correlated with College and Career Readiness (p <.05).Table III displays the correlations between the school quality grades and Regents passing rates for all exams required for graduation.As expected, the passing rates for all four required courses -Living Environment, English, Global History, and Algebracorrelated to positive Progress Reports and Quality Reviews.
Table IV represents the correlations between school quality scores and two school-level characteristics: socioeconomic status and the percentage of underrepresented minority students.The correlation with the proportion of Black and Hispanic students is troubling, suggesting disparate impact regarding the opportunity for traditionally underserved students to attend high quality schools.The correlations were mixed for socioeconomic status, requiring further review.

CONCLUSIONS
The results of this study imply that physics performance is not a significant consideration when assessing overall school quality in New York City Public Schools.Schools with high performance marks on the citywide Progress Reports and Quality Reviews tended to graduate students with demonstrated proficiency on core graduation requirements and Chemistry.However, it is perplexing to consider that a school's assessed quality had no correlation with the success of students in Regents Physics.Perhaps schools were directing resources towards remedial coursework and programs that would bolster graduation rates, a common outcome of test-based accountability policies [16].Regardless, the assessment systems in place need to be realigned so student success in physical science coursework is acknowledged and rewarded.
Another notable finding is the correlation between Physics performance and the College and Career Readiness metric in the Progress Report.This criterion measured post-graduation outcomes such as success in college, rigorous vocational programs, and public service.This finding is consistent with research that has suggested physics is an important gateway course for college success, particularly in STEM fields [17].Through future research, this correlation may be further substantiated to warrant additional resources for schools that provide the opportunity for students to participate and excel in physics coursework.
Finally, this research highlights the need to promote physics achievement in our nation's high needs schools.These schools have traditionally low numbers of students taking physics [18], which is itself problematic, but a secondary issue is the deficient teaching and/or learning for those electing to take physics.The low passing rates imply a lack of administrative priority in providing access to rigorous physics education.A robust science assessment system would provide recognition for opportunity to learn physics, particularly for those students who need it most.

TABLE I .
Science enrollment data for New York City public schools, 2013-14.

TABLE II .
Pearson correlation matrix for school assessment scores and Regents science exam scores.

TABLE III .
Pearson correlation matrix for school assessment scores and Regents exam scores for courses required for graduation.

TABLE IV .
Pearson correlation matrix for school assessment scores and school population characteristics.