Diagnostic tests for placing students into appropriate courses are commonly used in higher education math and science departments. We found no physics education research examining the validity of such exams, how well they are placing students into appropriate courses, and what guidance these exams can give for better focusing instruction. In this study, we present criteria by which to evaluate the validity and value of a diagnostic test, and we then apply these to the test developed for physics course placement at Stanford University. We examine the data collected from the first year in which all students intending to enroll in an introductory physics course were required to take the test and received an email suggesting which course would best match their level of preparation based on their score. We found that most students followed this advice. We also found that the diagnostic test score was a good predictor of performance in both the introductory physics course for physics majors and the course targeting science and engineering majors. It was not a good predictor of performance in two other courses, likely due to lack of alignment between course objectives and diagnostic measures. We found many individual questions which were particularly discriminating in predicting performance in the various courses, suggesting specific topics on which instruction should focus. We found that 27 of the 38 questions were predictive of performance in at least one of the courses studied, and only 8 questions that were not predictive of course performance or did not show any difference between students enrolled in different courses. These results provide evidence for the validity and usefulness of our diagnostic test, and we offer it to instructors at other institutions who might find it of value.