In science, technology, engineering, and mathematics education there has been increased emphasis on teaching goals that include not only the learning of content knowledge but also the development of scientific reasoning skills. The Lawson classroom test of scientific reasoning (LCTSR) is a popular assessment instrument for scientific reasoning. Through large scale applications, however, several issues have been observed regarding the validity of the LCTSR. This paper will review the literature on the assessment of scientific reasoning and provide a detailed analysis of the current version of LCTSR in regards to its validity and measurement features. The results suggest that the LCTSR is a practical tool for assessing a unidimensional scale of scientific reasoning. The instrument has a good overall reliability for the whole test. However, inspections of individual question pairs reveal a variety of validity concerns for five of the total twelve question pairs. These five question pairs have a considerable amount of inconsistent response patterns. Qualitative analysis also indicates wide ranging question design issues. Therefore, assessment of subskills involved with the five question pairs may have less power due to their questionable validity.