This study investigates the effect of computer animation on assessment and the conditions under which animation may improve or hinder assessment of conceptual understanding in physics. An instrument was developed by replacing static pictures and descriptions of motion with computer animations on the Force Concept Inventory, a commonly used pencil and paper test. Both quantitative and qualitative data were collected. The animated and static versions of the test were given to students and the results were statistically analyzed. Think-aloud interviews were also conducted to provide additional insight into the statistical findings. We found that good verbal skills tended to increase performance on the static version but not on the animated version of the test. In general, students had a better understanding of the intent of the question when viewing an animation and gave an answer that was more indicative of their actual understanding, as reflected in separate interviews. In some situations this led students to the correct answer and in others it did not. Overall, we found that animation can improve assessment under some conditions by increasing the validity of the instrument.