In this exploratory study, guided by dual process theories of reasoning, we used a low-stakes diagnostic test in a large introductory calculus-based physics course to test the effectiveness of using multiple-choice question pairs to improve student performance on conceptual multiple-choice questions. As part of this study, we measured students' tendency to engage analytic reasoning via the Cognitive Reflection Test, a three-item questionnaire embedded in a start-of-term diagnostic. These pairs of questions used a common question stem to ask about different but related concepts that students often conflate, such as acceleration and force in the context of a collision. Focusing on three questions from previously piloted question pairs, and controlling for measures of student knowledge and tendency to engage analytic reasoning, we used mixed-effects logistic regression techniques to observe that students who received the question as part of a pair were 7.2 times (95% confidence interval [4.8, 10.9], p < .001) more likely to answer the question correctly relative to having the question alone. Furthermore, the intervention was more impactful for students with a lower tendency to engage analytic reasoning. These results have implications for the design of short-answer physics questions in learning and assessment situations.