Providing individualized scaffolding for physics problem solving at scale remains an instructional challenge. We investigate (1) students' perceptions of a Socratic AI chatbot's impact on problem-solving skills and confidence and (2) how the specificity of students' questions during tutoring relates to performance. We deployed a custom Socratic AI chatbot in a large-enrollment introductory mechanics course at a Midwestern public university, logging full dialogue transcripts from 150 first-year STEM majors. Post-interaction surveys revealed median ratings of 4.2/5 for knowledge-based skills and 4.0/5 for overall effectiveness. Transcript analysis showed question specificity rose from approximately 10–15% in the first turn to 75 % by the final turn, and specificity correlated positively with self-reported expected course grade (Pearson r = 0.43). These findings demonstrate that AI-driven Socratic dialogue not only fosters expert-like reasoning but also generates fine-grained analytics for physics education research, establishing a scalable dual-purpose tool for instruction and learning analytics.