Introductory university astronomy courses have emerged as a powerful opportunity to improve science understanding, literacy, and appreciation of the scientific method to a wide range of future citizens and voters. Students in an introductory astronomy course were instructed to record and analyze their naked-eye astronomical observations over a semester (13 weeks). The effect this activity had on students' learning of basic astronomy concepts and attitudes towards astronomy and science were measured using an astronomy concept diagnostic test and attitudes survey administered at the start and at the end of the semester, and compared with a similar introductory astronomy course that did not include the observing assignment. The results suggest that the observing diary is a positive learning experience for the majority of students, and that completing this assessment assists in developing students' deeper engagement with astronomy and astronomy concepts. We recommend the use and assessment of observing diaries as an effective learning activity in introductory astronomy classes.