Women continue to be underrepresented in physics despite recent developments focusing on improving representation and equity within the discipline. Several recent papers have investigated the experiences of both white women and women of color within the discipline to understand the issues that most often affect them, and how those issues intersect. What has been less clear from previous research is how these factors contribute to women's decision to stay in physics or leave for another discipline. Much of the literature on persistence has focused on leaving STEM majors for non-STEM majors, but this does not include the possibility of a woman leaving physics for another STEM discipline. In this study, we used a social constructivist lens to investigate the experiences of 5 women in physics at a large research university. We focused specifically on how the women's experiences contributed to their decision to stay in physics (some even choosing to pursue a PhD) or leave physics for another STEM discipline. We find that lack of support, poor treatment by male peers and professors, and interest in physics are all major factors in these women's decision-making process, but that lack of support was most prominent in the decision to leave physics. This provides interesting counter-narratives of women defying adversity in pursuit of their deep interests in physics but also highlights the need for structural changes and action by male physicists to better support women.