Over the last decade, the "gender gap" in physics conceptual inventory scores has been extensively studied by the physics education research community. Researchers have identified many factors that influence the overall differences in post-test scores between men and women. More recently, it has been shown that the Force Concept Inventory (FCI) contains eight items that are substantially unfair; six are unfair to women, two are unfair to men. The Force and Motion Conceptual Evaluation (FMCE) and the Conceptual Survey of Electricity and Magnetism (CSEM), however, contain fewer unfair items. In this work, results from prior studies are used to further explore the gender gap in five large samples of conceptual inventory data: the FCI (N₁ = 3663), the FMCE (N₂ = 2551), (N₃ = 3719), and the CSEM (N₄ = 1767, N₅ = 2439). The gender gap in these samples is partitioned into four components: the gender gap resulting from the student's academic performance, the gender gap resulting from prior preparation in physics, the gender gap resulting from instrumental fairness, and the gender gap of students with equal academic performance and physics preparation on the fair instrument. For all samples, very little of the gender gap was explained by differences in academic performance between men and women, measured by ACT or SAT math percentile scores or physics test average. The percentage of the gender gap resulting from instrumental fairness varied across samples from 30% in the FCI to 2% to 6% in the CSEM. A substantial part of the gender gap in four of the five samples (30%–40%) was explained by differences in prior physics preparation, measured by pretest scores on the conceptual inventories. Further correcting for conceptual physics prior preparation using the post-test score in the previous class reduced gender differences substantially.