Affective factors such as anxiety, confidence, and motivation can impair and enhance task performance. Here, we used drift diffusion modeling (DDM) to examine how these variables affect visualization, manipulation, and decision making on a mental rotation task (MRT). The effects of affective factors on visuospatial reasoning are largely unknown, perhaps in part because analyses are generally concerned with overall accuracy and reaction time (RT), without decomposing the stages of processing. With DDM, we decompose performance on a MRT into separate processing components, particularly the speed of information update (drift rate) and the amount of evidence accumulation (decision threshold). 106 adult participants performed a two-alternative forced-choice (2- AFC) MRT, and throughout, they rated their levels of anxiety, confidence, and motivation. We found that although anxiety, confidence, and motivation all impacted drift rate, only confidence affected the decision threshold. Moreover, we observed a unique role for confidence in mediating the links between gender and model parameters, as well as a unique moderating role of motivation in this mediation. Altogether, these findings shed light on the interrelations between affective factors in accounting for mental rotation performance in men and women, including the unique combination of confidence and motivation in explaining the gender difference in mental rotation performance.