The study examined sex-related differences in regional gray matter (GM) in 44–48 year old healthy individuals. T1-weighted MRI scans were acquired in 411 subjects aged 44–48 from a random community sample and optimized voxel-based morphometry was applied to detect regional GM difference between men and women, correcting for effects of age, years of education, handedness, and total intracranial volume (TIV). Men had larger brain volumes and higher white matter (WM) to TIV ratios compared with women. Women had higher GM to TIV ratios than men. After controlling for age, years of education, handedness, and TIV, there were no significant differences between men and women in the total GM volumes. Regional sex dimorphism was present, with men having more GM volume in midbrain, left inferior temporal gyrus, right occipital lingual gyrus, right middle temporal gyrus, and both cerebellar hemispheres. Women showed more GM in dorsal anterior, posterior and ventral cingulate cortices, and right inferior parietal lobule. Our results suggest sex dimorphism in GM in middle aged healthy individuals, which is not likely to be explained by brain pathology. These differences may provide the structural brain basis for sex differences in certain cognitive functions.