Background: Multiple sclerosis (MS) affects brain structure and cognitive function and has a heritable component. Over a 100 common genetic risk variants have been identified, but most carriers do not develop MS. For other neurodegenerative diseases, risk variants have effects outside patient populations, but this remains uninvestigated for MS. Objectives: To study the effect of MS-associated genetic variants on brain structure and cognitive function in the general population. Methods: We studied middle-aged and elderly individuals (mean age = 65.7 years) from the population-based Rotterdam Study. We determined 107 MS variants and additionally created a risk score combining all variants. Magnetic resonance imaging (N = 4710) was performed to obtain measures of brain macrostructure, white matter microstructure, and gray matter voxel-based morphometry. A cognitive test battery (N = 7556) was used to test a variety of cognitive domains. Results: The MS risk score was associated with smaller gray matter volume over the whole brain (β_standardized = −0.016; p = 0.044), but region-specific analyses did not survive multiple testing correction. Similarly, no significant associations with brain structure were observed for individual variants. For cognition, rs2283792 was significantly associated with poorer memory (β = −0.064; p = 3.4 × 10⁻⁵). Conclusion: Increased genetic susceptibility to MS may affect brain structure and cognition in persons without disease, pointing to a “hidden burden” of MS.