To study the relation between the microstructure of white matter in the brain and hearing function in older adults we carried out a population-based, cross-sectional study. In 2562 participants of the Rotterdam Study, we conducted diffusion tensor imaging to determine the microstructure of the white-matter tracts. We performed pure-tone audiogram and digit-in-noise tests to quantify hearing acuity. Poorer white-matter microstructure, especially in the association tracts, was related to poorer hearing acuity. After differentiating the separate white-matter tracts in the left and right hemisphere, poorer white-matter microstructure in the right superior longitudinal fasciculus and the right uncinate fasciculus remained significantly associated with worse hearing. These associations did not significantly differ between middle-aged (51–69 years old) and older (70–100 years old) participants. Progressing age was thus not found to be an effect modifier. In a voxel-based analysis no voxels in the white matter were significantly associated with hearing impairment.