Diamond membrane devices containing optically coherent nitrogen-vacancy (NV) centers are key to enable novel cryogenic experiments such as optical ground-state cooling of hybrid spin-mechanical systems and efficient entanglement distribution in quantum networks. Here, we report on the fabrication of a (3.4 ± 0.2) μm thin, smooth (surface roughness r_q < 0.4 nm over an area of 20 μm by 30 μm) diamond membrane containing individually resolvable, narrow linewidth (< 100 MHz) NV centers. We fabricate this sample via a combination of high-energy electron irradiation, high-temperature annealing, and an optimized etching sequence found via a systematic study of the diamond surface evolution on the microscopic level in different etch chemistries. Although our particular device dimensions are optimized for cavity-enhanced entanglement generation between distant NV centers in open, tunable microcavities, our results have implications for a broad range of quantum experiments that require the combination of narrow optical transitions and micrometer-scale device geometry.