We present a technique to fabricate ultrathin (down to 20 nm) uniform electron transparent windows at dedicated locations in a SiN membrane for in situ transmission electron microscopy experiments. An electron-beam (e-beam) resist is spray-coated on the backside of the membrane in a KOH-etched cavity in silicon which is patterned using through-membrane electron-beam lithography. This is a controlled way to make transparent windows in membranes, whilst the topside of the membrane remains undamaged and retains its flatness. Our approach was optimized for MEMS-based heating chips but can be applied to any chip design. We show two different applications of this technique for (1) fabrication of a nanogap electrode by means of electromigration in thin free-standing metal films and (2) making low-noise graphene nanopore devices.