We present a new approach for boundary integral equations for the wave equation with zero initial conditions. Unlike previous attempts, our mathematical formulation allows us to prove that the associated boundary integral operators are continuous and satisfy inf-sup conditions in trace spaces of the same regularity, which are closely related to standard energy spaces with the expected regularity in space and time. This feature is crucial from a numerical perspective, as it provides the foundations to derive sharper error estimates and paves the way to devise efficient adaptive space-time boundary element methods, which will be tackled in future work. On the other hand, the proposed approach is compatible with the current time dependent boundary element method's implementations, and we predict that it explains many of the behaviors observed in practice but that were not understood with the existing theory.