This paper presents time-optimal transfer trajectories between planar solar-sail displaced libration point orbits around the $L_1$ and $L_2$ points of the Sun-Earth and Earth-Moon systems. Initial guesses for these transfers are generated as two-segment trajectories, with a fixed sail attitude along each segment. The position and velocity discontinuity between the segments is minimized by means of a genetic-algorithm approach. The time of flight of these trajectories is then optimized by means of a direct pseudo-spectral collocation method as well as a multiple shooting differential correction (MSDC) method. The results of these methods are subsequently compared. Pseudo-spectral collocation was found to outperform MSDC in the solar-sail augmented Sun-Earth system, obtaining trajectories with transfer times as short as 101 days. Differential correction generally yields trajectories with slightly longer flight times than pseudo-spectral collocation, but converges more reliably. In the time-dependent solar-sail augmented Earth-Moon system, pseudo-spectral collocation fails to provide reproducible results. Transfer trajectories with flight times as short as 15.6 days are obtained in the Earth-Moon system using MSDC, but these trajectories feature control discontinuities that are likely to be problematic in practice.