Spatial modes of light can be used as carriers of information in classical optical communication or as an alphabet in quantum optical communication. In order to exploit the spatial domain, it is required to (de)multiplex different modes from a shared input channel into different output ports. Mode sorters have been employed in free-space and fiber systems but to date have not been realized for planar guided waves. Here we present a general method for compact on-chip sorting of different planar beams with a micrometric footprint and nanometric thickness. The designs were generated using a linkage-tree-based genetic algorithm and were experimentally demonstrated on a surface plasmon polariton platform by sorting of Hermite-Gaussian beams. The method used here can be readily applied to optimize complex, large-scale optical devices involving beam propagation methods.