Always-on sound classification is a desirable but power-intensive function for a variety of emerging Internet of Everything applications. This work explores the accuracy-complexity tradeoff by using summary statistics for classifying semi-stationary sounds. Compared to contemporary solutions including deep learning, this approach requires one to three orders of magnitude fewer parameters and can therefore be trained over ten times faster. We propose a mixed-signal design using N-path filters for feature extraction to further improve energy efficiency without incurring a large accuracy penalty for a binary classification task (less than 2.5% area reduction under receiver operating characteristic curve).