Discrimination between potentially immunogenic protein aggregates and harmless pharmaceutical components, like silicone oil, is critical for drug development. Flow imaging techniques allow to measure and, in principle, classify subvisible particles in protein therapeutics. However, automated approaches for silicone oil discrimination are still lacking robustness in terms of accuracy and transferability. In this work, we present an image-based filter that can reliably identify silicone oil particles in protein therapeutics across a wide range of parenteral products. A two-step classification approach is designed for automated silicone oil droplet discrimination, based on particle images generated with a flow imaging instrument. Distinct from previously published methods, our novel image-based filter is trained using silicone oil droplet images only and is, thus, independent of the type of protein samples imaged. Benchmarked against alternative approaches, the proposed filter showed best overall performance in categorizing silicone oil and non-oil particles taken from a variety of protein solutions. Excellent accuracy was observed particularly for higher resolution images. The image-based filter can successfully distinguish silicone oil particles with high accuracy in protein solutions not used for creating the filter, showcasing its high transferability and potential for wide applicability in biopharmaceutical studies.