The high-precision scrap sorting for effective metal recycling can bring substantial environmental and economic benefits. This paper presents a magnetic image sensor that can help to identify the ferrous contaminants inside non-ferrous scraps of large sizes. First, the concept and the theory for detecting ferrous contaminants are described. Especially, an inversion algorithm is proposed to characterize the size and the position of ferrous contaminants inside the main scrap bodies. Then, based on computed and measured results, the feasibility of sensor design using either 1D Hall arrays or 1D pickup coils is demonstrated. Finally, methods are suggested to minimize disturbing signals from very large non-ferrous pieces passing through the slightly uneven magnetic field. The obtained findings in this study may apply not only to nonferrous scraps but many other materials of which the mass ratio of the ferrous contaminant to the main material is small.