Introduction Breast cancer is the most frequently diagnosed type of cancer in women in 2020. Treatment and prognosis of breast cancer is highly dependent on early and accurate diagnosis. In recent years, many studies have evaluated the use of artificial intelligence for the detection of breast cancer in mammographic images. Another imaging modality, breast-specific gamma imaging (BSGI), or molecular breast imaging, has not yet been subject to AI algorithms to detect breast cancer. In this paper, we aim to develop and evaluate convolutional neural networks (CNNs) that can detect malignancies in breast-specific gamma imaging, and evaluate the efficacy of different machine learning classifiers to classify breast tumors based on estrogen receptor (ER) status, progesterone receptor (PR) status and human epidermal growth factor receptor 2 (HER2-neu) status. Methods Three CNNs were created and trained and tested on a total of 3,503 BSGI images. The models varied in complexity in terms of convolutional layers and filter sizes. A semi quantitative lesion segmentation was created based on adaptive thresholding and shape and location analysis. Radiomics features were extracted, and univariate feature selection was applied to disregard redundant features. Different machine learning classifiers, which are widely used in literature for binary classification problems, were evaluated. Results In detecting malignancies in a dataset containing clean breasts and breast with malignant lesions, the best performing network reached an area under the receiving operating characteristic (AUROC) of 0.93, while an AUROC of 0.88 was achieved when using the same networks in the classification of malignant versus benign lesions. The best performing machine learning classifiers were the linear discriminating analysis (LDA) classifier for ER and PR status, reaching accuracies of 75% in both receptors. In Her2-neu prediction using machine learning, the best accuracy of 69% was achieved by the RF classifier. Discussion & conclusions Based on the results presented in this paper, CNNs can accurately detect malignancies in BSGI images, and discriminate malignancies from benign lesions to a certain extent. The combination of radiomics and machine learning, however, is in its current implementation not accurate enough to predict the ER, PR and Her2-neu status in BSGI images. Future research, however, could focus on using a combination of imaging modalities, such as BSGI, MRI and mammography to improve the predictive accuracy of machine learning.