Evaluating GNN Explainer Faithfulness in Molecular Property Prediction Using Comprehensiveness and Sufficiency

Abstract

Predicting properties, such as toxicity or water solubility of unknown molecules with Graph Neural Networks has applications in drug research. Because of the ethical concerns associated with using artificial intelligence techniques in the medical field, explainable artificial intelligence techniques are used to explain how GNNs make their decisions. To evaluate the performance of those techniques, different metrics are used. The BAGEL benchmark proposes four such metrics, designed to be useable with any GNN explainer. Of these, the applicability of faithfulness was investigated in molecular property prediction, measured by the submetrics of comprehensiveness and sufficiency. While comprehensiveness and sufficiency were designed to be task agnostic, several shortcomings were identified that make it unsuitable for molecular property prediction. Future recommendations are to investigate other pre-established faithfulness metrics or to develop ones that do not require splitting molecules.