Single-cell sequencing allows measuring individual cells' molecular features and their responses to perturbations. Understanding which cells respond to a particular perturbation and how these responses vary across populations can be used to, for example, improve vaccine immunogenicity. However, an exhaustive exploration of single-cell perturbation responses in every population is usually experimentally unfeasible. Several machine learning models have been developed to predict perturbation responses, but they are limited to single-modality data. Single-modality data alone, such as only transcriptomics, is not suited to capture all cell responses accurately. For example, the identification of immune responses requires transcriptomic and proteomic measurements. Here, we introduce cellPMVI, a method built to predict perturbation responses from multi-modality data. cellPMVI combines the single-cell data modeling from scVI with a mixture-of-experts posterior integration to allow for multi-modality input data. In this work, we validate cellPMVI for immune response prediction of adjuvants across populations. The model is trained on two-modality CITE-seq data containing gene and protein measurements from three different populations. We show that cellPMVI can model both modalities of the CITE-seq data without information loss in either modality and predict immune responses with a high correlation to the observed responses across different populations. Hence, cellPMVI is the first model to capture and predict immune response for multi-modality data with the potential to be applied for other perturbations, such as drugs.