Remarkable variances were observed in permanent deformation curves of a dense asphalt mixture captured in triaxial repeated load permanent deformation (TRLPD) tests. In order to explore the causes of such variances, each permanent deformation curve was characterized firstly by an indicator (B) of the permanent deformation rate at a steady state stage. Meanwhile, a linear relationship was found between the values of indicator B and the resilient moduli of specimens after 1000 load repetitions. Secondly, a profound discussion on the potential relationship between resilient moduli and air voids content was performed. The discussion showed that air voids content appears to have no contributions to the observed variances. Nevertheless, three failure modes, tension failure, shear failure and shear failure with barreling, were observed in the X-ray Computed Tomography (CT) images of failed specimens subjected to triaxial compressive stresses. Furthermore, a significant correlation between failure modes and the resilient moduli was found by means of one-way analysis of variance (ANOVA). Moreover, it was found that the internal weak zones observed in the CT images of intact specimens dominate the behavior of the permanent deformation of asphalt mixture and the internal structure is the essential cause of such variances in the case of this study. This exploration highlights the influence of the internal structure on the mechanical performance of asphalt materials and thus recommends that considerable attention should be paid on the internal structure other than the volumetric properties when selecting test specimens.