Exploring and modeling the spreading process of rumors have shown great potential in improving rumor detection performance. However, existing propagation-based rumor detection models often overlook the uncertainty of the underlying propagation structure and typically require a large amount of labeled data for training. To address these challenges, we propose a novel rumor detection framework, namely, the Uncertainty-Inference Contrastive Learning (UICL) model. Specifically, UICL innovatively incorporates an edge-wise augmentation strategy into the general contrastive learning framework, including an edge-inference augmentation component and an EdgeDrop augmentation component, which primarily aim to capture the edge uncertainty of the propagation structure and alleviate the sparsity problem of the original dataset. A new negative sampling strategy is also introduced to enhance contrastive learning on rumor propagation graphs. Furthermore, we use labeled data to fine-tune the detection module. Our experiments, conducted on three real-world datasets, demonstrate that UICL can not only significantly improve detection accuracy but also reduce the dependency on labeled data compared to state-of-the-art baselines.