Contrastive Learning for Multi-Modal Medical Imaging

Abstract: Positron Emission Tomography combined with Computed Tomography (PET/CT) plays a critical role in oncology by integrating functional and anatomical imaging. However, the scarcity of annotated medical imaging data, combined with the complexity of the dual-modality information, presents a significant challenge for supervised learning approaches. This work explores a self-supervised learning framework for PET/CT imaging that does not rely on manual labels and instead aims to learn meaningful representations by leveraging the naturally paired modalities. Our proposed method adapts the contrastive learning paradigm, specifically inspired by the SimCLR framework, to the context of PET/CT imaging. Rather than creating artificial views through augmentations alone, the approach constructs positive pairs from aligned PET and CT slices of the same anatomical region. The intuition is that, while PET provides metabolic information and CT provides anatomical structure, both reflect complementary aspects of the same underlying physiology. This modality-aware strategy tries to enable the model to learn features that are invariant to modality-specific artifacts but sensitive to shared anatomical and pathological cues. Three architectural variants are developed and compared: a single-branch model that treats PET and CT as interchangeable inputs, a dual-branch model with independent encoders, and a single-backbone model that receives both modalities at once, and applies contrastive learning within the output embedding. These architectures are evaluated in a downstream classification task that predicts whether a given slice contains a lesion, based on binary labels derived from 3D tumor segmentations.