Multimodal Entity Resolution

Abstract: Real-world entities across modern data ecosystems are inherently multimodal, naturally captured through multiple representations. Yet, traditional approaches to entity resolution rely heavily on single-modal (primarily textual) data, limiting their effectiveness in modern complex use-cases. The few existing multimodal entity resolution systems treat modality fusion as static feature merging, which tightly couples the data types and requires dimensionality modifications and retraining whenever a new modality is introduced. To overcome these limitations, we propose SUMMER, a scalable framework that shifts modality fusion from static feature merging to structural message passing. We explicitly model pairwise comparisons using a heterogeneous multiplex graph structure. We present the use of attention in setting the contribution of each modality to the matching task and present the use of a heterogeneous graph attention network to propose a scalable modular architecture where new modalities can be seamlessly integrated without altering the underlying architecture and requiring only partial training. An extensive empirical study across multiple benchmarks shows our solution to match or exceed state-of the-art baselines, with the largest gains under adversarial settings. We further show that quality improves monotonically with each added modality, and that new modalities integrated via partial training reach near parity with full retraining.