6D Dental Pose Estimation for AR-Assisted Craniofacial Surgery

Abstract: 6D pose estimation is a fundamental task in computer vision with applications in augmented reality (AR). In this work, we introduce a framework for AR visualization of skull anatomy, precisely aligned with the patient's physical appearance to enhance precision in craniofacial surgeries by improving spatial alignment. While existing surgical AR systems primarily rely on marker-based methods for image registration that lack adaptability in dynamic surgical scenarios, our approach leverages the upper teeth as externally visible and reliable landmarks that are rigidly connected to the skull. Specifically, we introduce a novel method for monocular 6D pose estimation of the teeth, involving the estimation of both their 3D position and orientation with respect to the camera. A significant challenge arises from the reliance of deep learning-based pose estimation methods on synthetic training data with precise ground-truth pose annotations, whereas generating a dental pose dataset with realistic oral graphics is impractical on a large scale. To address this challenge, we propose a strategy that directly predicts the pose of the teeth given only their segmentation mask, minimizing the domain gap between synthetic and real-world data by training a pose estimation model on a synthetic dataset of dental segmentations. To obtain these segmentations from real-world images, we introduce a specialized dental segmentation framework. Our method demonstrates high performance across various pose estimation metrics.