Multi View Anesthesia Identification in Operation Room Using Synthetic Data

Abstract: Surgical anesthesia preparation is a critical phase in which medication errors can occur due to visually similar containers, confusing labels, and the pressures of the operating room. This seminar presents a computer vision system designed to automatically detect and distinguish medications in clinical settings, providing an additional safety layer for anesthesia providers. To address real-world challenges such as clutter, overlap, and varying orientations, we constructed a custom multi-camera aluminum frame that captures each scenario from left, right, and top viewpoints. With this frame we collected dataset from Rambam Healthcare Campus. This dataset is used for benchmarking only. To reduce dependence on costly labeled clinical training data, we developed a Blender-based synthetic rendering pipeline that combines parametric 3D vial and ampoule prototypes with real FDA labels to generate diverse multi-view training data. For detection, we use SAM3 in a zero-shot, open-vocabulary setting. For medication representation, we adapt a pretrained DINOv3 using a triplet-margin contrastive objective and Low-Rank Adaptation for synthetic-medications-to-real-general-data domain adaptation. The resulting embeddings are combined through a cross-view matching algorithm that uses both geometric alignment and embedding similarity, producing a fused representation for each physical container. Final identification is performed through few-shot, open-set retrieval against a gallery index, enabling deployment in new hospitals without retraining. This work presents a scalable and adaptable approach for strong performance medication identification.