Hands-On AI: Robust Hands Detection and Explainable Assessment of Surgical Suturing Using 3D Hands Reconstruction

Abstract: Detecting and assessing surgical activity through hand analysis in the operating room (OR) presents significant challenges. this seminar presents a comprehensive approach to enhancing hand detection and surgical skill assessment using robust AI models and 3D hand reconstruction techniques. The work is divided into two main components: RoHan, a robust hand detection pipeline in operating room (OR) settings, and ExSut, an explainable suturing skill assessment framework. RoHan addresses the challenges of detecting hands in complex surgical environments, characterized by varied glove types, occlusions, multiple hands, and dynamic lighting conditions. The approach employs a semi-supervised domain adaptation pipeline that refines hand detection without relying on labeled medical datasets. Through synthetic glove generation, spatial and temporal filtering, and iterative model fine-tuning, RoHan significantly improves detection accuracy in realistic OR scenarios. ExSut leverages 3D hand reconstruction to assess surgical suturing skills by analyzing hand-tool dynamics. Using video data from medical students’ tests on simulator-based tasks, such as suturing, knot tying, and facial closure, the model extracts both per-frame and global features to predict human-graded skill scores. A multi-stream temporal convolutional network (MS-TCN++) combined with temporal attention pooling allows for interpretable predictions, offering insights into key movement patterns and skill indicators. This seminar underscores the potential of AI to support surgical training and performance evaluation.