Enhancing Open-Surgery Gesture Recognition Using 3D Pose Estimation

Abstract: Gesture recognition is the task of identifying gestures at each frame in a video. In surgery, this can automate workflow detection and highlight unusually long actions. Although recent advancements have improved surgical gesture recognition, much research relies on simulations or minimally invasive surgery. These methods don't capture the complexities of real-world operating rooms and aren’t applicable to all surgeries. Data from open surgeries is crucial, reflecting the variability and challenges of actual clinical workflows, thereby providing a solid foundation for meaningful applications. Most studies focus solely on video data. In this study, we collect data from open surgery wound closures and evaluate the impact of incorporating surgical tool pose estimation and 3D predictions of the surgeons' hands to enhance these results.