Using a computer with blood on your hands can be a messy affair, so surgeons typically have another clinician navigate radiologic images during an operation. A few teams around the world have been working on using gesture recognition technology coupled with 3D cameras to allow for hands-free interaction with image viewers (see flashbacks below).
Some of the challenges of creating a practical gesture based interface for the OR is the complexity of the environment, the constant movement of the people within it, and the importance that context plays in interpreting what’s going on. Researchers at Purdue University have developed a system that uses a Microsoft Kinect 3D camera to continuously track a surgeon and make smart decisions based on the context of hand movements.
From Purdue:
The researchers validated the system, working with veterinary surgeons to collect a set of gestures natural for clinicians and surgeons. The surgeons were asked to specify functions they perform with MRI images in typical surgeries and to suggest gestures for commands. Ten gestures were chosen: rotate clockwise and counterclockwise; browse left and right; up and down; increase and decrease brightness; and zoom in and out.
Critical to the system’s accuracy is the use of “contextual information” in the operating room – cameras observe the surgeon’s torso and head – to determine and continuously monitor what the surgeon wants to do.
“A major challenge is to endow computers with the ability to understand the context in which gestures are made and to discriminate between intended gestures versus unintended gestures,” Wachs said. “Surgeons will make many gestures during the course of a surgery to communicate with other doctors and nurses. The main challenge is to create algorithms capable of understanding the difference between these gestures and those specifically intended as commands to browse the image-viewing system. We can determine context by looking at the position of the torso and the orientation of the surgeon’s gaze. Based on the direction of the gaze and the torso position we can assess whether the surgeon wants to access medical images.”
Findings showed that integrating context allows the algorithms to accurately distinguish image-browsing commands from unrelated gestures, reducing false positives from 20.8 percent to 2.3 percent.
“If you are getting false alarms 20 percent of the time, that’s a big drawback,” Wachs said. “So we’ve been able to greatly improve accuracy in distinguishing commands from other gestures.”
The system also has been shown to have a mean accuracy of about 93 percent in translating gestures into specific commands, such as rotating and browsing images.
The algorithm takes into account what phase the surgery is in, which aids in determining the proper context for interpreting the gestures and reducing the browsing time.
Press release: Surgeons may use hand gestures to manipulate MRI images in OR
Study in Journal of the American Medical Informatics Association: Hand-gesture-based sterile interface for the operating room using contextual cues for the navigation of radiological images
Flashbacks: Siemens Kinect Based Hands-Free Radiological Image Browser; Coolest Radiological Image Browsing Just Got Cooler; Kinect-Powered Surgical Robot to Replace Scrub Nurses?; Microsoft Kinect 3D Camera for Hands-Free Radiologic Image Browsing
