It is only a matter of time before augmented reality becomes a critical and useful tool in the operating room. Surgeons will eventually be able to use holographic displays like Microsoft’s Hololens or Google’s Magic Leap for localizing tumors, highlighting specific anatomy, as a way for instructors to help guide trainees, and for other uses that are being discussed or that have not yet been explored. It’s clear that augmented reality enhanced surgery will be incredibly cool and useful, but we do not yet know the best way for surgeons to control these headsets when they are “scrubbed in.”
Aseptic surgical technique is a concept that came about in the 19th century thanks to the unsung efforts of Ignac Semmelweis and the later more impactful efforts of Louis Pasteur and Joseph Lister. The idea, which is now intuitive, is that during surgery the surgeon, patient, instruments, and drapes must all be sterilized and completely free of microorganisms to minimize the risk of infection. This means that once a surgeon is gowned and gloved, he/she cannot touch anything that is not also sterile. If a surgeon does touch something unsterile, he/she is instantly “contaminated” and may need to change his/her gloves and potentially also the gown. This can make controlling electronic equipment somewhat challenging, especially something that is worn on the head, a classically unsterile area.
There are various options available for controlling electronics in the OR including voice control, optical hand tracking, EMG-based gesture tracking, remotes/controllers, and foot pedals. I will briefly go over each technology and evaluate their relative advantages and disadvantages from a surgeon’s perspective.
Almost all augmented reality devices include voice recognition technology. Voice recognition based control is an attractive option for use in the operating room because it is relatively fast and does not require any physical contact. I have never personally witnessed a voice control based system in the operating room and this may be due to a number of factors. Firstly, the OR is typically quite loud at baseline. There is noise coming from the constant beeping of the pulse oximeter, from the suction which is quite loud, and from the constant verbal communication between operating room staff. The amount of noise present would make accurate voice recognition much more challenging. Furthermore, voice recognition may not provide enough fine tuning to make subtle adjustments in images or to control whatever application a surgeon is using. Theoretically, if the headset was attached to any sort of critical system that could potentially harm a patient, voice control would be far too risky a proposition. A more primitive form of voice recognition could potentially be quite functional, in which an unsterile “operator” is controlling the augmented reality device atop the surgeon’s head based on the surgeon’s verbal instructions.
Going forward it is very likely that most augmented reality devices will have some optically based hand tracking and gesture recognition abilities. Even without built-in tracking external hand-tracking devices like the Leap Motion make adding high quality optically based hand-tracking relatively straight forward. This may be a very attractive solution for intra-operative control of augmented reality headsets. It requires no physical contact, so sterility is not really an issue. High fidelity hand-tracking can make finer gestures and adjustments possible, increasing the control a surgeon would have over his AR device. Unfortunately, there are a few challenges with implementing optically based hand-tracking in the operating room. Typically in a surgery there are multiple hands working in the surgical field including the surgical tech and any assistants or co-surgeons that are present. This may be confusing and potentially disrupt the input to any hand-tracking system. The extreme lighting conditions in an operating room may also decrease the effectiveness of cameras for hand-tracking as there are bright OR lights and glare off of metallic objects to contend with. Finally, it is not uncommon for fluid to splash up in the face and head area during a procedure, and these spatters may obscure the vision of a hand-tracking system.
An alternative to optically based hand tracking would be electromyography (EMG) based gesture tracking. Products like the Myoband and the open-source Gesto platform are worn on the forearm and use skin-based EMG sensors to detect the position of the hand. These devices can be worn under the surgical gown so that they do not interfere with the sterility of the procedure. There is no concern for occlusion or confusion of the device by other people’s hands since it can only sense the hand position of the arm it is currently on. The device would likely communicate with the surgeon’s headset wirelessly. This technology is still somewhat in its infancy, so the precision of control is less than what you could get with optically based hand tracking. Furthermore, there could be a situation in which a gesture is inadvertently triggered which could cause problems mid-procedure.
A handheld remote/controller is a reasonable option for controlling augmented reality headsets. Such a device would be very straightforward to use, highly functional and likely have a short learning curve. On the other hand, electronic devices that can survive sterilization in an autoclave require sophisticated housing that can be quite expensive. An alternative would be to keep the unsterile controller within a sterile bag or sterilized hard-casing. Another potential downside is that having to reach for a separate remote or controller every time it is needed can extend surgical time or add extra steps to an already complex procedure. Relying on a physical remote could be risky since if it is dropped and becomes unsterile it could be unavailable for the rest of the case, rendering the surgeon’s AR headset useless if there is no control alternative or backup remote. One solution to this problem could be to wear a bulky “spacesuit” in the OR which makes the surgeons head sterile. In this case a headset mounted trackpad like that on the Google Glass would be reasonable to use, which could avoid some of the downsides of a free-floating remote. Most surgeons, however, would not want to go through this much expense and trouble just to be able to use a head-mounted trackpad.
A foot pedal is an excellent alternative to a handheld remote and is already frequently used in the operating room to control devices like drills, shavers, fluoroscopy machines, and coagulation wands. Foot controls can provide high precision, are unobtrusive, and can remain unsterile. On the other hand, foot controls need to be relatively simple, so sophisticated control of multiple functions could be challenging. Also, since the controls are not directly visualized erroneous input or accidental triggers are not uncommon.
We have only really scratched the surface of what augmented reality can do in the OR. Ultimately, its function in the operating room will dictate its controller requirements. Different controllers will be useful for different situations. Since there are so many options for the intra-operative control of head-mounted augmented reality devices this is unlikely to be a bottleneck for their use during surgery. Other variables such as weight, image quality, processing power, software, object tracking, and patient privacy will likely be more important limiting factors for the advancement of augmented reality in the surgical world.