From: herbt@apollo.sarnoff.com (Herbert H Taylor III) Subject: CONF REPORT: VR Apps at Neurosurgery Conference, Wiesbaden (G), Jun 15-19, 1993 Date: Thu, 29 Jul 93 12:52:26 -0400 International Congress on Minimal Invasive Neurosurgery H.H. Taylor This conference was held in Weisbaden, Germany, during the week of June 15-19. The focus was on emerging techniques in neuroendoscopy, stereotactic surgery and radiosurgery. About 95% of the >300 in attendence were neurosurgeons, which gave the gathering a decided "hands-on" feel but also served to heighten the contrast with many of the non surgical presentations - particularly those claiming to be about "virtual reality". With only a few exceptions, presentations by non surgeons poorly represented the field. Frankly, we have to stop showing our nice, simulated, "here is what the future will be like" fanciful visualizations and representing them as solutions to hard problems. We are going to lose our audience! Neurosurgeons seek solutions to very specific problems - accurate real-time object tracking (with low latency), multimodality rendering (CT & MRI), object "fusion" (i.e. MRI-ANGI with MRI), stereo video overlays, volumetric segmentation, registration and deformation, etc. A number of abstracts submitted to this conference promissed but did not deliver detailed technical treatments of at least some of these issues. Fortunately, the presentations by the neurosurgeons more then made up for the others shortcomings. This is a group which fully understands and appreciates the potential of virtual reality; yes, immersive, real-time interactive "worlds" derived from images of your brain and combined with stereo microscopic real-time views - to form what some have redubbed "augmented" reality. One might say that VR research has been fundamental to the advance of neurosurgery for at least the past 50 years and possibly longer if you consider the very early work in endoscopy begun at the turn of the century. The SOA in neurosurgery involves using preoperative MRI and CT images and real-time position tracking to guide the surgeon through the neuroanatomy using endoscopic probes. You probably know someone who has had their knee "scoped" - well, now you can have essentially the same procedure done on your brain. The reference images are usually displayed in sagittal, axial and coronal views with "cross hairs" on each to mark the location of any active surgical probe. Surgeons can become quite adept at navigating via these views, however, there is obvious interest in being able to use real-time rendered 3D images to navigate by. Current surgical attempts in this regard remain limited by the compute intensive nature of the rendering and more fundamental questions about the accuracy of the resulting images. Accuracy is an issue for both voxel ray tracing and polygonal rendering methods. Is a polygonal or voxelized object sufficiently accurate for surgical targeting? (See paper in July Computer by Kaufmann, et al for a fantastic discussion of voxel methods. Also, Bill Lorensen co-inventor of the marching cubes algorithm will present a surgical navigation system at this years Viz93. Rumor has it that he will also briefly describe his system during his Siggraph tutorial.) In current practice neurosurgeons never fully "trust" their instruments or their images - either 2D or 3D. Only when the view through the microscope confirms the rendered view can they resect. This degree of mistrust will ultimately establish the limits to endoscopy. However, the drive to safer, more effective methods for localizing deep seated lesions will cause them to push the envelope. But do you fully invade the neuroanatomy to gain a direct view, or "trust" the imagery and position tracking? As M.G. Yasargil stated at this conference, "We have no right to go through the brain..." In addition to accurate 3d representation there is also interest in being able to modify the reference volume to reflect anatomical changes during surgery (aka volume deformation). This of course is a Grand Challenge problem similar to dynamic terrain in CIG's BUT with the requirement that any changes to the reference volume must not only be updated in real-time (< 33msecs) but must be "real" accurate as well - within 1-2mm. An exciting development in medical imaging is real-time acquisition via forward looking 3D volumetric ultraousnd. Standard ultrasound systems look sideways, or in effect tell you were you've been, rather then where you are going. Forward looking systems present the view in front of you - what you see in the "headlights" so to speak. Volumetric forward looking ultrasound systems when combined with real-time volume rendering will revolutionize image guided surgery. At least four presentations described instrument tracking in the operating room. I heard no definitive statement regarding latency but a non surgeons description of a system with 250msecs of latency turned a few faces white... Clearly, minimizing the time from when the surgical probe moves to when it is referenced on the display is very critical for any form of image guidance. Acceptable latency is on the order of a video frame time for image guided systems. This ain't flight simulation its brain navigation! Position tracking approaches to surgery fall into four classes: mechanical (robot arms), electromagnetic (i.e. polhemus), video based and ultrasound. A group from Osaka [Taneda,Kato] uses a video overlay system to observe the endoscopic probe. The video and microscopic views are superimposed on one another. Often, the microscopic view is subject to obstruction from unretractable structures. The video extends the surgeons FOV enabling greater accessibility to obstructed lesions and minimizing the need to "look away" during surgery. A number of mechanically based commericial products are available which combine a rigid arm with the endoscope. The arm is mounted to a frame which in turn is physically mounted to the patients head. This is known as "frame based stereotaxy". Position is tracked via the joint motion in the arm as in current boom mounted HMD's. Frame based approaches are problematic for the patient and surgical team in that they require long hours of preoperative setup, they are very uncomfortable (the patient is often awake during procedures) and they encumber the operating area with bulky equipment. There is therefore great interest in developing so called "frameless" approaches where the need for constant registration is obviated by real-time tracking and registration. The burden of constant registration is transfered from the rigid frame to position sensing and tracking systems, which must register the patient and any surgical instruments within the preoperative images. Several frameless systems have been developed in which the patient is marked with fiducials which are imaged during preoperative MRI/CT. These form easily recognizable landmarks for surgical navigation. However, with the trend toward less and less invasive surgeries it becomes difficult to locate persistent sites to place the fiducials. In addition, access to deep seated lessions is not readily supported by this approach. Operating Room of the Future ---------------------------- M.L. Appuzzo of UCLA gave a talk ostensibly on stereotactic guided microsurgery, however, I came away from his half hour with a whole vision of the operating room as a complex visual space. In his various slides I counted 14 different visual surfaces ranging from Picker boxes to video monitors, projection displays, heads up displays for microscopes, etc. Ultimately, "hardcopy" film will give way to "softcopy" display as superbright high resolution displays are developed and all imagery in the OR will be computer based. In addition, functional imagery will be integrated throughout the course of the surgery not just preoperatively. A number of speakers at the conference stated the need for surgical simulation systems. ( Richard Satava has published several excellent articles, for example, see Surgical Endoscopy 7:203-205, 1993) Apuzzo suggested that such capabilities would also be useful in the operating room - for precise preopereative surgical rehersal. The aim is to minimize the operative corridors - i.e. carefully plan the surgery step by step, identify which structures would be navigable, etc. Such developments as multimodality registration, fusion and segmentation would be critical to real-time interaction and effective surgical planning. Giorgi from Milano described a system in which the MRI/CT preoperative imagery can be fused "intraoperatively" with "echographic" (presumedly Ultrasound) or endoscopic imagery. Supercomputer Assisted Surgery ------------------------------ The ultimate system will emerge as robust methods of real-time 3D volume acquisition are developed. This is an active area in MRI imaging and 3D real-time ultrasound acquisition systems are in advanced stages of development and early commercialization. In addition, a number of xray and optical microscope based systems are now capable of real-time 3D operation. One computational implication of these developments is that future massively parallel supercomputers will need ultra high bandwidth, transparent I/O and sufficient ops to process volumes in real-time. As far as I know none of the current crop of MPP systems (MasPar, CM, Intel, NCube, etc) can continuously and transparently acquire and process data from a real-time instrument at a sufficient bandwidth to sustain volumetric acquisition. System bandwidth alone is insufficient to achieve input transparency. There must be a method of transfering the real-time source stream into "system" memory which is transparent to but simultaneous with the rendering process. This is the area of my own interest and the reason I attended the conference. We are developing a system to support endoscopic surgery on the Princeton Engine. On the PE, 172 Mbytes/s of input bandwidth is transparent to the rendering. We can continuously access volumetric data up to 256^3 at 10 volumes/s AND simultaneously ray trace the volume at about 15 volumes/s (250 megavoxels per second). This creates a true real-time 4D environment. Because the input is transparent to the rendering process the volume apears fully dynamic. One application in surgery for this is to use preoperative CT/MRI for coarse navigation (this requires real-time tracking of the probe) and use the real-time forward looking 3D ultrasound acquired at the tip of the endoscope to localize targets and trajectories. The rendered ultrasound appears as a moving volume inside the preoperatively acquired MRI/CT volume image. Final remarks from the pulpit ----------------------------- Several people have asked me why I was so interested in medical applications of VR. I believe my reasons would be of interest to this audience because hopefully they suggest that we can make a difference with this technology as it continues to advance. My own involvement in VR began following my wifes treatment for a brain tumor in 1987. She was nine months pregnant and suddenly lost sight in both eyes. The tumor was pressing directly on the optic nerve. In the frantic days which followed we were confronted with a series of very difficult decisions - made even more difficult by significant differences in medical opinion. Two diametrically opposed medical opinions were presented to us. One stated that my wife would need a craniotomy, radiation treatments, hormone replacement therapy, suffer permanent infertility, etc, followed by a long recovery process - a horrible prognosis. The other opinion stated that she only needed a minimally invasive "transphenoidal" approach to the adenoma, about five days in the hospital, a few weeks to recover, no radiation, no drugs and hopefully more children. Two neurosurgeons, both well educated, with about the same number of years experience, each with expert opinions which differed like night and day. After checking with several other neurosurgeons from around the country it became apparent that the more optimistic prognosis was the correct one. In fact, we couldn't find a single citation or opinion anywhere which agreed that a craniotomy was the correct approach. In over 400 medical abstracts reviewed not one described craniotomy for my wifes kind of growth. Yet around the US and the world patients are receiving craniotomies who do not need them. I have to believe that there are countless patients receiving open surgery now who could be treated with a laparoendoscopy or other minimal invasive technigue simply because the surgeon doesn't know the new method. ( BTW, the baby was born 100% healthy, my wife's eyesight returned to 20/20, the tumor was removed without a craniotomy and she has had no recurrence in 6 years - altogether a miraculous set of events. ) Following this episode I resolved to try and make a difference in the way things are done. It became clear to me that only through the development of radical new ways of disseminating knowledge and acquiring and using expert skills could any real impact be made. In the case of neurosurgery such skills uniquely combine expert knowledge with technique. What does this have to do with Virtual Reality? I think three elements relate directly: Km--More--(85%)m 1) Access to Information. Virtual consultation. Patients are usually refered to surgeons by GP's or non surgeon specialists. In the case of neurosurgery it is usually by a neurologist. It amazes me that in such a highly technical field knowledge about advances to the state of the art are still conveyed primarily by periodical! As connectivity advances it will eventually be possible for experts of all kinds to share information spaces, or "Infospaces" as James Leftwich has put it. What form these will take is unclear but accessing them must be painless, like walking into a conference room and speaking directly to the "expert" whether real or verisimilitude. 2) Training. Neurosurgeons are trained by other neurosurgeons. They perfect a specific technique for each kind of surgery they perform. Unfortunately, taking six months off from your practice to learn new techniques does not pay the bills or keep your staff happy. As Virtual reality developes it will be possible to construct surgical simulators of such accuracy that surgeons can effectively advance their skills without "leaving home". It is claimed that there are pilots who have flown new kinds of aircraft for the first time in a commercial flight - without first having ever flown the plane in practice. I think it will be a few decades (if ever) before we see a surgeon make a similar transition! Having said that it might be entirely possible for an experienced surgeon to practice and ultimately perfect a new technique via simulation. This is certanily going to be the case for various laparoendoscopic procedures. 3) Broader surgical application of minimal invasive technique. Endoscopic surgery is limited primarily by the imaging modalities which support navigation. The development of real-time volumetric imagers such as forward looking ultrasound will enable intraorgan surgical procedures to further advance. Advances in instrument position tracking, the emergence of HDTV resolution HMD's, image manipulation, etc will largely compliment this effort. These imaging developments will include "volumetric" morphological operations such as deformation, segmentation, registration and fusion. These are all problems of signifcant technical scope which lie at the heart of further advance in surgical methods. July 1993.