Virtual and augmented reality enter the game
The attention directed towards augmented and virtual reality by medical 3D printing (3DP) enthusiasts has been increasing steadily since the release of cheap, commercially available, mixed reality (MR) devices in 2017. A great example of this is how the special area devoted to 3D printing at the Radiological Society of North America’s annual meeting has changed in name from “3D Printing in Medicine Showcase” in 2017, to “3D Printing & Advanced Visualization Showcase” in 2018, to the “3D Printing and Mixed Reality Showcase” in 2019. While the roles of mixed reality technologies in medicine are diverse and numerous, it is often of particular interest to proponents of the superior 3D visualization and interaction that 3D printing can provide, since models generated for printing can easily be engaged with in mixed reality environments with little to no additional work involved.
Justin Sutherland speaking at the 3D Printing in Medicine Showcase at RSNA 2017
Since early adoption of VR/AR in medical visualization has largely come from 3D printing practitioners, these virtual technologies have often been regarded as offshoots from 3D printing, acting almost as alternative printers, rather than novel drivers of innovation in their own right. While there are many similarities between the use of mixed reality technologies and 3D printing in medicine – and in many cases work towards one can be applied towards the use of the other – they are not identical and the considerations for their use can differ significantly. This post will discuss some of the similarities and differences concerning the point-of-care use of these technologies.
Preparing models for 3D printing vs. virtual use
The requirements for 3D models made for mixed reality use differ from those made for 3D printing. In fact it’s fair to say that the requirements for VR/AR are largely less onerous. This reduction in work comes in the form of avoiding post-processing tasks that 3D printing requires such as:
- orienting models with respect to printing platform for optimal printability,
- combining models for printing efficiency,
- avoiding the requirement that model meshes are “watertight”,
- accounting for the minimum resolution of printers,
- incorporating fixtures and supporting structures into models, and
- adding labels for identification of structures and anatomical laterality (since these can be provided through digital meta data in mixed reality)
It’s reasonably common that content creators add non-anatomical cuts through models so that their internal structures can be visualized when printed. On mixed reality platforms, these cuts can be simulated fluidly and in real-time, removing the need for their careful selection as they can changed on-the-fly. Additionally, for mixed reality use, the selection of relevant anatomical structures can be more inclusive and selectively displayed at the discretion of the user, whereas such decisions take more care when committing to a permanent printed model as it will take substantially longer to print the additional desired volume.
Content creation considerations for 3D models destined for augmented and virtual reality also exist, and they are primarily driven by the computational requirements of rendering images and the computational ability of a given system. For VR, computers have to render images for two HD displays, one for each eye, with both displaying data in a synchronized manner at a high frame-rate (typically greater than or equal to 72 Hz). This can impose constraints on the total number of triangles that can exist in a given scene to be rendered, but with today’s high-powered graphics cards, this constraint is rarely a limitation.
Mobile mixed reality devices, however, often have a significantly reduced computational ability and, as such, have more strict constraints on the number of triangles that can be rendered in a given scene. This can potentially create a challenge for model creators that need include enough detail to maintain anatomic fidelity while also avoiding models that have become ‘dis-featured’ due to exceeding model size limitations. In most mobile-use cases currently, the number of triangles in meshes can be reduced through decimation to accommodate rendering constraints. However, fully-detailed models should soon be widely accessible to mobile platforms through the a growing number of cloud-rendering services where images are rendered remotely by high-powered systems and served in real-time to mixed reality displays through Wi-Fi or 5G networks.
Model production and use for 3DP vs. MR
Lab and equipment
The design of 3D content aside, there are also a number of differences when it comes to how one goes about producing and using 3D content with 3D printing vs. mixed reality. Experts in medical 3D printing consistently emphasize that when designing a point-of-care 3D printing program, you simply can’t have too much space. Space can be an extremely limited resource in hospital environments and some radiology-based 3D printing groups even report “borrowing” space from surgical departments since they ultimately benefit from the printed models. Readers who are largely informed by depictions of people using VR and AR for video games might be surprised to consider that the space requirements for such technologies are significantly reduced for the standing-only or seated work-oriented configurations of many platforms. In contrast, gamers often derive entertainment from being able to walk around a virtual environment.
In most cases a 3D printing lab will likely require significant space for printers which come in various sizes, but also for housing supplies, accessories, and post-processing of printed models. In many cases, entire rooms need to be devoted to post-processing with ventilation and equipment to perform activities such as support removal, heat treatment, sterilization, and cleaning. Larger printing equipment may require engineering support to ensure adequate functioning with regular maintenance and a printing-oriented quality assurance program.
For mixed reality use, inventory requirements can include high-powered PCs, head mounted devices (mobile or PC-tethered), controllers, and potentially disposable face cushion covers if hygiene is a concern for shared worn devices. For Some VR systems that use external devices such as cameras or base-stations to perform positional tracking, the placement and securing of these tracking devices will need to be considered.
Using the models and content dissemination
There are significant differences when it comes to actually using models and disseminating 3D content. The obvious difference between interacting with physically printed and virtual models is the ready access to real tactile feed-back that a printed object provides. Being able to touch and manipulate and object with your hands likely has a difficult-to-define benefit for spatial understanding. On the virtual side of things, an increasing array of haptic feedback devices are being implemented with VR platforms. While these technologies may provide varying levels of physical fidelity, they do offer a repeatable and non-destructive way to physically interact with models in surgical simulations.
When it comes to interacting with single instances of 3D objects, there are conveniences and inconveniences for physical and virtual objects. With a physical object in hand, it’s easy to have a collaborative discussion with multiple local participants. To do the same with virtual objects requires multiple mixed reality devices. When it comes to disseminating multiple copies of objects or remote collaboration, virtual platforms have an ability for duplication and discussion at-a-distance that are undeniable. With the increasing abundance of VR and AR technology and the software platforms to use them, the value of remote collaboration has yet to be seen. Virtual models also require no disposal or physical storage.
Another important difference is the time for model availability. Virtual models require no additional time after the 3D model is designed whereas 3D printing can take many hours from model design to having the model in-hand. This might make virtual models a more appropriate option when they are required on-demand and an additional delay cannot be accomodated.
The expanding world of mixed reality
There are some obvious similarities between mixed reality and 3D printing applications in medicine. But ultimately they are very different technologies. In the same way that virtual objects cannot meet physical surgical guide production needs like 3D printed objects can, there are considerable applications of mixed reality technologies that are impossible with objects in the physical world. It’s a category error to consider virtual reality or augmented reality as simple printer alternatives. They are novel human-computer interfaces which have considerable unexplored potential. At Realize Medical, we happen to believe that VR can benefit 3D content creation just as much as it can be a powerful platform for engaging with 3D content once created. That’s why we made Elucis. We see the world of using immersive medical content exploding in the coming years.
