I have been working on an Mixed Reality project in the ETC called MediSIM (Medical Simulated Interactive Mannequin). It is a medical simulation tool using the HoloLens in conjunction with Abe, a physical abdominal simulator, designed for medical students to practice abdominal examination and prepare them for future examinations on patients during real world situations.
Since Microsoft only began shipping the development kit of the HoloLens on the 31st of March, we were working with the first generation of Meta, another augmented reality headset 2 years older than the HoloLens.
However, there were many limitations we faced compared to the HoloLens. The Meta 1 only offers a much smaller field of view which restricted the display by a considerable margin. It was difficult to see the display using the headset due to the resolution and it was extremely uncomfortable to put on even for a short period of time. Despite that, even while developing on this preliminary headset, we received overwhelmingly positive feedback from play testers. Medical students who tried our prototype were very excited about it and felt that it would be immensely useful to aid in their learning experiences. The faculty at ETC also felt that the experience will be meaningful and practical.
Since I have been working with Augmented Reality, I would like to share some of my thoughts about it.
First of all, it is very suitable for educational or medical purposes, where learning by practicing in context is essential. By using Augmented Reality, those contexts can be created and put into real world simulations. Hence, I think transformational games will work really well in Augmented Reality since it provides the opportunity to make things happen in front of you or make impossible possible. If combined with a physical simulator, like Abe, or physical props, the experience will be even more immersive and realistic, providing a way for the player to grasp the knowledge or skill faster. For example, in our project, we superimposed a virtual patient in the augmented world with detailed symptoms so that medical students can not only practice their abdominal examination skills on the physical simulator, but also diagnose the virtual patient. This simulation cannot be realized in the real world because it is impossible for students to find patients readily.
The power of Augmented Reality lies in the private experience the guest has while still being aware of everything in the real world. Even with the headset on, they will still be able to interact with everything around them. This makes it interesting to create social or multiplayer games using Augmented Reality. Imagine playing a multiplayer game with friends, while each of you can see “personalized” information on your own glasses and at the same time, seeing a shared part of the game. Players can even do sneaky things right in front of others’ faces. As game designers, we will need to think about the information that can or cannot be seen by players in order to create different feelings/moments so that the experience can be unique every time.
A unique aspect of the Hololens compared to the Meta 1 is that it can detect space and objects around you. This is a feature that helps improve the experience in Augmented Reality. Based on this feature, characters in the virtual world can “play” with you in your own room by being spatially aware of your environment. You can hide behind an object like a sofa to dodge attacks or “change” the world that you are in by mounting defenses on your dining table. Artificial characters can move around the space, which allows every player to get a unique gameplay experience since their real world environment is unique. Take Microsoft’s RoboRaid game for example, it allows you to blast virtual holes in your walls. Young Conker, Microsoft’s first platformer game, tailors each of the levels you play to your real world.
The common comparison or confusion with Augmented Reality is Virtual Reality. There are several key differences between both technologies.
In Virtual Reality, you are brought into a totally different world where everything is synthetic. You can be transported to a fantasy world or any environment possible. On the other hand, for Augmented Reality, you are placed in a familiar environment while interacting with it in different ways. This results in very different experiences when designing for both technologies. Virtual Reality tends to be more successful in creating fantasy like experiences compared to Augmented Reality. Hence, the inverse should also be true, although not yet proven, Augmented Reality should be more successful in creating experiences that model reality. Due to the nature of the technology, Virtual Reality experiences are primarily isolated whereas Augmented Reality may feel more social.
In terms of presence, they are kind of different. As we can see, VR has already done a quite good job trying to really bring you into the virtual world by letting you see your own “hands” in VR or even do natural hands and feet movements (room-scale VR). The AR technology might still have some way to go before it can really make you feel completely convinced.
I’ve tried and been working with the HoloLens for days. It actually works fairly well and the holograms are uncannily real. However, there are still limitations like the field of view is still somewhat narrow and the device cannot detect the position of the hand yet (I guess they have that but just haven’t released it). It would need to cover much more of the world than HoloLens allows right now. I am looking forward to see what Magic Leap can achieve to bring the technology a huge leap forward.
Magic Leap’s latest video