Mixed Reality for research and education

Introduction

(Note: part of this article was written before the full Covid-19 lockdown that started mid-December 2020 in The Netherlands)

In front of me, in an otherwise almost empty SURF office in Amsterdam due to Corona-virus restrictions, I see a man suspected of having Covid-19 sitting on one of our office chairs. He is coughing and is apparently having difficulty breathing. I walk around the chair and the man stays put, ignorant of me observing him. But while I am there in our office, he is not. But yet he is.

A voice-over is describing the patient’s details to me and there are rectangular boards with text and graphs showing descriptive information and vital statistics, such as heart rate, floating just left and right of the man’s head. I use my finger to press a button labeled “Next” in mid-air and the man’s clothing changes into a hospital gown. He now apparently also receives oxygen through a tube in his nose. The voice-over describes that this is the situation two hours later and the set of diagnostic tests that have been performed on the patient so far are reported. In the minutes that follow I witness two more phases in which the patient’s state gradually worsens, ending up breathing with extreme difficulty, wearing an oxygen mask that covers almost his full face.

Covid-19 has never come this close for me in the many months that we’ve been living under the consequences of its existence. I never met a real Covid-19 patient, and technically still haven’t, but this experience is very convincing. So this is what it’s like to become seriously ill with the Corona-virus. I flip up the display of the Microsoft HoloLens 2 I’m wearing on my head and the man is gone, no longer visible, his presence in our office reduced to the truth: he was never physically there in the first place.

The write-up above tries to convey what Mixed Reality technology is capable of providing and how it can show realistic virtual content rooted in the real world. Unfortunately, Mixed Reality is better experienced than described, and the pictures above only give an impression. To be clear: only the man (and air hose equipment) in the photos is virtual content, the rest is a real room. The software application described above is called HoloPatient: COVID-19 and was produced by a group that includes the British National Health Service. It can be installed from the Microsoft store, but only on a Microsoft HoloLens 2 device. But that actually illustrates how far technology and infrastructure for Mixed Reality has come: these kinds of apps, Mixed Reality content and the devices to use them on are becoming more commonplace and easier to use.

In this article we’d like to do three things: 1) provide an overview of Mixed Reality (MR) and what it can provide, 2) describe why MR is interesting to the SURF community and 3) describe some of the relevant issues and challenges with Mixed Reality as a technology.

In a separate article we’ll take a closer look at a specific Mixed Reality headset, the Microsoft HoloLens 2, as the concept of Mixed Reality will become less abstract if we look into a concrete implementation of the technology. It also highlights some of the current technical limitations.

This article is published as part of the activities within the SURF eXploRe project on use of XR for research and education. Within the SURF 2-Year Plan 2021-2022 XR has been identified as innovative technologies to further develop and apply. The SURF eXploRe project is a starting point together with the SURF community to move forward on use of XR in research and education.

Mixed Reality

The technology for VR, AR and MR has been steadily improving in recent years, but unfortunately the terminology has not. “Virtual Reality” (VR) is relatively well-known and well-defined as providing a fully computer-generated virtual 3D environment to a user, mostly through headsets such as the Oculus Quest. But “Mixed Reality” (MR) is closely related to “Augmented Reality” (AR), both of which might mean several different things depending on who you ask. And all of these concepts are part of the umbrella term Extended Reality (XR), which makes it all a bit confusing.

So what we mean by “Mixed Reality” here is the application of technology to add virtual content rooted in the physical world in a realistic way. This includes natural interaction with such content. Microsoft and others refer to such virtual content as “holograms”, which strictly speaking would refer to a different technology (think the security hologram found on a credit card). But the term suffices to get the idea across, also thanks to sci-fi movies frequently showing similar concepts. Holograms are three-dimensional computer-generated virtual objects that look and act as if they are part of the real world. Holograms stay in place as you walk around them, they can be viewed from all angles, they occlude real objects behind them, any sounds emitted from them appear to be coming from their location, etc. In advanced cases holograms may even be influenced by real-world lighting, in order to enhance their integration with reality.

Augmented Reality (AR) shares a lot of similarities with what we call Mixed Reality above, but the crucial distinction we use here is that 1) AR content is usually only overlayed on top of a view of reality (and so doesn’t stay in place when moving around), and 2) a hand-held mobile phone or tablet is used instead of a head-mounted Mixed Reality device. Not using a head-mounted MR device provides a less realistic and less immersive presentation of the virtual content for the user. It is also more distracting in use due to having to physically hold the device. In contrast, a head-mounted MR device is worn on the user’s head, leaving their hands free. This allows more realistic and practical interaction and provides a higher immersion. Interaction with holograms in MR is then usually done using hand gestures, voice commands or eye gaze.

Mixed Reality technology has been in full development for quite some time, but is not really mainstream yet. For head-mounted MR there are therefore only a few products commerically available at the moment: mostly the Microsoft HoloLens 2, the Magic Leap 1 and the Varjo XR-3 (the latter was only just released). There are quite a few other head-mounted devices providing similar features (or a subset), such as the nReal Lite, but these usually provide more limited functionality, which impedes the MR experience. There have been rumours for some time that Apple and Facebook might provide similar products in the near future, which would be a good thing in terms of competition, and thus choice and price, but only some bits of information have been published.

In January of 2020 Facebook’s chief scientist predicted that it will take “5-10 years” for AR technology to be developed enough for mass-adoption [4]. The term AR here was implied to mean “augmented reality glasses”, which still isn’t very specific. But 5-10 years is not a very long time for a technology which currently has only very limited and often experimental adoption. The first version of the Microsoft HoloLens was released in March 2016, its second (substantially improved) iteration in November 2019, with competing headsets in a similar timeframe. In that short period of time of 3 years Mixed Reality went from “interesting concept” to something that is already of value by providing unique capabilities, and is available through multiple commercial products.

Use of Mixed Reality in the SURF community

Even though Mixed Reality headsets are not widely in use yet, partly due to high device cost, quite a number of experiments and research projects into ways of using MR have been realized over the years within the SURF community. In some cases already leading to regular usage. But there is much more potential for applying the technology in education and research. To provide some overview and inspiration here we list a few examples.

At the recent SURF Education Days multiple XR applications where shown, including a few using Mixed Reality technology. The Friesland College uses an app (both on mobile devices and HoloLens) to let students explore human anatomy and teach first aid. And the Centre for Innovation of the University of Leiden developed, together with the Leiden University Medical Center, the AugMedicine application on HoloLens for interactively visualizing CT scans in Mixed Reality, for teaching medical students about transplants cases (Figure 3 above).

The University of Amsterdam’s 4D Research Lab used AR and MR in a series of experiments in teaching and understanding archaeology, history and heritage. At the University of Utrecht a study was performed on visualizing X-ray radiation for radiation safety training using Mixed Reality (Figure 4 above) [1]. Wageningen University is involved in a multi-year research project into the added value of MR for agriculture [2]. Reseachers from VU Amsterdam evaluated HoloLens 1 as a means to provide holographic 3D cues for alleviating freezing of gait in people with Parkinson's disease [5], where the HoloLens tracking capabilities were used to analyze the user's movements.

Some other examples not directly from academia in The Netherlands, but which show different aspects of MR:

Microsoft’s Designing Holograms app is a tutorial on a number of aspects of Mixed Reality development taught within the Mixed Reality environment of the HoloLens 2. Instruction is in the form of a 3D maquette of a room with an animated user and objects to explain MR concepts. This is then intertwined with applying those concepts in the actual room the user is present in. This seemlessly blends a virtual teaching environment with its application in the real world.

Apollo 11 Mission AR is a short Mixed Reality documentary on the Apollo 11 mission to explore the moon. If features nice graphics and animated 3D models, achieved by streaming content to a HoloLens, as the device itself isn’t powerful enough for such high-quality graphics.

Kippy’s Escape is a playful introduction to Mixed Reality in the form of a simple game where the player has to discover how to guide a 3D figure across a series of obstacles. It intuitively teaches some of the available hand interactions of the HoloLens.

A recent commercial example, by the company called Velicus, is a HoloLens application to provide certified CPR and AED training in Mixed Reality. The novel thing here is that the Mixed Reality training apparently matches real-life training well enough to provide a certification.

The wide variety of the selected examples above gives some indication of what is possible with MR. On a more general level, Mixed (and Augmented) Reality in academia are often used to allow new modes of teaching and interaction, to develop new practical skills in a safe and/or more affordable way, to enable new forms of collaborative working or to allow users to leverage image and data processing for augmenting and analyzing real-world scenes.

The particular strengths of (head-mounted) Mixed Reality are:

• MR allows a unique blend of reality with virtual content, in a natural way. The amount of virtual content (holograms) shown can be chosen to suit the particular application. This can range from a few small virtual annotations on a real object (e.g. virtually pointing out the major components of a piece of lab equipment) to a large virtual scene with many components and complex interactions. And as the basis for the scene is reality this never has to be provided explicitly (as is the case with VR).
• Users are still able to perceive and interact with the real world. This is especially relevant for safe navigation and for interacting with other users nearby. For the latter MR allows for realistic and natural interaction. This in contrast with VR, where due the goal of fully immersing the user in a virtual world an unnatural and limited solution like use of avatars is needed to embody users.
• Multi-user applications especially can take advantage of the combination of a regular physical environment, with shared Mixed Reality content. Instead of meeting participants looking at a shared presentation screen they can choose their own view of the shared virtual content, or even make personal adjustments when the MR application provides such a feature. This can be used, for example, to only show specific information that is relevant to a particular user. In essence, the presentation medium becomes a multi-user environment. And with advanced so-called “anchoring” it becomes possible to place virtual content in a fixed physical location, even over long periods of time. Anchored content can then be viewed, edited or interacted with by multiple users, either simultaneously or at different moments.
• Due to the see-through display technology used, a user always still perceives some parts of the real world, minimizing the chance of nausea or other physical effects (something well-known from Virtual Reality). Using a head-mounted MR device is very much like wearing a pair of advanced glasses, versus the full enclosure and immersion of a VR device which can feel more uncomfortable.
• The use of a head-mounted device potentially leads to more natural interaction by hand, and frees the user from having to manage the device physically, which also distracts.

Of course, there are some downsides in applying and using Mixed Reality as well, some of which we will describe in more detail in the upcoming article on the HoloLens 2 mentioned earlier. In short, downsides are:

• The last strength listed above, natural hand interaction, actually still needs more technological development to become really robust and deliver the promise that it holds. It is currently usable, but misdetection of hand movements, or a user interface that is not optimized for such interaction, currently leads to user frustration
• Use of an MR headset in general will take some getting used to, as the methods of interaction are currently not fully intuitive, and certain tasks have a learning curve. MR devices are therefore less suitable for casual usage and/or use without instruction.
• The price of MR headsets is still relatively high, which hinders large-scale deployment (e.g. a class of students) and makes it less attractive than VR in economic terms.
• Current MR devices are also graphically not as capable as (consumer) VR devices, unless one chooses an MR device tethered by cable to a PC, like the Varjo XR-3. But this limits the freedom of movement of the user and requires extra hardware in use.
• MR headsets are currently somewhat bulky and relatively fragile. For mass adoption this will need to improve substantially.

Responsible MR

Just like with VR devices there’s a host of potential privacy and security issues with MR devices like the HoloLens. For starters, as Mixed Reality technology is assumed to augment the physical world, any MR device will need to continuously gather a detailed representation of the user’s environment. This in turn provides data on a user’s location, but with much more precision and detail than, say, a mobile phone can provide. An MR device can measure your head position and orientation, including (in case of the HoloLens 2) your eye gaze. The multiple cameras included in the device record the user’s surroundings, including any other people and objects visible. There’s also microphones in the device used for voice recognition, which, by default for HoloLens 2, use a cloud-based recognition service, meaning voice data actively leaves the device. So an MR device continuously gathers a lot of data that is directly linked to the user and its environment, which coupled with knowledge on the virtual content being shown can become very sensitive and personal. Controlling this data stream and safe-guarding it against misuse is a challenge, not only for the user but also the device vendor and any service providers involved.

Specifically for the HoloLens 2 there’s an interesting issue with logging into the device. This is done either using a pin code or using an iris scan of a user’s eye (it’s the same Windows Hello technology used in regular Windows 10). Using the latter might be a step too far for many users, given the sensitivity of storing and using biometric data. But the pin code option isn’t without issues either, as the code needs to be entered on the virtual keyboard floating in mid-air using finger interaction. Anyone observing a user logging in in this way can probably deduce information on the pin code from the user’s hand movements (this is perhaps a nice security research project).

Apart from the more technical challenges there’s also social and ethical implications if Mixed Reality technology becomes popular and use of devices wide-spread. Most people do not like being filmed by strangers, but that is exactly what an MR headset does all the time when used in a crowd, with no clear insight into what happens with the images recorded. And only the wearer of the MR device knows what virtual content is being displayed, which can also lead to irritation, or MR use even being forbidden in certain spaces. As with any technology, augmenting reality through MR (and AR) can be done for good or for bad. Especially if virtual content can be easily shared then cyber-bullying can take on new forms by anchoring virtual insults, or worse, to specific people or locations, only to be visible to a selected set of MR users.

For the Mixed Reality user itself there can also be negative consequences. If an MR device like HoloLens 2 ends up adding much (economic) value for a certain task or domain, employees might have no choice but to wear such a device in certain jobs. This would give potentially unprecedented insight into employee behaviour, given the detailed data gathered by the device on a user’s movement and focus. And if MR devices coupled with cloud-based cognitive and analytical services really end up providing all kinds of benefits to its users then it can also make someone more dependent on using MR technology in order to function at a higher level.

All in all it’s up to society to define what is responsible use of XR technology, and to place limits (either technological or legal) where needed. In education and research, issues like the ones described above are usually even more critically regarded. But the development and sale of MR and VR technology is done by a small number of commercial entities, some of which are very interested in harvesting and selling the data generated by the devices.

For now, use of MR devices like the HoloLens 2 is still fairly limited. And the Mixed Reality experience they provide needs to improve for frequent daily use. The application of the technology is therefore still relatively experimental and innocent, but if the technology keeps developing at a rapid pace that could easily and quickly change.

Finally, not much is known yet on the long-term effects of using MR (and VR) technology, particularly on the user’s eye sight due to the display technology used. In most End-User License Agreements for XR devices a minimum age is specified for safe usage, usually around 13 years (because devices are constructed based on a minimum distance between a user’s eyes). Effects such as nausea and dizziness are quite common with VR usage, less so with MR. Long-term psychological and cognitive effects can also not be ruled out, but no clear insights and relevant guidelines exist yet. And perhaps at some point users will suffer from a “hologram neck”, as some have a “texting thumb” from mobile phone use.

 For a more extensive overview and discussion of most issues described here, and possible ways of dealing with them, we refer to the report “Nep Echt - Verrijk de wereld met augmented reality” from The Rathenau Institute (Dutch only, see the Further Reading section below).

Conclusions

There is a lot of potential for Mixed Reality in education and research. The various experiments and uses already shown (which is just a small sample) also show the interest in Mixed Reality from the SURF community.

As with any new technology its adoption is initially hampered by high cost, and devices and relevant expertise not being widely available. Head-mounted Mixed Reality is no exception. The HoloLens 2 currently costs around 3,500 US dollar, which is roughly ten times the price of a mainstream VR headset. It’s therefore no surprise that Microsoft currently markets the HoloLens as “Mixed Reality Technology for Business” [3]. But the technology has ample potential in areas outside of business, such as education and research, as we’ve hopefully shown in this article. Virtual Reality is more mature at this point in time, in terms of technology, applications and adoption. But VR is now used regularly at many institutes in the SURF community. It is going to be very interesting to see what Mixed Reality developments and applications will come next in the near future.

As mentioned in the introduction XR is one of the priorities identified in the SURF 2-Year Plan 2021-2022, under the innovation theme “Future computing & data”. We are planning a number of activities in 2021 to discuss with and gather information from the SURF community on shared goals and challenges. For news on these activities please watch the SURF eXploRe project page.

For those wanting to come into contact with other XR users within Dutch academia the XR ERA is a new initiative, started at the Centre for Innovation at Leiden University. It is a community-driven platform to promote the collaboration on XR for education and research in academia. The community was started recently and is slowly growing, but more members are welcome!

Further reading

A further discussion and overview of XR, specifically for its use in academia, is given in a series of blogs on XR as part of the SURF eXploRe project. They are written by the Centre for Innovation at Leiden University and published on the XR ERA website.

The XR4ALL Landscape Report (November 2019) provides a very detailed and general overview of the potential of XR technology and its applications, including backgrounds on relevant business trends in the use of XR. It is available through this page.

Nep Echt - Verrijk de wereld met augmented reality (October 2020) by Dhoya Snijders et al. from the Rathenau Institute investigates the potential impact of AR (and MR) in how we see and treat “reality” when it is augmented with all kinds of virtual content.

References

1. Klunder, 2018, Visualizing X-ray radiation levels with the Microsoft
   HoloLens, MSc Thesis, University of Utrecht, http://dspace.library.uu.nl/handle/1874/363579
2. Balendonck, 2019, Met een AR-bril de kas in, https://www.wur.nl/nl/nieuws/Met- een-AR-bril-de-kas-in.htm
3. https://www.microsoft.com/en-us/hololens
4. https://www.theinformation.com/articles/facebooks-chief-scientist-mass- adoption-of-ar-is-years-away
5. Geerse et al., 2020, Quantifying Spatiotemporal Gait Parameters with HoloLens in Healthy Adults and People with Parkinson's Disease, Sensors, volume 20, June 2020,  https://www.mdpi.com/1424-8220/20/11/3216