Augmented reality (AR), and virtual reality systems (VR) are built on cutting-edge hardware technologies. They transform complex digital information into something that we can understand visually, aurally, and tactilely, rather than using conventional representations on a screen or page. Developers can place virtual objects in the user’s real world using AR applications, or place them in a virtual reality in VR applications. This is a fast-growing segment of the mixed reality (MR0_ ) continuum.
MR Leverages Our High-Bandwidth Sensor Pathways
MR applications offer a rich and immersive experience in entertainment, learning or professional practice. They take advantage of the high information bandwidth of the human sensory system. [2] research shows that human vision can process more than 550 gigabits per seconds, while hearing can process 1.4 million bits per sec (Mbps). A single palm can produce a tactile response of nearly 70,000 pressure points, which is equivalent to a bit rate of more than 190Mbps. The latency is only 21 milliseconds.
These pathways allow complex information from MR apps to reach the brain’s cognitive centres more efficiently. The information is delivered in a familiar context, so users can relate more easily to their environment and needs. According to [3] researchers, AR “books” resulted in higher levels of comprehension and retention for students than traditional learning materials.
Key Technologies Allow MR Immersive Experience
MR systems rely on high-throughput data stream processing, often from multiple sensors, to achieve seamless impressions of virtual objects and complete environments. MR developers are able to access a wider range of subsystem solutions as the core technologies that underpin their applications are compatible with some of the most important application areas in the industry, such as machine-learning (ML), computer vision, robotics positioning, and automotive advanced driver assist systems (ADAS).
Rapid ML model development and advances in ML computer vision meet MR’s crucial requirement to identify objects in the real world. The ability to accurately determine the distance between objects and the user is possible using distance measurement technologies such as cameras, ultrasounds, lasers, Lidars, radar, and other technology that has been optimized for use in ADAS building automation and collision detection. Lastly, MR systems can create a “map” of real-world objects using technologies such as simultaneous localization (SLAM), which allows them to place them in the right position, perspective, orientation, and relative to the user.
All that technology is only the beginning of what lies ahead in MR systems. While VR systems are focused on the maintenance of the virtual world for users, they also need the map to ensure that users can safely move within their virtual environment. The MR system renders visual objects in AR applications at a position and orientation that is natural to the user. Advanced versions of this ability allow virtual objects to obscure real-world objects behind them relative to the observer, and vice versa.
Different MR Classes, Different Hardware Requirements
MR applications can be used with a variety of hardware capabilities, which are limited by the processing and sensor complements of the hardware. These applications use high-resolution cameras, precision measurement capabilities using Lidar and are limited to smartphones and other mobile devices. These devices are quite capable of providing AR-based mobile apps such as those that allow users to read virtual labels for buildings and translate signs from their displays by simply pointing their device’s camera at a sign or building.
Hardware design requirements for professional MR applications and consumer VR have increased in complexity due to the addition of a head-mounted monitor (HMD). This has led to more complex packaging and performance requirements. While the basic enabling technologies are the same for all these systems, the processing requirements change dramatically when you move from consumer VR entertainment headsets, to MR headsets designed for professional applications in medicine and construction, as well as industrial operations.
Hardware designers often need to include a high-performance, specialized processing pipeline in order to build the hardware foundation for these professional systems. Although conceptually similar, MR pipelines are often significantly different than VPU pipelines. This is due to the higher volume, velocity and variety of data required by MR applications. Microsoft, for instance, addresses the throughput problem in its HoloLenseMR system by using a custom-built holographic processing device that was specifically designed to handle this complicated processing workload.
Acceptance of MR by Humans Limits
Applications for HMD MR systems have more fundamental challenges than just performance requirements. The developers must provide a hardware solution that can deliver the required sensing and processing capabilities, while still keeping within the constraints of size, weight and power (SWaP).
Because the human anatomy is not built to support a heavy HMD, SWaP may prove to be more problematic than software or hardware capabilities. Researchers in the aviation human factors field and more recently in MR HMD systems, have found evidence of user fatigue [4], and an increase in stress on the musculoskeletal systems of the neck and head [5], due to HMD weight- and balance issues.
Beyond musculoskeletal limitations, the nature of the human visual system can make working with HMD-based MR applications difficult for some users due to the vergence-accommodation conflict (VAC) experienced in viewing three-dimensional (3D) images in HMDs. The real world is where our eyes meet and then the lenses of both eyes adjust to allow us to see clearly. HMDs, on the other hand, use stereoscopic effects to create 3D effects. This means that convergence on objects in the background or foreground of an image and accommodation to those objects do not occur at the same time. This can lead to fatigue, headaches, nausea, and even hypertension [6]. Health experts also warn against children using HMDs for extended periods of time due to long-term concerns regarding the development of their visuomotor system.
Mixed reality (MR) systems are revolutionizing the way we interact with digital information, taking advantage of the high information bandwidth of the human sensory systems to create a highly immersive experience. The key technologies that make MR possible include machine-learning (ML) computer vision, robotics positioning methods, and automotive advanced driver assistance systems (ADAS), among others. These technologies help developers achieve a seamless impression of virtual objects or complete environments in the user’s real-world surroundings. However, the hardware requirements of MR systems can be a challenge, as the human anatomy was not designed to wear a heavy head-mounted display (HMD) for extended periods. In addition to musculoskeletal limitations, some users may find HMD-based MR applications difficult due to the vergence-accommodation conflict (VAC) experienced in viewing 3D images in HMDs.
