The past 30 years have set the stage for technological and cultural change at an unprecedented scale. The rise of Personal Computing, Internet as well as Smartphones have completely redefined the way we interact with our world as well as each other – today, over 99.9% of the world’s information is stored digitally and half of the world’s’ population is already part of the internet, a figure which is expected to near 100% by 2020. Every day we’ve been growing more reliant on modern technology as we collectively craft a separate digital universe of information byte by byte – a universe we use to manage everything in our lives from money to our closest social circles.
The digital world, however, holds little resemblance to our physical world – after all, the former is still governed by screens and 2D imagery. Companies have used their mastery of design to make interactions with the digital realm feel more natural through features like slick user interfaces and touchscreen interaction, but there’s only so much that can be done in plain screens. Currently, we interface with the digital world at a distance, looking through the windows of the devices we own, but never daring to step any further. The digital world is a world of information, while our physical reality is one of experiences – and because of that fact those two realities have always remained separate, only connecting to each other through tangents.
But that’s bound to change. The recent rise of Virtual Reality has brought new ways of experiencing information into the light, inspiring a new wave of interaction design and experiential software and creating a true sense of presence in digital worlds. Now, a new form of computing stands on the horizon, creating a plane of reality that intersects the physical and virtual world that’s both exceptional and familiar. It is the result of the convergence of several advanced fields of technology that when put together will spur a new age of contextual computing, where digital information seamlessly blends into our physical reality as active parts of our environment – a world where computers can understand our surroundings, where technology feels unbound from the human experience.
This is Mixed Reality – it’s real, it’s here, and it’s the biggest technological race since the rise of personal computing.
For starters : the Virtual Reality, Augmented and Mixed Reality distinction
While Virtual, Augmented and Mixed Reality technologies are all similar in many aspects, they have some fundamental differences. Virtual Reality headsets like the Oculus Rift or the HTC Vive completely replace your current reality with a new one, putting you in a 3D generated world with little concern for your immediate surroundings. It invites you into its world, and it’s isolationist by design, which is one of its main strengths (fully immersive) as well as weaknesses (disconnect from your real world).
Augmented reality, however, overlays digital information on top of your real world, the most notable example being Pokemon Go. And while it can make for some interesting applications, it is nothing but a rough digital overlay – it doesn’t truly understand your space, which stops you from making applications that truly use your world as a canvas.
Mixed Reality, however, is a completely different beast that combine several types of technology into one device – differently from AR and VR, MR devices are constantly scanning your room and gathering a 3D understanding of your surroundings, using that information to seamlessly place digital information in your space, which can be viewed through transparent displays in the form of goggles (while you interact with objects using your hands). Unlike Virtual Reality, it doesn’t invite you to a completely different world – rather, it invites the digital world into yours.
The players today
The race for Mixed Reality is already happening, albeit mostly under shadows. All major tech companies like Google, Apple, Microsoft and even Snapchat have been advocating resources to Mixed Reality hardware, with new players like the Meta and Magic Leap also entering the game, each with their own distinct vision of the future. And while most of the hardware projects conducted by these companies are currently secret, Microsoft’s HoloLens as well as Meta’s Headset have development kits of their own in the market, offering a glimpse of the future that’s about to arrive.
The HoloLens has a sleek design and looks nothing like an unfinished prototype. In fact, it’s a completely functional untethered holographic computer, requiring no wires or any external power to run whatsoever. It possesses 3 distinct types of sensors and 5 environment understanding cameras working together to allow the device to understand 3D space and place holograms, and the resulting mixed reality is finally brought to life by the two high-pixel-density lenses that sit in front of the user’s eyes.
Through a simple “gaze and click” interface the user can do familiar tasks like browse the web as well as perform Skype calls in virtual screens you can scatter around your environment to any location, in any room, in any size – you can also play around with a range of novel Mixed Reality apps ranging from creative tools to Games that showcase some of the unique powers of the medium. The HoloLens also adds a sense of permanence to digital objects that’s rarely seen – since it creates and stores a 3D mesh of all the physical environments you use it in, it remembers where you set up your holograms – so if you drop a couple of 3D models in the living room, some others in your bathroom and then leave your home for 3 days, they will all be there exactly where you left them when you return. The only drawback to the HoloLens is its limited field of view – but it’s a temporary shortcoming that will surely be addressed in the consumer version – the amazing part is that all of its parts come together.
Another MR device that’s already out for developers is the Meta 2, currently being developed by a startup based in Silicon Valley. While this MR device is cheaper than the HoloLens (and possesses 90-degree field of view, compared to the HoloLen’s 50-degree FoV), but currently requires to be connected to a computer at all times, limiting your experiences to your immediate workspace. To handle interaction, Meta has opted for something less familiar in the realm of software but perhaps far more intuitive, focusing on accurate hand tracking that allows you to physically manipulate digital objects in your environment as you would in real life. The Meta development is still set to release this year and Meron Gribetz, its founder, says their mixed-reality HMDs (head mounted displays) will be replacing all monitors in his company within the next few months.
What about the other tech giants? There’s very little we know about their unnanounced projects other than the fact that they’re all directing resources toward these new forms of immersive computing – with all cases, interests in Virtual Reality and Mixed Reality seem to go hand-in-hand. Google has been spending considerable amounts of money in their Virtual Reality department, while also reportedly developing a Mixed Reality device of their own. Intel has announced Project Alloy, a VR/MR headset hybrid set to release next year. Apple has several patents for Mixed and Virtual Reality headsets, both which should see the light of day sometime in 2018. Additionally, Facebook and Samsung have created entirely new departments with hundreds of employees dedicated to VR and related technologies, spending billions of dollars in the process. Even SnapChat seems to be tiptoeing their way into the MR as well, spending considerable amounts of money in computer vision (which currently powers their famous SnapChat filters) and now releasing a low-risk pair of wearable glasses with 180 cameras that are only a few steps away from turning into MR displays. It’s no wonder all of these companies are paying attention – after all, these are technologies that could eventually replace all of our laptops, smartphones and computers, causing disruption on a mass scale.
But out of all these ventures, the most impressive one has to be Magic Leap, which may be on their way to becoming one of the biggest tech companies in the world. It has been backed by giants in the industry like Google and Qualcomm and is currently evaluated at $4.1 billion. Magic Leap’s goal is simple: they want to create a device that interfaces perfectly with the human brain, creating the transparent display to rule them all. They achieve that by going into a completely different direction than its competitors, aiming to reproduce exactly what happens in nature: projecting light straight into the user’s retina through a digital lightfield that’s indistinguishable from nature.
This means two things: firstly, that pixels are not visible. Because Magic Leap’s lightfield is just light entering your eye like anything else in the world, it effectively spreads it evenly across your retina, allowing you to naturally focus on digital objects as you would in real life, pixel free. The resulting effect is so convincing that it fools the brain completely – for all intents and purposes, everything you see in Magic Leap’s displays is neurologically real. And if you take into account the rate at which these displays will progress in the future, it becomes hard to imagine a future 20 years down the road where digital objects are any less crisp from the ones in the real world.
The Dawn of Personal Universes
If Mixed Reality becomes popularized, everything is bound to change – the eventual replacement of today’s computers and smartphones devices with technology of this type means that all digital interfaces that we we know today will have to be rethought of and rebuilt from the ground up. Overall, MR is expected to cause disruption on a vast scale, completely revolutionizing enterprise applications, education, social media and the $2 trillion dollar entretainment market – while also opening a door to a new way of being alive.
Mixed Reality isn’t just a new type of technology – it’s an entirely new plane of existence. It’ll spur new ways of creating and consuming content, sure, but when you begin to predict how MR will converge with other emerging technologies of the decade like 5G and the Internet of Things, the possibilities become even more expansive. Users will be able to have all the information they want, when they need it, presented perfectly on top of their real world instead of diverting you away from it. I tend to think that technology is only truly mature when it’s invisible to the user – in this case, MR is what technologically-mature computers look like, blending seamlessly into the human experience instead of detracting you from it.
Imagine educating yourself not in classrooms, but simply by exploring the world, where you can learn about surrounding plants and animals by simply looking at them. Perhaps you would like to travel into the 1920’s for a day, replacing all of the surrounding cars, people and buildings with digital vintage counterparts. Or maybe you would like to have the power to see through the walls of your office, so you know exactly where your co-workers are. What if you want to replace all billboard advertisements in your world with pieces of modern art, or if you wish that one bar resembles the Mos Eisley Cantina from Star Wars, filled with digital alien creatures amidst the humans in the space? Everything is possible, and ultimately, Mixed Reality technology allows users to have agency over their own realities in a way that we’ve never encountered before, essentially giving you the power to choose to live anywhere within a ‘vanilla’ and ‘fully virtual’ reality spectrum. Mixed Reality represents a shift from a world of Personal Devices to a world of Personal Universes – an industry geared around giving people superpowers and the ability to do with the world what they see fit.
As with similar times in history, however, whenever there’s a new disruptive technology of this scale on the horizon, there is also fear. I’m sure some of the readers are terrified by the prospects of people “losing” themselves in a large scale “fabricated” reality, which is understandable. I won’t get too much into that topic, mostly because it’s a completely subjective and philosophical discussion, but one could argue that the reality we already live in is already being “fabricated” by our brains. While the digital content in Mixed Reality devices may be ‘artificial’, the experiences and memories are still genuine, and in the grand scheme of things I believe the good of this technology will outweight the bad in almost all instances, pushing us to be more empathetic, curious and free (also supercharging education in a way that we’ve never seen before – how different is it to learn from pictures in a book than it is to see and interact with the things you’re learning about?)
There is one serious problem that needs to be talked about with MR, however, which is a direct continuation of something we face today: mass surveillance. Since these devices function by acquiring a 3D understanding of our environment, we would be opening our most personal spaces to corporations, which eventually would lead into an unprecedented level of acquired knowledge about our lives. These devices could also function as a gateway into our minds – after all, if you can tap into the reality of any single person, you can cleverly manipulate thought and behavior without being noticed, placing seeds of thought almost in an Inception-like manner. All of the companies involved in MR understand that trust and privacy are huge concerns and are building products that address these fears, but we still need to be aware of the possible risks this technology brings so the privacy discussion is always ongoing. “If a smartphone is a surveillance device we voluntarily carry in our pocket, then MR will be a total surveillance state we voluntarily enter.”, said Peter Wang, writer at Wired.
Virtual and Mixed Reality bring about new forms of interacting with digital information that will revolutionize not only all the things we do, but also the way we perceive the world. It will open the doors to a new, unexplored reality filled with wondrous possibilities, where we have the power to live our lives in any way we wish, away from the limitations of traditional computers and screens. It’s an unknown world – so while it might sound scary for some, the possibilities are too great to pass. MR is, simply put, the science of magic, and I’m glad to be part of it as a developer and designer. If you haven’t gotten acquainted with VR and MR technologies yet, find a place that’s demoing these devices and experience as many different applications as you can – these experiences are not something you describe as much as something you live through, and this is just the beginning of a much larger trillion-dollar industry. As Magic Leap founder Rony Abovitz puts it, “Ours is a journey of inner space. We are building the internet of presence and experience.” I for one look forward to replace the hours I spend staring at screens out into the real – or unreal – world.