Java in virtual reality. Is it true that VR/AR is the future, and is there a place for Java in it?

In the previous articles of this series, we have already examined almost all more or less trendy technologies of the modern IT market, such as artificial intelligence , blockchain , cloud services , big data , etc., finding out how the Java programming language is used in these niches and to what extent he plays an important role. Today we’ll talk about another trendy and very interesting technological segment - namely, VR and AR, that is, virtual and augmented reality. “I think that a significant part of the population of developed countries will very soon interact with AR solutions on an ongoing basis every day. Very soon, AR will become a part of our daily lives.” Tim Cook, CEO of Apple

What are VR and AR?

First, let's figure out what these technologies are, how they are used and how they differ from each other. Let's start with VR, that is, virtual reality. Virtual reality applications run on special devices that completely capture the user's vision to create the impression that they are somewhere else. All popular VR headsets, such as the HTC Vive or Oculus Rift, completely block the user's vision when worn. When such a device is turned on, the LCD or OLED panels inside it are refracted by special lenses to completely fill the user’s field of view with the transmitted content, be it a game, 360-degree video, or simply the interface of a program. Most VR headsets, including the aforementioned Oculus Rift, as well as Vive, PlayStation VR, Windows Mixed Reality and others, use six degrees of freedom (6DOF) motion tracking technology, which is achieved through external sensors or cameras. This allows devices to detect not only the direction in which you are looking, but also the movement the user makes towards that direction. This, along with 6DOF motion detectors, allows users to move around in virtual space, making such applications much more immersive. As for AR, that is, augmented reality, its difference from VR is that AR does not strive to completely absorb your vision, but adds certain elements to it. AR devices such as Microsoft HoloLens and various types of smart glasses allow the user to see everything around them by projecting images onto where they are looking. This concept also includes numerous smartphone apps and games such as Pokemon Go, Temple Treasure Hunt, ARrrrrgh, Ingress and many more. They use the camera of a mobile device to monitor the environment and supplement it with various types of information placed on the screen. AR displays can overlay some very simple data on top of reality, like a watch dial, for example, but they can also implement such complex objects as holograms of monsters in the middle of a room. For example, the game Pokemon Go projects Pokemon characters onto the screen of your device, where the camera lens is pointed. While various types of augmented reality glasses allow users to place interface elements and application icons on top of the image of reality they observe. “I believe augmented reality will be the biggest technological revolution of our time.” Tim Sweeney, American programmer and video game developer, founder of Epic Games

VR and AR are not the same thing

This, in fact, is the main disadvantage of augmented reality compared to virtual reality. If VR completely replaces the image in your view, then AR only complements it, often occupying only a very small area of ​​the user’s vision. But this is also the main advantage of augmented reality, because this method gives developers almost unlimited possibilities. For example, mobile AR applications that can recognize objects in a camera image and transmit information about them to the user have been around for several years. Special AR glasses, such as HoloLens, allow users to place floating windows around themselves that can be viewed on the device's display. Not to mention the many games that use augmented reality as a core gameplay element (Pokemon Go, RoboRaid, Fragments) or as an additional feature. Thus, virtual reality and augmented reality are two very different technologies that are used for different purposes, although AR and VR are usually referred to as two types of the same approach. If the purpose of VR devices and applications is to replace reality and virtually transport the user to another place, then AR works with reality, supplementing what is already before our eyes with new information. Both of these technologies are still in their early days, but are considered extremely promising and have the potential to revolutionize the way people use computers in the near future. “I think that in the near future, many stories will be told within the virtual space.” Joe Russo, American film director

AR/VR and Java

Although the AR/VR niche uses a number of programming languages, including C#, C/C++, JavaScript, Swift and even Python, Java is one of the most in demand. The reasons, in fact, are the same as in the case of other technologies: Java code allows applications to run quickly, object-orientation makes it easier to create individual application components, and platform independence makes AR/VR products in Java more universal. In the case of AR, that is, augmented reality, Java will be one of the main programming languages, because the vast majority of AR applications are created for mobile devices. Accordingly, Java is the main option for AR applications created for devices based on the Android mobile operating system, while in the case of Apple products, Objective-C and Swift languages ​​are used. Today, AR applications support many different devices, including helmets, glasses, handheld devices and even contact lenses. In many ways, it is Java that allows you to create AR solutions that will work on different, often very different, platforms. Today, Java-based AR applications are actively being implemented in clinics, banks, restaurants, museums, etc. “AR is already revolutionizing the way we approach commerce and shopping, creating a bridge between the digital and physical worlds.” Muchaneta Kapfunde, founder of FashNerd.com

Types of AR Applications

Now let's take a quick look at the main sets of tools (SDK) and frameworks that make sense to use for developing AR applications. But first, let’s look at the two main types of AR applications: marker and geolocation applications.

• Marker AR applications.

  Marker AR applications are based on image recognition technology. They use black and white markers as triggers that activate the display of AR content to the user. Thus, in order to see an element of augmented reality in front of him, the user of such an application must point the camera at the corresponding marker in his environment. Once the device recognizes the marker, the application overlays digital data on it and shows the user an element of augmented reality. These kinds of applications, in particular, are widely used for advertising purposes by shops, restaurants and other businesses where this functionality is appropriate.

• Geolocation applications.

  The second main type of AR app, location-based, operates without markers, relying instead on user location information provided by GPS, accelerometer, or digital compass, and overlaying AR objects on top of actual physical locations. And again we use Pokemon Go as an example - this is probably the most famous geolocation AR application.

  “The key task for the development of AR and VR technologies is to create a single open platform where developers can implement all their ideas.”

  Mark Zuckerberg, creator of the social network Facebook

The best SDKs for developing AR applications

• Google ARCore

  ARCore is Google's key platform for AR app development, supporting devices based on the Android mobile OS and working with Java/OpenGL, Unity and Unreal libraries.

  Among the key capabilities of ARCore: motion tracking (ARCore can determine the position and point of direction of the device using the built-in camera), environmental modeling (the platform is able to determine horizontal surfaces, which allows, for example, to place virtual objects on tables or on the floor), lighting estimation ( determines the level of illumination, if necessary, highlighting virtual objects so that they look natural in the environment), and others.

• ARToolKit

  ARToolKit is an open-source set of tools for developing AR applications.

  Despite the fact that it is an open and free kit, ARToolKit includes a very rich set of functions, including support for Unity3D and OpenSceneGraph, the ability to create real-time AR applications, integration with smart glasses, support for different types of cameras, automatic camera calibration and etc.

  ARToolKit supports a range of platforms, including Android, iOS, Linux, Windows and Mac OS.

• Vuforia

  Vuforia is perhaps the most popular and well-known platform for developing augmented reality applications.

  Among the key functions of Vuforia: recognition of various kinds of objects and images; support for recognizing basic texts, creating custom markers, creating 3D maps of various locations, support for cloud services and local data storage, etc.

  Vuforia supports platforms such as Android, Universal Windows Platform, Unity and iOS.

• Apple ARKit

  Well, Apple also has its own set of tools for developing AR applications, which was introduced along with iOS11 and is called ARKit. Naturally, it only supports iOS devices and includes all the basic functionality for creating augmented reality applications.

  “Our current goal is to make VR experiences more fun and immersive, and allow people to experience these experiences together.”

  John Hanke, CEO of Niantic

The best platforms and frameworks for developing VR applications

And finally, let’s go through a number of the most popular open platforms and frameworks for developing virtual reality applications.

• GuriVR

  GuriVR is an open source VR editor that allows you to convert text descriptions of 3D scenes into real 3D models for VR applications. Supports text, 3D models, images, audio recordings, voiceovers, 3D charts and many other functions.

• OpenSpace3D

  OpenSpace3D is a free and open platform for developing VR applications and games based on VR technologies. Supports a large number of different platforms and devices, including Leap Motion, HTC Vive/Oculus, Google Cardboard and others. It also includes functionality for creating AR applications.

• OSVR: Open Source Virtual Reality Platform

  OSVR is another universal platform for VR and AR solutions. It allows you to detect, configure and manage hundreds of VR/AR devices, as well as various kinds of peripheral devices. OSVR supports multiple game engines and operating systems, and provides services such as asynchronous time warping and direct mode to support low-latency rendering.

• Primrose VR

  Primrose VR is an open browser framework for creating simple VR experiences, such as virtual product demonstrations, virtual meetings, or virtual activities. Primrose VR is compatible with all modern browsers and runs on WebGL.

• IdeaSpace VR (CMS)

  IdeaSpaceVR is not a framework or platform, but an open content management system (CMS) for VR content. Powered by PHP/MySQL, IdeaSpaceVR is a locally installed modular system with a built-in content player for virtual reality applications.

What else to read:

Shvets/reaper/backend developer. Why is programming the best career plan B? How to maintain your career without being branded a fool? Cybersecurity rules for remote work See you in court! What should a programmer know about work contracts and NDAs? COVID-19 and robots. How coronavirus will cause a boom in investment in automation The end of the era of open space. How will offices change after the pandemic?