Fused Reality

Technology has been munificent in the arenas of computational speeds, Internet connectivity and instantaneous communication between electronic devices. Juxtaposed with equally stunning advances in artificial intelligence, the ‘reality’ of the world around us has become vulnerable to exotic metamorphosis. The two technologies that have been at the forefront of this marvel are Virtual Reality and Augmented Reality. In both the terms, the suffix ‘Reality’ really serves only to underscore the fact that both are designed to visually and aurally deceive one into believing what one is seeing is the real world.

While Virtual Reality blinkers out the real world around the viewer and aims to convince that he is now surrounded by a new ‘reality’, Augmented Reality does not dismiss the real world, but uses potent computational power to supplement and enhance the viewer’s experience of the real world through virtual information superimposed on the real world. Thus while Virtual Reality needs one to blank out visual and aural connections with the real world using a head-set covering eyes and ears, Augmented Reality permits full view of the world, but adds on to it through headsup display (HUD). Although Google Glass twinkled only for a few months before withdrawing from public visibility and Microsoft Hololens is yet to make a full impact as an Augmented Reality artifact, there is no doubt that Augmented Reality is a game changer. Aviation being possibly the most technology demanding industrial complex, is displaying a voracious appetite for whatever this technology produces or promises.

Augmented Reality in Aviation

HUDs for fighter cockpits go back to the 1970s although their use in civilian aircraft is more recent and not very popular yet. These display targets or instrument information projected on the front windscreen and focused at infinity, permitted the pilot to behold the real world outside while viewing, assimilating and analysing other information relevant to the flight profile or mission. As far back as 2000, the US Joint Strike Fighter (JSF) team led by Lockheed Martin, validated technology intended to allow pilots to conduct complex training simulations while flying their own aircraft through a technique called Embedded Training. Lockheed Martin claimed that the technology offered more realistic training than ground-based simulations and eliminated the cost of supplying extra aircraft for training missions. The stage was set for further innovation.

In the US, Small Business Innovation Research (SBIR) programme encourages domestic small businesses to engage in Federal Research/Research and Development (R/R&D) that has the potential for commercialisation. Through a competitive awards-based programme, SBIR enables small businesses to explore their technological potential and provides the incentive to profit from its commercialisation. Under this programme, California-based Systems Technology Inc (STI) began development in 2003 for US Naval Air Systems Command and the US Army of a system that combined real world video with interactive computer generated environments to create a highly immersive training experience for practising complex tasks such as landing, flying in formation with other aircraft and aerial refuelling. While flying, the pilot wears a special helmet with an optical system that combines the real worldview from a camera with computer-generated graphics of an airfield or another aircraft. Possibly parodying the ‘Mixed Reality’ that Hololens claims to create, the system’s inventor at STI, Ed Bachelder, christened it ‘Fused Reality’.

Fused Reality

In 2012, the Fused Reality concept first became remarkable when the US National Aeronautics and Space Administration (NASA) and STI conducted three test flights on a specially modified Learjet in-flight simulator at Mojave. The concept was hailed as a revolutionary mixed reality visual system that amalgamated augmented reality, virtualised training, gesture technologies and haptics. Sometimes referred to as kinesthetic communication, this is technology which recreates the sense of touch through the application of force, vibrations or motions to the user. It offered the promise of a technology that could be used equally effectively in flight and in ground-based simulation.

In the airborne mode, it would allow pilots to perform critical and tricky flight manoeuvres such as air-to-air refuelling, close formation flying, precision offset landing and so on without incurring the inherent risks involved as they can perform operational tasks in a safe environment without the involvement of other aircraft and/or the physical environment. However, the research pilots concluded that, although the Fused Reality system showed great promise for flight-test and training, improvements were needed to make the concept meaningful. Armstrong’s Center Innovation Fund paid for the next stage of development to eliminate minor problems and to improve the system to an extent that, in the next phase of flight-testing in 2014-15, the team successfully demonstrated a much-improved Fused Reality system onboard a Gippsland GA-8 Airvan research aircraft.

A series of nine flights executed a series of tasks generated by the Fused Reality system. These included landing on a simulated runway at altitude, formation flight, and aerial refuelling drogue tracking. According to Scott Howe, one of the test pilots, “I think what you gain here is the benefit of taking the simulator into the air where you are exposed to the actual flying environment, but with the ability to superimpose a realistic simulation on top of it.” All the test pilots noted that Fused Reality did not interfere in any way with the actual feel of the handling qualities of the aircraft.

How Real is Simulated?

Computer-based flight simulation is almost as old as computers and the current generation of flight simulators have 3D or ‘full’ motion built into them to give a near feel of flight, but that is what it remains — ‘near feel’. No simulator can actually substitute for the experience of real flight due to the sensory and physiological cues that are implicit in real flight. The leading edge of simulation technology is still far away from modelling a flight simulator that reproduces an aircraft’s aerodynamic and physical behaviour as experienced by the pilot.

On the flipside, training for critical tasks in actual flight is fraught with risk. Fused Reality somewhat solves this conundrum by blending simulation and flying, termed as ‘overlaid simulation’, using a commercial head-mounted display (HMD) to project a simulated aircraft or runway onto the outside world to allow the pilot to safely practise complex or risky tasks such as steep approaches, restricted area operations, bad weather or low visibility operations and so on. The value of such in-flight simulation training is self-evident when seen as a foundation for immense confidence in a pilot in his proficiency to safely deliver a payload of precious lives or goods because of the ‘real’ experience of flying using Fused Reality.

In terms of fighter combat training, an artificial intelligence computer developed by a doctoral graduate from the University of Cincinnati has demonstrated that it can beat a professional fighter pilot with decades of experience. In a series of flight combat simulations, it successfully evaded retired US Air Force Colonel Gene Lee and shot him down every time. With such an application superimposed onto the real world in Fused Reality, the training value would be priceless. Fused Reality has also been considered for use on helicopter training and a prototype simulator called Prototype Aircrew Virtual Environment Training (PAVET) System was introduced in 2006 with the objective of training US Navy MH-60 aircrew in all operational aspects including aerial gunnery, search and rescue and vertical replenishment.

Conclusion

In an interesting and somewhat related development, Airbus is experimenting with 3D technologies to help helicopter pilots fly in bad weather where visual environment has been degraded significantly. Considering the fact that rotary-wing flying often takes place in close proximity of tall obstacles such as skyscrapers, power lines and terrain and can be quite challenging in bad weather, a headset that can see through the weather using Augmented Reality would enhance safety and operational efficiency. A system using reflected pulses from a light detection and ranging (LiDAR) helps to create a 3D map of the area, which is then projected onto the pilot’s head-set to help him to effectively look through clouds and fog. The potential for adding to helicopter safety is self-evident.

Another related concept is Live, Virtual and Constructive (LVC) which can be described as a concept that entails linking live aircraft with manned simulators in the ‘virtual’ world and computer-generated ‘constructive’ forces; it is revolutionising the way aviation training could be conducted in the future. Last year, in a demonstration, Rockwell Collins validated the use of several ground-based simulators connected to an aircraft in the air thus showing the capability to train users in multiple locations using simulated and live missions with the ultimate aim of cutting down costs.

Coming back to Fused Reality, NASA has an established scale to check the Technology Readiness Level of any programme it is associated with. The scale ranks ideas from basic principles (TRL-1) to flight-proven in mission operations (TRL-9). Fused Reality is currently at TRL-7 status and can be considered to be almost ready for commercialisation. As a technology platform, it enables the next-generation of simulation beyond Virtual and Augmented Reality capable of layering what is perceived of the real world with virtual elements, in real-time, seamlessly and simultaneously. As the degree of virtual overlays can be controlled at will, the potential for Fused Reality as a training aid is noteworthy. Although currently it excites the military aviation regime, its potential for civil aviation, especially in terms of training in upset recovery, stall management, bad weather approaches and landings, flying in close proximity of difficult terrain, is equally enormous.