E-PILOTS: Welcome to the first stage of the aviation of the future

The Covid-19 situation is hitting hard our lives, and E-PILOTS is not an exception. However, this project keeps rolling and, although all the difficulties and challenges it has been to face, the results after two years of its kick-off are very promising. 

E-PILOTS is aimed to evaluate the potential of cognitive computing flight deck supporting tools, to preserve the situational awareness of the pilots while improving their performance in stressful flying scenarios. Scenarios where pilot cognitive tasks must deal not only with a procedure but also interleaving with different interruptions such as the controller instructions, the Electronic Centralized Aircraft Monitor (ECAM) warnings and other emergent issues. 

With these objectives in mind, the E-PILOTS consortium has designed a use case on a go-around decision support system, based on the probability of a hard landing event (when the acceleration at the moment of the touchdown is very high and it could damage the aeroplane), to validate the functional and non-functional requirements of flight deck intelligent supporting tools. For this purpose, two machine learning systems have been developed: a machine learning system to predict hard landings, and a machine learning system to continuously monitor in real-time the cognitive state of the pilot(s) and decide when is the best temporal window to provide elaborated information through extra perceptual variables to improve their situational awareness. 

Regarding the first machine learning system, E-PILOTS has identified several variables related to physical magnitudes and actuators states, that can predict a hard landing during the approach phase. “This machine learning system can predict these events with a 78% of accuracy. This is a very promising result, considering that there are not so many systems able to do this. It is necessary, of course, to get higher accuracy, but we are already doing significant improvements”, Dr Dèbora Gil, Computer Vision Center (CVC) researcher, explained. 

For the second machine learning system, to monitor the cognitive state of the pilot(s), sensors such as electrocardiogram and electroencephalogram play a key role. “Taking measurements from electrocardiogram and electroencephalogram, we can differentiate when the pilot is in a baseline state –relaxed- from when they are doing or having a heavy workload and, depending on that, choose the right moment to communicate the situation to the pilot”, Dr Aura Hernández, CVC researcher, described. 

Dr. Hernández and Dr. Gil agreed that this machine learning systems is the crown jewel of E-PILOTS:  

Better Than Expected Results 

So far, the results are exceeding expectations. “The ability to detect cognitive states from sensors could be helpful not only for pilots but also for any specialized highly demanding jobs, like a clinician in the intervention room. Also, it could be useful for developing decision supporting systems for people with special needs, like old people that starts to have mental cognitive problems”, argued Dr Gil. 

Dr. Hernández, from her part, wanted to highlight the unexpected results obtained: 

The Covid-19 Impact 

The main problem E-PILOTS has to face since its beginning is being the pandemic situation. “Until now, we could only conduct experiments in simulators with a reduced amount of volunteers, due to the difficulty to travel to the UK where the consortium planned the use of Rolls-Royce Future Systems Simulator with the support of Cranfield University partner”, Dr Hernández exposed. But this is not the only complication the Covid-19 is creating. “We could not deploy the systems in more realistic situations or test it with more pilots. Also, you need to place sensors in the chest to monitor the cardiac activity and a helmet with sensors in the head to get the signals related to the cognitive state, and during the first months of the pandemic, this was considered very risky due to close interaction with volunteers”, delved into Dr Dèbora Gil. 

To try to solve these problems and push forward the project, the E-PILOTS consortium is actively looking for ways to achieve more data in simulator experiments and be able to move forward till the final objective of this project, test the machine learning systems in a realistic Single Pilot Operation (SPO) flight simulation framework. 

The Pilots’ Point Of View 

One of the main E-PILOTS’ strengths is the multidisciplinary nature of the consortium: experts in machine learning, aeronautical engineering, human factors, socio-technical simulation, research centres, universities, big private companies and, of course, professional pilots. 

The main objective is, therefore, to adapt all the systems to the pilots’ particular needs. For that reason, it is essential to know their point of view. 

Jordi Manzano, the pilot of Air Europa and teacher at the Universitat Autonóma de Barcelona, joined E-PILOTS because “it offers me the opportunity to learn and to obtain information about new tools and procedures that otherwise I would never know about”. 

Mr. Manzano explained the reason why he decided to join E-PILOTS and his experience until now: 

For Mr Manzano, E-PILOTS contributes in many ways to real pilots, but he underlined that “the best part of being involved in E-PILOTS is to have the opportunity to learn and to propose different procedures for the aviation of the future”. 

He has no doubt about that E-PILOTS tools will be useful in real fly situations in the future: 

For his part, Miquel Gallego, pilot and flight instructor of the Barcelona Flight School, pointed out that “E-PILOTS gives me the chance to spend many hours in a flight simulator, facing on hard landing events, that for sure it is very positive for my training” 

Also, he feels that his contribution is very useful for the development of the project: 

Both of them are convinced that E-PILOTS could help pilots in real flight situations and expressed their willingness to use it beyond the simulator.