E-Pilots: combining machine learning and cognitive monitoring to improve flight safety

Credit: Freepik

The European Project E-Pilots aims to devise the guidelines in aviation security of the next decade implementing machine learning based solutions within the flight cabin. As a main goal, the project aims to develop intelligent tools for the assistance in decision making, both in air (pilots) as on land (control centres).

UAB Researchers (Universitat Autònoma de Barcelona), along with members of the Computer Vision Center, Cranfield University (UK) and Aslogic (Barcelona), are actively working on the European project E-Pilots, with a clear aim of designing the future of assistance technology for pilots.

The analysis of the interdependence of different tasks which coexist and which must be undertaken simultaneously when flying will allow the design and development of new technologies to significantly reduce work load both for pilots and air controllers. What’s more, these advances will not only contribute in maximizing security standards; the improvement of flight processes and manoeuvres will inevitably optimise fuel use and help minimize environmental impact.

The European Agency Eurocontrol, in its long-term 2008 analysis of the aviation commercial sector (EUROCONTROL Long-Term Forecast. Flight movements 2008-2030), estimated an increased demand of international flights which would double the current number in 2030. This figure, along with the clear forecast of an imminent lack of pilots, has led the aviation sector to start designing one pilot cabins. In this context, the use of tools to assist with the most complex manoeuvres will be essential. “To achieve a sole pilot cabin is a challenge in which the main aviation companies are currently working on, such as Airbus or Boeing”, states Dr. Miquel Àngel Piera, Full Professor of the Telecommunication and System Engineering Department of the UAB and coordinator of the project.

With this project, Dr. Piera adds, “we aim to improve flight decision-taking in aerial spaces., especially those in which we have a high number of planes, such as close to airports in the space known as the Terminal Maneuvre Area. It is in this area, where planes coexist and compete to descend with the best profile for a correct landing and minimum use of fuel”. Within this context, the ability of identifying behaviour patterns and anticipating manoeuvres of nearby planes will allow pilots to make appropriate changes in the plane’s configuration and thus take the most efficient decisions.

Most aviation incidents tend to occur due to human errors. With this in mind, the project aims to introduce a set of intelligent tools and make sure that all actors involved in the different flight scenarios take the adequate decisions in each particular moment. Furthermore, the project is set to provide tools for real-time monitoring of the cognitive state and workload of pilots.

The flight cabin, a scenario of constant change

The flight cabin has experimented a great technological evolution from the 30s to our day. The Clipper 314, also known as Boeing 314, was, in its moment, the major aviation commercial transport fabricated in series worldwide. This plane had space for 5 crew positions: a navigator, a radio operator, a flight engineer and two pilots, with a clear operative division of tasks: navigating, communicating and administering the system.

Nowadays, flight cabins are highly equipped and have space for two flight members. The E-Pilots Project wants to outfit those cabins with intelligent tools in order to improve the pilot’s attention, as well as its communication with professionals on land (air controllers, for example). “The idea is to design a roadmap for new intelligent applications which helps in the set of tasks with need of a great cognitive activity when taking decisions” declares Ignasi Ingerto, CEO of Aslogic and partner of the project.

Machine Learning in the advancement of highly intelligent cabins

One of the main innovating areas of the project is the ability to recreate a sole and shared perception and comprehension of the flight and its manoeuvres with the different actors involved: pilots (on air) and control centres (on land). The use of machine learning algorithms for the recognition of patterns opens a window of opportunities within the flight cabin.

“We consider the pilot and its operational context as a main actor. The idea is to constantly analyse data provided by the plane and its surroundings in order to endow the cabin with elaborate information. This, in turn, allows us to have a consistent perception and comprehension of future events such as the forecast of different scenarios depending on the decisions that have been executed”, explains Dr. Piera. This task is currently shared with the co-pilot, however, the development of intelligent tools based on pattern recognition opens the door to develop one pilot cabins and, in this way, give response to a rapidly increasing demand of flights.

Human-machine symbiosis proposed by the project will not only to give intelligence to the on board tools, but also monitor pilots in real time. The Computer Vision Center and Aslogic have developed a set of tools which explore and measure the accumulated tension of professionals when managing highly demanding tasks, such as piloting a commercial airplane. With artificial vision, the monitoring of facial expressions and the search of stress and fatigue signals is automatic and real time. To this technique, researchers have added the use of physiological sensors, which measure cardiac and respiratory frequency as well as muscular tension, in order to combine data and be able to anticipate moments of extreme tension, as well as possible collapse of pilots.

If we put together the data of the integrated sensors in wearables and the data from the cameras, we found a set of detailed and highly valuable information to determine the cognitive state of a person, as well as its work load peak”, states Dr. Aura Hernández, CVC researcher and Computer Science Professor at the UAB. “With this data, we can evaluate the need of interfering, be it by increasing the flow of information to the pilot and assisting in the decision-taking process, or directly proposing the best solution for a determined action based on justified information”.

Project tested at the Barcelona Flight School

The project, which started in early 2019, will be tested in the first months of this 2020 at the Barcelona Flight School in flight simulators with students. The Barcelona Flight School is one of the reference centres in the Spanish state for the training of pilots and, therefore, is an excellent site in order to obtain tangible and measurable results.

The Project has a strong market push and researchers are convinced that the technology developed, once validated and certified, will be initially applied to cargo flights.

More information:

This project has received funding from the Clean Sky 2 Joint Undertaking (JU) under grant agreement No 831993. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and the Clean Sky 2 JU members other than the Union.


In the Media:

COPE (12.01.20): Buscan en la inteligencia artificial aviones más seguros con un único piloto

Agencia EFE (12.01.2020): Buscan en la inteligencia artificial aviones más seguros con un único piloto

La Vanguardia (12.01.2020): Buscan en la inteligencia artificial aviones más seguros con un único piloto

Breaking News France (13.01.2020): L’intelligence artificielle recherche des avions plus sûrs avec un seul pilote

Innovadores (La Razón) (13.01.2020): Aviones más seguros con un único piloto gracias a la inteligencia artificial

RTVE (13.02.2020): Investigan el desarrollo de aviones comerciales con un único piloto asistido por la inteligencia artificial

Diario de León (13/01/2020): Inteligencia artificial para volar más seguro.




Carlos Sierra

The author Carlos Sierra