Sixth Sense - Air Traffic Control Prediction Scenario Augmented by Sensors ∗ Florian Grill, Theodor Zeh
Nelson Silva, Volker Settgast, Eva Eggeling Fraunhofer Austria, 8010 Graz, Austria
{nelson.silva, volker.settgast, eva.eggeling}@fraunhofer.at
Frequentis AG, 1100 Vienna, Austria
{florian.grill, theodor.zeh}@frequentis.com
Dieter Fellner Institute of ComputerGraphics and KnowledgeVisualization (CGV) Technische Universität Graz, Austria GRIS, TU Darmstadt & Fraunhofer IGD, Darmstadt, Germany
[email protected] ABSTRACT
General Terms
This paper is focused on the fault tolerance of Human Machine Interfaces in the field of air traffic control (ATC) by accepting the overall user’s body language as input. We describe ongoing work in progress in the project called Sixth Sense. Interaction patterns are reasoned from the combination of a recommendation and inference engine, the analysis of several graph database relationships and from multiple sensor raw data aggregations. Altogether, these techniques allow us to judge about different possible meanings of the current user’s interaction and cognitive state. The results obtained from applying different machine learning techniques will be used to make recommendations and predictions on the user’s actions. They are currently monitored and rated by a human supervisor.
Algorithms, Human Factors, Experimentation, Verification
Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics—complexity measures, performance measures; I.2.1 [Computing Methodologies]: Artificial Intelligence—Applications and Expert Systems; I.5 [Computing Methodologies]: Pattern Recognition—Miscellaneous ∗Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). i-KNOW ’14, Sep 16-19 2014, Graz, Austria ACM 978-1-4503-2769-5/14/09. http://dx.doi.org/10.1145/2637748.2638441
.
Keywords Human machine interface, recommendations, inference, probabilistic, mental map, sensor fusion, cognitive, pattern, air traffic control, machine learning, expert systems, knowledge
1.
INTRODUCTION
Single European Sky Air Traffic Management Research, or SESAR, targets productivity while keeping the humans in the core of the continuous evolving systems. This means that we need to improve the coexistence of humans and machines to keep up the pace. In our project Sixth Sense we address mainly the longterm innovative research topic of ”paving the way to full automation”. Therefore, we have the goal to improve error detection through sensor augmentation while keeping the humans in the main decision loop of air traffic control (ATC). The first step to achieve this goal, is to record and analyse the overall user’s body language and use it as a source of data to illustrate the user’s current mental map. In our project the user is an air traffic controller, and we analyse his body language at the working place, the so called integrated controller working position. To collect this data, we integrate multiple sensors like eyetracking, mouse data, Kinect, leap-motion, voice recognition, bio-sensors and air traffic control information, such as weather and flight data. We fuse this data into most likely interaction patterns. This sensor fusion is needed because, it is the most plausible way to collect information about the body language of the user. This body information can help the system on identifying moments of stress, or extreme relaxation. We can discover when the user is agitated, or we can learn about where the user is looking at or what the user is trying to achieve. This extra data provided by the sensors, when combined with the air traffic systems knowledge and the user interface interactions, will allow the algorithms to make valuable correlations and predictions. Autonomous correlations, based on high frequency time events, are performed. This allows the system to infer about different probabilities of the user’s next interaction and current cognitive state.
These correlations are achieved with the help of new technologies, such as, complex event processing (CEP) [5] and a domain specific language (DSL) for processing events [6]. In the Sixth Sense project we make use of a tower simulator prototype developed by the company Frequentis AG. This prototype incorporates simulated air traffic data like flight, weather, runaway info and also data about user interface actions and other useful information. To collect data for the project, the air traffic controllers will perform their normal tasks, while using the tower simulator in a pre-defined and realistic ATC scenario. The intention is not to give feedback to the air traffic controller or support him in anyway at this stage. Instead, the prototype aims to augment the working environment with extra sensors and collects data about the user’s body language, such as: arm movements, body positions, hand gestures. It correlates this body language with the choices made with the user interface of the simulator. It also uses the information about the current air traffic conditions (wheather, airplanes types, positions). The work of the controller is then analysed. Based on this analysis our framework generates recommendations for the current work performance of the user and presents this results on a dashboard for the ATC supervisor. The workflows of ATC centres and control rooms represent the production process of other known producing enterprises. It defines what needs to be done by whom and when, especially in ATC centres this process is defined quite well. In Sixth Sense we use inputs from another SESAR project, the ZeFMaP [2] project, where a detailed analysis of the tower process has been performed in the form of a task and respective value stream analysis. Additionally, we use inputs from earlier work [9],[4], where the different roles and tasks of a tower control centre are described.
1.1
Overall Hypothesis
The process of making decisions can be greatly supported by having intuitive and ergonomic user interfaces. If Human Machine Interfaces are designed in an integrated, task-oriented and appealing manner, operators will find it easier to use these tools and actually rely on them. The results of the Sixth Sense project can be used to provide valuable information for a wise redesign of these same user interfaces. However, there is an essential question, that we strive to answer: ”When working with an Air Traffic Management system (ATM), can the user’s body language and personal choices provide valuable cognitive information to uncover hidden patterns, that will allow us to distinguish between good and bad (erroneous) inputs and interactions and ultimately avoid mistakes and bad decisions?”
1.2
Assumptions
Having the overall hypothesis in mind, we know that the ATC working position is now augmented with extra sensors, that were not there before. These extra sensors, provide big amounts of data about the user’s actions, decisions and overall body language. We also make the assumption that at this stage we are not giving direct feedback to the air traffic controllers performing on their working positions, but instead we are presenting the results of the prediction engines to the supervisor. The supervisor is currently monitoring and ranking both the user’s actions and the prediction engines’ suggestions. In the following sections, we will have a deeper look on more details about the experiments with the combined prediction architecture approach.
2.
PREDICTION ARCHITECTURE
The idea of the Sixth Sense project is to make use of the whole body language of the user, to depict the most likely wanted interaction out of all possibly received ones. For example the air traffic controllers might mismatch similar call signs, e.g. LH357 and LH375. If we combine voice recognition with eye tracking we can depict that the controller talks to LH357 while watching LH375. However, this is just a very simplistic example. Our prediction scenario architecture is composed by four essential functional blocks: • The air traffic control tower simulator framework which is managing multiple graphical user interfaces and is distributing data via the network. • The complex event processing (CEP) block incorporates an SQL database that is used for logging and reconstructing events on the tower simulator and a NoSQL graph database [3] that is used for recommendation and correlation. • In the prediction (DM/ML/AI) block there is an Apache Hadoop and Mahout [7] recommendation engine and also a Bayesian inference engine setup [1] • A data visualization block which features HTML5 chart visualizations, filtering and data exploration capabilities.
2.1
Sensor Augmentation Strategies
All the available air traffic information is collected into an ActiveMQ message queue system (AMQ), in the form of XML messages separated using topic descriptors. The same was achieved for all the different sensors, as an effort to incorporate this new information into existent legacy systems. With the goal of saving processing time on repetitive lookups, we also aggregate specific air traffic data with sensor data, such as integration of air plane call-sign with eyetracking gaze position. We would like to pin point some use case scenarios for the different sensors, based on the expected usage in the context of ATC: Mouse and Eye-tracking Data - Possibility of error detection due to the correlation between eye-tracking and mouse positional information. Detection of excessive demand based on eye-tracking data. Detection of user tiredness based on eye-tracking information. Kinect Data - Possibility of error detection due to the correlation between air traffic information and the user gestures. Detection of excessive demand based on body and gesture tracking information (speed of gestures). Detection of user tiredness based on body posture. Recognition of Voice Commands - Used to control the user interface. We treat the available air traffic information like other data coming directly from physical sensors. In this sense, we consider voice recognition as a virtual sensor.
2.2
Prediction Workflow
Our prediction workflow (see Figure 1) starts with an ”observe phase” and the collection of data. We assume here, that there is already enough data collected (a priori) for training the recommendation algorithms. This data (available in the form of queue messages) is derived from user interactions with the ATC system augmented by sensor information. The messages are immediately transformed into events through the complex event processing platform, where we have the possibility, to aggregate, filter or correlate multiple different event sources (i.e, ATC system, eye-tracking). Furthermore, we can create time windows to store the arriving events for n seconds (other possibilities are available).
This is called the ”Detection (Derive)” phase. Simultaneously to these messages and events there is a ”prediction phase”, where a Hadoop Mahout Recommendation Engine system together with a Bayesian Inference Engine will generate recommendations, based on a priori and on current knowledge, provided by the complex event processing platform. These results will then be assembled back into new events. After some optimization and temporal decision processes, the platform will be able to present predictions or display the current decision workflow to the ATC supervisor, by means of the visualization component of Sixth Sense. Also, in parallel to these processes, there is place for a ”long term analysis and training phase”, supported by the recommendation engine and by the SQL message queue logger database. We make use of a Kalman filter and Hidden Markov Model algorithm on the mouse and eye-tracking gaze position. The algorithm predicts (within a time window frame) the mouse and eye-tracking position for the next n seconds. The results are stored in a graph database for posterior correlations.
2.3
Unexplored Algorithms Combination
Situation assessment is a process, that attempts to develop a description of current relationships among entities and events in the context of their environment. Because of that, available information is being processed automatically and interpreted in order to find high-level relationships and to make additional inferences on the current situation, which can support the supervisor to gain a good level of comprehension of the situation that the air traffic controller has at hand. In our scenario, the system has to constantly learn from the ”observed” behaviour. This means that no explicit definition of input-output relations shall be required. The system continuously learns from ”sensed” event descriptions, which are combinations of current situation parameters (like temperature, numbers of aircrafts in a sector, performed user interface actions) and decisions (like opening a document or changing a flight strip position) and information. Based on the parameters used for ”learning”, the system has to make proposals in the next step. This means that given the current situation parameters the system shall provide a list of most likely actions. Prediction technologies like CEP, Mahout recommendation engines or probabilistic programming[8] are currently heavily used in other domains and industries. This inspired us to also use them in the field of ATC. Therefore, this work comprehends experiments on how to implement these prediction engines and the creation of prediction models. We want to discover if they are usable and effective in this specific context of sensor augmentation in the ATC working position.
3.
DATA EXPERIMENTS AND AVAILABLE RESULTS
At this stage we are analysing the available ATM data in order to evaluate the best correlations, that must be implemented at the filtering and aggregations level. We are also creating lookup tables to support the prediction of the gaze and mouse position. Due to technical impediments, it is not possible to return for a given x,y position the respective user interface control at that location. This information is useful to know in advance (at a certain future time frame), and based on the x,y position prediction of the mouse or the gaze point of the air traffic controller, what will be the next interesting user interface control. Also, we need to better evaluate the already collected data, generated by the air traffic controllers when they utilize the ATC simulator (already augmented with sen-
sor information) in simulated realistic scenarios. This step is required to bring the framework to an initial knowledge status and to train the recommendation algorithms. After finalizing these tests with sample data, another step will then take place. More air traffic controllers with diverse experience levels, will be using the system and multiple supervisors will be in the monitoring and ranking position. This is work in progress to determine how many users we need for the tests and to draw further conclusions. We need to know in which conditions the controllers will perform, or what type of specific workflow errors can be introduced into the simulations, so that the system can later recognise them. We are currently creating a verification plan and preparing more realistic and complete simulation scenario.
3.1
Enriching Eye-Tracker and Mouse Data
Sequences of mouse movements and gaze positions are stored in a database. Short time frames of a few seconds will be used to analyse the past movements and predict future mouse cursor positions and gaze points. Hidden Markov models (Markov chains) are used as a way to describe a decision workflow that can be represented in a graph database.
3.2
Recommendation and Inference Engines
The recommendation engine allows us to create predictive features, such as personalization, recommendation and content discovery. It allows the usage of state of the art machine learning algorithms and their fine tuning, evaluation and scientific implementation. It also comprehends the usage of parallel implementations for many of these algorithms, while allowing us to take advantage of Hadoop benefits. The inference engine allows running Bayesian inference in graphical models. It also provides state-of-the-art messagepassing algorithms and statistical routines that are needed to perform inference for a wide variety of applications.
3.3
Realistic Air Traffic Simulation and Data Collection
The necessary air traffic data and descriptive technical documents for the preparation of our framework is provided by Frequentis AG, the company that developed our air traffic tower simulator. The sensor data is stored as logs (indexed text files), that can be processed directly by the recommendation engine. The ATC simulation and prediction architecture evaluation will be part of our verification plan, which is currently in preparation. In a project of this nature, it is a challenge to get the right data and the right amount of data. Furthermore, there is a high number of technical limitations that must be overcome. For example, the integration of sensing technologies with the existent ATM system used for the simulation scenarios. Another example is the need to aggregate knowledge about events, in short time windows, just before the work of the algorithms takes place. But we believe, that the Sixth Sense project will provide a new level of active support for the ATC supervisors in their daily work of air traffic management. Only a small part of all possible existent sensors can be analysed within our current project. We believe, this is a great starting point for adapting a system to the user. As a major outcome of the project we expect to bring the supervisor much more into the core command of the overall operations. In this manner, the sensors provide information about the intentions of the controller, and the corresponding analysis will result in a suggestion or reaction that is beneficial for the supervisors work. We are also working on ways to introduce a mechanism, to make usage of ontologies as a hierarchy of user interface concepts, where we can take advantage of the interrelationships of those concepts.
Figure 1: Sixth Sense - Major Functional Blocks
4.
CONCLUSIONS
ATC systems can be improved by monitoring the controller’s actions and decisions. The proposed system allows to record and collect the controller’s direct inputs as well as sensor data, including body language together with the current status of the ATC system. It is possible to divide the controller workflow into tasks and identify individual decisions. Supervisors will rank these decisions and the system predictions. By combining all this data we expect to improve the overall performance of the ATC system. We are trying to predict what will be the next choice of a user and if the user is moving or relaxed, by observing the respective user’s movements. We want to register the best workflows of the different users on similar working conditions and we want to show recommendations about the best optimal actions to be taken next, based on previous knowledge. Eventually, this will lead to a prediction system that is able to identify stress situations and unusual behaviour. At this stage, we already could overcome several challenges and we are conducting the necessary data analysis and doing the fine tuning of the system. The final step of the Sixth Sense project will be to confirm if the framework and overall prediction concept is performing well. We will also receive feedback from the supervisors regarding the results and advices given by the platform. This might open up the field of ATC to other research topics, leading for instance to the implementation of more comprehensive cognitive models for the air traffic controllers. This facilitates, for example, the acceptance of this nature of intelligent systems by the controller’s community, or the integration of more forthcoming sensing technologies in the near future.
5.
ACKNOWLEDGEMENT
This work is co-financed by EUROCONTROL acting on behalf of the SESAR Joint Undertaking (the SJU) and the EUROPEAN UNION as part of Work Package E in the SESAR
Programme. Opinions expressed in this work reflect the authors’ views only and EUROCONTROL and/or the SJU shall not be considered liable for them or for any use that may be made of the information contained herein.
6.
REFERENCES
[1] C. Bishop. Pattern Recognition and Machine Learning, volume 4. Springer, 2006. R - Integrated Controller [2] Frequentis and DFS. iCWP Working Position. SESAR, 2012. [3] S. Jouili and V. Vansteenberghe. An empirical comparison of graph databases. In 2013 International Conference on Social Computing, pages 708–715. IEEE, Sept. 2013. [4] J. Langlo et al. Usefulness of FMECA for improvement of productivity of TWR process. In Proceedings of Second SESAR Innovation Days, 2012. [5] S. Lehmann, R. D¨ orner, U. Schwanecke, N. Haubner, and J. Luderschmidt. Util: Complex, post-wimp human computer interaction with complex event processing methods. In 10. Workshop ”Virtuelle und Erweiterte Realit¨ at” der GI-Fachgruppe VR/AR, Aachen, pages 109–120, 2013. [6] I. L. Narangoda et al. Siddhi : A second look at complex event processing architectures. In GCE ’11 Proceedings of the ACM workshop on Gateway computing environments, 2011. [7] C. E. Seminario and D. C. Wilson. Case study evaluation of mahout as a recommender platform, 2012. [8] F. Xu and T. L. Griffiths. Probabilistic models of cognitive development: Towards a rational constructivist approach to the study of learning and development. Cognition, 120(3):299 – 301, 2011. [9] T. Zeh et al. Zero Failure Management at Maximum Productivity in Safety Critical Control Room. In Proceedings of the SESAR Innovation Days, 2012.
