2024 IEEE Conference on Cognitive and Computational Aspects
of Situation Management (CogSIMA)

Video surveillance can be cognitively very demanding as it imposes operators to stay focus for a long time and to provide the right response for relevant stimuli among a lot of information. In this challenging activity, it is relevant to consider the use of decision-aid techniques to improve operators' alertness. The purpose of this study was to examine the impact of a real-time gaze-based tool named Scantracker—which can identify instances of neglect, over-focus and vigilance decrement using eye tracking and display visual notifications to mitigate such situations—on surveillance performance measures during a surveillance simulation. Augmented reality glasses were used to monitor eye movements in real time for all non-expert participants, but notifications presentation to support attention was visually active for only half of them (Scantracker group), as opposed to the control group without support from the Scantracker. No significant differences were observed across those two experimental groups. However, a within-group comparison contrasting trials with active notifications vs. a silent condition showed a reliable improvement in task accuracy and a reduction in screen neglect duration. Results are discussed in light of potential applications of Scantracker with augmented reality.

FFor many years, there has been an interest to support human situation awareness through data automation tools and information fusion systems design that perform situation assessment. Recently, large data ingestion and computing power support historical narrative presentation, information fusion estimation, and plausible future prediction. The tools and techniques are of interest within human-machine teaming (HMT) to combine user reasoning with analytical agents. Deployable HMT solutions require user evaluations, which should be conducted throughout the system design life cycle. Likewise, to determine which design solutions meet the user needs, a preliminary scaling is required to rank candidate solutions. This paper presents a notional approach for human-agent designs to support situational analysis evaluation highlighting user value selection through an analytical hierarchy processing (AHP) metrics developed from the Multisource AI Scorecard Table (MAST). The MAST AHP technique helps to rank situation analysis tools in development design, while the paper highlights one example from the Association of Chats to Tracks (ACT) that facilitates cognitive reasoning of situational scenarios.

Situation management involves the acquisition and dissemination of knowledge to effect situation awareness and assessment. In a companion paper, the concept of Knowledge Acquisition, Representation, Processing, & Presentation (KARPP) system has been introduced. In that paper, it was proposed to manage an adaptive data fusion process including the decomposition of a sensor and database knowledge acquisition and aggregation system into a service oriented architecture (SOA) of knowledge aggregation services (KAS) with standards-based inputs and outputs which are implemented in hidden processes. This paper reviews current approaches to SOA and expands the KARPP architecture into a taxonomy of required and support services as well as discusses particulars of a self-organizing implementation of KARPP in a dynamic SOA of microservices. In addition to a taxonomy of services, a self-documenting approach is introduced to allow for preplanned product improvement (PPI) and incremental development as well as the concept of a Knowledge Aggregation System Service broker.

In search and rescue missions, agents are tasked with locating targets in dynamic and potentially adversarial environments. These agents commonly operate under energy constraints and rely on wireless communication susceptible to intentional interference by adversaries. Thus, it is important to study the impact of adversaries deploying stealthy wireless jamming attacks to disrupt such missions. To this end, this paper introduces a Markov Decision Process (MDP) model designed for the attacker, determining when to jam communication based on the evolving state of the mission. A reinforcement learning- based approach is developed to identify intelligent and selective jamming policies, particularly in scenarios where the identifi- cation of optimal strategies is computationally prohibitive due to the expansive state space. These policies judiciously optimize the tradeoff between the cost of the attack and the expected damage inflicted. We evaluate the performance of these policies in environments of varying sizes. We demonstrate that our approach achieves optimal performance in tractable setups and outperforms alternative heuristics as the complexity of the state space increases.

The foundational principles of warfare have always required a combatant to "know the enemy and know yourself", and to consider the likelihood that methods of deception will be employed on both sides. Further, the need to estimate adversarial situations, threats, and intentions imputes a need for forecasting these states to future times to be relevant to decision-making. These combined requirements lead to a very challenging context for the design, development, and evaluation of multisensor information fusion processes and systems. This paper reviews the nature of these requirements, points out that there is a wide range of related research across varied communities on these topics, and that there is relatively little work in the situation management and information fusion communities currently addressing these challenges. This paper is a call for new and more directly applicable research involving multidisciplinary efforts to advance Level 3 situation and threat estimation capabilities with human and sensor-based fusion systems that is essential for comprehensive situational awareness and assessment in adversarial environments.

The research described in this paper is addressing the problem of designing a comprehensive knowledge ontology of information quality combining ontology of quality characteristics and ontology of models for computing their values and incorporating the designed Knowledge Ontology in the architecture of information fusion systems. A use case illustrating the utility of the designed ontology in an earthquake scenario is also presented.

Knowledge Acquisition, Representation, Processing, & Presentation (KARPP) is a new system level approach to the acquisition and aggregation of mission-valued and qualified situation knowledge for use by a decision maker. KARPP is decomposed into a service oriented architecture (SOA) of knowledge aggregation services (KAS) with well-defined inputs and outputs which are implemented in hidden processes. The transfer of knowledge from one KAS to another is through a qualified knowledge request (QKR) which results in an aggregated knowledge response (AKR). The notion of qualified knowledge is introduced to specify the knowledge that is needed by a requesting KAS as well as the knowledge quality attributes and the magnitude of those qualities which are required to meet mission needs. The goals of this approach are to provide an architectural framework for realizing managed, adaptive fusion processing capabilities, and to integrate the sensor/database control and data acquisition with the fusion-based data processing which will maximize the expected knowledge value rate (EKVR) provided to mission managers.

Online disinformation is an ever-increasing phenomenon that badly affects society stability by using myriads of forms, techniques and channels. Up to now, there is a lack of standard models to analyze the strategies behind disinformation attacks that increase the proliferation of fake or misleading contents over the Internet. Fake or misleading contents are mostly generated by AI-based techniques to be then spread by the minute worldwide. Due to the myriads of different attack strategies and spread speed, analyzing the attack attribution is not an easy task. In this regard, this article presents a knowledge-based approach that exploits the DISARM framework inheriting Cybersecurity principles alongside Situation Awareness concepts to provide analysts with a robust and exhaustive model to analyze and fight back disinformation attacks through the extraction of threat actors' behaviors by reasoning on their attack patterns. Case scenarios on real disinformation incidents demonstrate the practical utility and usability of the proposed model to extract rules for detection and analysis of specific kinds of attackers.

Situation analysis (SAn) multi-dimensional modeling challenges computer support for situation assessment (SAs) rarely presents results that match a user cognitive reasoning for situation awareness (SAw). However, new opportunities exist that can support situation analytics from high performance to go beyond computer simulations. Two recent advances afford SAn that include (1) artificial intelligence methods such as deep learning (DL) and neurosymbolic architectures as well as well as (2) digital engineering to include open architectures and digital twins. While deep learning and open architectures are common in the last decade, recent advances in neurosymbolic approaches are becoming popular to combine neuro (e.g., DL) with probabilistic symbolic reasoning for cognitive understanding. Likewise open architectures allow many digital models to be connected through digital twins. This article highlights the opportunities for digital twins to enhance situation analytics to support situation control, situation understanding, as well as situation forecasting of plausible futures.

Extending an earlier work in which the human takeover times for the operation of automated ships were determined for the first time, this article further discusses details of the problem mainly based on the differences between a simple take over (as reaction) and the discussed case of a cognitive-based take over within a complex dynamical situation. The unsurprising but nevertheless astonishingly long takeover time for the considered example of an automated/assisted inland vessel has consequences in terms of a long preparation time to be considered for takeover situations to be successful. This means that situation-specific, complexity-oriented ship management is necessary to take into account the possible human preparation for the takeover. The contribution introduces into the topic, illustrates the problem based on experimental results and describes a possible strategy to define the required time as preparation time. The analysis is based on a simulator-based experimental study using inland vessels interacting in a highly interactive environment.

Evacuating buildings during emergencies like fires or terrorist attacks demands heightened environmental awareness, swift decision-making, and immediate action. In recent years, there has been a surge in research aimed at bolstering this process and assisting individuals involved through dedicated technical means. However, many questions still linger unanswered. This study elaborates on the concept and the initial design of a mobile application, constituting part of an intuitive evacuation assistance system tailored for evacuation leaders to enhance the situation awareness of individual leaders and the evacuation team in emergencies. Through the proposed system, leaders will receive real-time assistance regarding environmental hazards, evacuation procedures, requests for additional assistance, and the count of occupants in the building or specific areas, among other aspects, thereby enhancing overall situational awareness. Achieving this entails implementing a suitable sensory infrastructure, deploying intelligent algorithms to construct an artificial situation awareness, and crafting a user-centric interface design, all detailed in this contribution. The analytical discourse presented in this contribution highlights the concept's strength, while the mobile application's and sensory infrastructure's initial trials demonstrate its practical viability for real-world implementation.

Large language models (LLMs), like OpenAI's ChatGPT and Google's Gemini, operate as probabilistic models, leveraging their ability to generalize and discern intricate patterns within data. By assigning probabilities to different tokens based on patterns learned during extensive training on large datasets, these models can generate a wide range of contextually appropriate responses, spanning from textual scripts to auditory and visual outputs (both static and moving images). However, the inherent probabilistic nature of LLMs introduces a notable challenge, leading to the phenomenon known in the field of artificial intelligence as ‘AI-hallucination,' where the model may produce responses that sound plausible but are factually incorrect or nonsensical. Despite being perceived as a drawback, we posit in this paper that AI-hallucinations can be reframed as a distinctive feature of LLMs rather than a mere limitation. Our argument stems from the understanding that attempts to mitigate the harms caused by AI-hallucinations might inadvertently lead to increased model rigidity. This delicate balance between minimizing harm and preserving the model's flexibility is a central theme in our discussion. Furthermore, we revisit the concept of ‘context,' contending that a complete definition goes beyond the mere description of circumstances, environment, or surrounding facts. We assert that context is enriched by a conscious embodiment, involving the choice or refusal of action (considering all associate ethical implications) among a set of available options.

The recently introduced foundation models for language modeling, also known as Large Language Models (LLMs), have demonstrated breakthrough capabilities on text summarization and contextualized natural language processing. However, these also suffer from inherent deficiencies like the occasional generation of factually wrong information, known as hallucinations, and a weak consistency of produced answers strongly varying with the exact phrasing of their input query, i.e., prompt. Hence, this raises the question whether and how LLMs could replace or complement traditional information extraction and fusion modules in information fusion pipelines involving textual input sources. We empirically examine this question on a case study from crisis computing, taken from the established CrisisFacts benchmark dataset, by probing an LLM's situation understanding and summarization capabilities on the target task of extracting information relevant for establishing crisis situation awareness from social media corpora. Since social media messages are exchanged in real-time, typically targeting human readers aware of the situational context, this domain represents a prime testbed for evaluating LLMs' situational information extraction capabilities. In this work, we specifically investigate the consistency of extracted information across different model configurations and different but semantically similar prompts, which represents a crucial prerequisite for a reliable and trustworthy information extraction component.

To collect feedback in the early stages of research when researchers have to work with expert users, we propose a method based on the large language models and careful elaboration of personas. This method enables low-cost simulation of answers to interviews and provides inputs for decisions on preliminary research directions. To evaluate our method, we propose to simulate two existing studies in the aviation domain. These studies focus on concerns and expectations of airline pilots about future single pilot operations and higher level of automation in the cockpit. Our results show similar level of concerns to human participants in these two simulated studies. However, the results of the simulations include approximations, errors in the actual work of pilots and over- and underestimations of some potential problems of single pilot operations. We conclude that our method can help guide the preliminary stages of research if researchers have sufficient prior knowledge of the work domain and if this method is complemented by human-in-the-loop methods.

The mineral exploration industry is a vital contributor to the Canadian economy, yet it remains among the most hazardous sectors due to the complex and risky nature of mining operations. The primary objective of this research is to comprehend and predict the specific nature of injury severity within Canada's mineral exploration field to enhance existing occupational health and safety measures. The proposed research is distinctive as it utilizes data from the entire mineral exploration industry in Canada, gathered by the Prospectors and Developers Association of Canada (PDAC). Advanced machine learning (ML) techniques are employed to construct a framework capable of predicting and monitoring injuries in four different classes. Following a description of the dataset's distribution, eight distinct machine learning methods, including Support Vector Machine, Convolutional Neural Network, Bayesian Neural Network (BNN), logistic regression, decision trees, random forest, Gradient Boosting, and Long Short-Term Memory (RNN LSTM), were applied to predict the nature of injury in different mining activities. The results of the framework indicated that multi-classification with RNN-LSTM outperformed other algorithms, accurately identifying the degree of injury with 97% accuracy across all metrics. These findings have the potential to significantly contribute to injury prevention efforts by increasing awareness of potential safety risks and providing quantitative predictions of fatal injuries and future accidents in mining exploration fields.

Much attention was devoted to the ChatGPT and other Large Language Models' (LLM) capability assessment regarding syntax correctness, factual accuracy, adequate world representation, ethical alignment, common sense and formal logic reasoning. However, most of the research focused on ``statically'' generated texts, when the result of only a single iteration between a human and LLMs was recorded. More advanced techniques of open-ended discussions or debates between a human and LLMs produce much more interesting results, demonstrating such faulty rhetorical behaviours as circular arguments, self-contradictions, evasion, change of topic, lack of consistent position, and the mix of passive aggression with attempts to please human disputant. We present an original observation of such behaviour during the ChatGPT dialogue session discussing the translation of Japanese song lyrics.

In the evolving landscape of North American Aerospace Defense Command (NORAD) operations, the collaboration between human operators and autonomous systems is imperative for effective Command and Control (C2) in the face of increasingly complex threats to Aerospace Defence (AD). As traditional threats persist and novel challenges such as Hypersonic Glide Vehicles emerge, the mental burden on human operators grows, necessitating the integration of Artificial Intelligence (AI) and automated tools. This paper focuses on Human Autonomy Teaming (HAT) for bridging the communication and trust gap between the human and the machine, presenting a novel cloud-compatible solution, the Verifiable Human Autonomy Teaming (VHAT) system. VHAT aims to enhance AD capabilities by automating target identification, tracking, and trajectory predictions, addressing the challenges associated with the static nature of AI decision-making. The system is designed to instill trust and allow auditing of AI decision support while featuring scalable machine learning models, advanced training capabilities, tailored AI models for air defense, and distributed data processing. The paper presents the conceptual system architecture of VHAT, its AI-based capabilities, and the integration of HAT features through a graphical user interface (GUI).

Asset protection constitutes identifying threats and responding to them in time. The process of analyzing a situation that potentially presents a threat and deciding on how to respond to it typically engages both automated machine learning algorithms and human analysts. Human judgment is more accurate than that of a trained algorithm, but humans take more time to piece the available evidence together. The best assessment strategy is to let human analysts partially rely on machines, i.e., generally trust the machine's judgment but at times take time to make their own judgment supersede the machine's judgment, depending on the asset value at stake. This paper presents an optimal just-in-time policy to assess the incoming situations -- threat cases -- by a human-machine tandem where humans have a limited bandwidth for personally reviewing the cases. This presents the problem of balancing the human attention allocation, conditioned on a prior assessment of the risks the cases might present. The problem is modeled and solved as a stochastic sequential assignment problem informed by the concepts of decision game theory and the value of information.

The Kidney Exchange Problem (KEP) determines organ exchange chains and cycles amongst a pool of patient-donor pairs (PDP) and non-directed donors (NDD) allowing for the maximum number of kidney transplants. The problem is complicated by optimization occurring over a sparsely connected, directed graph. The presence of an edge in this graph suggests a feasible transplant from a NDD or PDP to another PDP. Many traditional approaches treat the presence of edges in the exchange pool as known and certain. However, the certainty of edges in the exchange is unknown until optimization has been completed and transplants are offered. Edges that are thought to be present may fail because of physician preference, patient behavior, or previously unknown biological incompatibility. As a result, a disparity exists between the number of exchanges planned in optimal solutions and the number of exchanges that take place in the real world. Therefore, this work proposes an integrated KEP optimization methodology that learns a representation of features that affect the realization of optimized solutions. This methodology is implemented using graph machine learning and allows for the integration of additional patient-donor attributes and collaboration between the optimization process and physician behavior. To evaluate this solution method an approach for simulating the implementation of KEP solutions is developed. An analysis of the required data inputs for both the solving and assessment methodology is noted. A discussion of the limitations of the work is presented and directions for future works are proposed.

Decision making situations in emergency management are usually characterised by participation and collaboration of multiple actors with sometimes different competences and knowledge, lacking information and/or having uncertain information. Hence, there is a need for a coherent and effective collaboration leading to shared awareness and decision support in a timely manner. This can be achieved with a collaborative incident case management platform that permits all levels of response hierarchy supporting the Concepts of Operations (CONOPS) and which complies with the measures of every agency involved bringing all responders together toward a unified efficient response. This paper considers the Canadian Coast Guard (CCG) use case in Marine Incident response system on a large scale, its inconsistencies, redundancies, gaps, and misalignments, and proposes a novel collaborative marine incident case management platform, the marine incident Case Management System (CMS) for shared situational awareness and decision making in marine incident management. The system is developed with a single approach to incident management using the Scenario Based Design (SBD) methodology and a root cause analysis to review, compare past incidents and their associated After Action Review (AAR) as well as Cognitive Task Analyses (CTAs) to validate use-case scenarios. The cognitive and computational aspects of the proposed systems are presented and discussed.