Space and Artificial Intelligence

Our interdisciplinary seminar on Space and Artificial Intelligence was situated at the intersection of research on AI / computer science and space research. Since each of these is a very wide field on its own, below we give a broad outline of each of the two. We focus on the aspects that were topics of discussion at our Seminar.

Artificial intelligence studies computer systems that behave similarly to humans, in a way that the resulting behaviour would be considered intelligent if exhibited by humans. The field of AI thus focusses on the design and analysis of algorithms and systems that can replicate, support or surpass human perceptual, linguistic, and reasoning processes; learn, draw conclusions, and make predictions based on large or small quantities of data; replicate or enhance human perception; support humans in diagnosis, planning, scheduling, resource allocation, and decision making; and cooperate physically and intellectually with humans and other AI systems (https://claire-ai.org/what-is-ai/). All these topics are relevant for space research.

At a high level, AI can be categorised into three (non-exclusive) categories:

Data-driven AI

makes use of data in order to produce intelligent behaviour; it prominently encompasses machine learning, data mining, and pattern recognition approaches and is often referred to simply as machine learning. In this area, methods based on neural networks have been particularly successful and, as a result, become a major focus of attention for the last decade, but many other approaches exist and continue to be used with considerable impact, including support vector machines and random forest models for supervised learning, and various types of clustering methods for unsupervised learning.

Knowledge-based AI

is focussed on the explicit formalisation of human knowledge and its use for tasks such as reasoning, planning, and scheduling. Although knowledge-based AI is currently somewhat less prominent than data-driven AI, it has important and impactful uses, e.g., in ensuring the correctness of computer hard- and software, and in solving a broad range of real-world industrial optimisation problems. Many AI experts now believe that combinations of data-driven and knowledge-based methods are likely to provide the basis for next-generation trustworthy AI systems. In this context, explainable AI (XAI), where the results of AI solutions can be understood by humans, is gaining importance.

Embodied AI

concerns the design and study of AI systems that interact directly with the physical world. This area is also known as robotics and has very important applications in an increasingly broad range of application sectors, including manufacturing, medicine, and agriculture. Interaction with the physical environment (including other robots and humans) poses unique challenges, e.g., in terms of safety, robustness, and real-time requirements. Most experts in the field of robotics make use of knowledge-based and data-driven approaches, in addition to specialised methods for dealing with the previously mentioned challenges.

Legal and ethical aspects of AI have also started to attract attention recently. The legal part includes laws that regulate the use and development of artificial intelligence. The ethical part is concerned with the moral behavior of humans as they design, make, use, and handle artificially intelligent systems, but also with the moral behavior of machines (machine ethics).

The different forms of AI can be applied to a variety of problems in space-related research, of which here we highlight two major branches:

Space Operations (SO)

are concerned with all aspects of operating spacecraft, including the planning, implementing, and operating of all (also ground segment) systems required for reliable and efficient spaceflight missions. This includes all relevant mission operations, ground infrastructure, flight dynamics, mission planning, communications, and data acquisition functions. Large amounts of data about space operations are collected and can be utilized by ML / data-driven AI to address challenges that include autonomous spacecraft route planning, spacecraft anomaly detection, and optimal spacecraft operations.

Earth Observation (EO)

is a major instrument for monitoring our planet, its land and ocean processes, and their dynamics. A large number of spacecraft carrying a broad range of instruments generate a wide variety of sensor data (active / passive) of many resolutions: With these data now accessible to researchers and agencies, as well as the general public, a final barrier remains the need to convert the enormous quantities of raw EO data (generated on a daily basis) into valuable information for making decisions and taking concrete actions, e.g., towards achieving the Sustainable Development Goals. Needless to say, the potential for applying AI and ML in this context is almost unlimited.

Many other space-related AI applications can be conceived, typically related to the use of ML for the analysis of data collected during specific space missions. These include, e.g., modeling and forecasting space weather, mapping planet surfaces, galaxy profiling, identifying exoplanets and their environment, as well as analyzing astro-biology data. Several of these belong to astronomy and concern data collected via astronomical observatories in orbit.

The seminar covered many different aspects of Artificial Intelligence for space and touched upon a wide variety of topics. However, it focussed specifically on the following four topics – all of which are currently actively researched – structured along two dimensions (AI approaches and Space applications):

Data-driven AI, e.g., machine learning, for space.

The first topic of the seminar addressed machine learning methods for the analysis of the ever larger quantities of data resulting from space related research and exploration, their current state-of-the-art, and directions for further development.

Knowledge-driven AI, e.g., explainable AI, for space.

The second topic of the seminar was concerned with methods and techniques from knowledge representation and reasoning, and explainable AI, their current state-of-the-art, and directions for further development.

Space Operations applications of AI.

The third topic of the seminar concerned various aspects of operating spacecraft and managing missions, the potential applications of AI in this area, and the challenges they pose for Artificial Intelligence methods.

Earth Observation applications of AI.

The fourth topic of the seminar concerned various aspects of applying AI to Earth observation data, the vast variety of potential applications of AI in this area, and the challenges they pose for Artificial Intelligence methods.

Note that the topics along the space applications dimension interact strongly with the AI approaches dimension. For example, space operations applications, such as estimating the current and predicting the future states of spacecraft, have a strong temporal dimension requiring the use of data stream mining approaches from AI. On one hand, this poses challenges to address in the development of novel AI methods. On the other hand, this can provide excellent benchmarking opportunities for the evaluation of AI methods.

The above four topics were the focus of the seminar. Given the interests of the participants of the seminar, we also considered a few additional topics (to a lesser extent). These included, for example, legal, ethical, and social aspects of Space AI.

The structure of our seminar was standard for Dagstuhl. We started with an introduction round on Monday morning. The majority of the time was taken by plenary talks and parallel discussions in working groups: There were two of the latter, one on Tuesday morning and one on Friday morning. The social event on Tursday afternoon included a visit to the Völklingen Ironworks UNESCO industrial heritage site and a dinner.

Plenary talks.

Given the highly interdisciplinary nature of the seminar, participants from one discipline needed to be brought up to speed with the state of the art in the other relevant disciplines. Some of the talks were thus of an overview or tutorial nature. Examples of such talks are “Introduction to Space Operations” and “Introduction to Explainable Artificial Intelligence”. Other talks were more specific, addressing particular AI methods or classes thereof or particular (areas of) AI applications in space research.

The plenary talks can be clustered into four different groups

• $\mathrm{■}$

  Plenary talks on machine learning,

• $\mathrm{■}$

  Plenary talks on explainable AI,

• $\mathrm{■}$

  Plenary talks on earth observation, and

• $\mathrm{■}$

  Plenary talks on space operations.

Parallel discussion in working groups.

A substantial part of the seminar time was split into structured small-group work sessions. The aim of the structured work sessions was to address the focal topics of the seminar that were most interesting for the participants. The participants could more effectively share knowledge and experiences from their own areas of expertise in the smaller working groups. The highlights of these structured small-group sessions were presented to the seminar as a whole.

The parallel discussions in working groups on Tuesday morning addressed the following topics:

• $\mathrm{■}$

  Sustainable development goals and AI for good,

• $\mathrm{■}$

  AutoML and benchmarks,

• $\mathrm{■}$

  On-board and frugal AI, and

• $\mathrm{■}$

  Responsible AI.

The parallel discussions in working groups on Friday morning all addressed the same topic of challenges in AI & space and future research directions.

The seminar brought together a diverse set of players. These included researchers from academia, on one hand, and practitioners from space agencies (ESA, NASA) and industry, on the other hand. It covered a broad range of aspects relevant for the further development of the field.

The major outcomes of the seminar are as follows:

1. 1.

   It gave researchers from the different contributing disciplines an integrated overview of current research in the area of artificial intelligence for space.

2. 2.

   It reinforced the communication channels for researchers tackling challenges in space applications using AI, including both data driven and knowledge-driven approaches to AI, such as machine learning and explainable AI, thereby bridging the divide between computer science and space research.

3. 3.

   It defined the landscape of potential applications of artificial intelligence in space, in particular in the areas of Space Operations and Earth Observation.

4. 4.

   It identified the central research questions and challenges for artificial intelligence approaches that need to be resolved for successful use of AI in space applications.

5. 5.

   It put forward some strategies for designing artificial intelligence tools for space applications and for developing benchmarking suites for evaluating such approaches.

The talk will discuss recent work on semi-supervised (SS) [4] and multi-label classification (MLC) of remotely sensed images (RSI) [1]. For MLC, we employ deep neural networks (DNNs), either as feature extractors for predictive clustering trees (PCTs) and ensembles thereof, or in an end-to-end manner. In the former case, we leverage the existing capabilities of semi-supervised PCTs and ensembles: explainability of single tree models and state-of-the-art predictive performance of random forest ensembles [3, 2]. Furthermore, the parametrization of the amount of supervision in PCTs allows us to build supervised, semi-supervised, or unsupervised models, depending on the demands of the dataset at hand. This provides a safety mechanism enabling the semi-supervised models to consistently perform better or as good as their supervised counterparts.

We also develop end-to-end semi-supervised DNNs for multi-label and multi-class classification of remotely sensed images. This method mimics the mechanism of semi-supervised PCTs that have been proven to work well. We introduce a novel loss function that combines classification loss (computed on labeled data) and reconstruction loss (computed on both labeled and unlabeled data) with a weight parameter that enables the same aforementioned “safety mechanism”.

The capabilities of PCTs and ensembles of PCTs (e.g. Random Forests) enable us to perform hierarchical MLC of remotely sensed images – a novel formulation of the classification task in this field. To this end, we exploit the intrinsic label hierarchies of the BigEarthNet dataset and explore the effects different label hierarchies and their different handling have on predictive performance.

Recently, eXplainable AI (XAI) gained momentum both in academia and industry to explain the results of black-box machine learning algorithms. A landscape of XAI branches along with strategies for developing explainable models are provided. Latest empirical studies have confirmed that explaining a system’s behavior to human users fosters the latter’s acceptance of the system. However, providing overwhelming or unnecessary information may also confuse the users and cause failure. For these reasons, parsimony has been outlined as one of the key features of XAI with parsimonious explanation defined as the simplest explanation that describes the situation adequately. Our work proposes HAExA, a human-agent explainability architecture to formulate parsimonious explanations for remote robots. This is particularly applicable to space since the communication with Earth has limited bandwidth and significant delay. Finally, some challenges, opportunities, and applications of XAI directed to different space stakeholders are presented.

The talk introduces space operations fundamental concepts in brief. First, the phases in time. The operations preparation phase, includes the setting up and customization of the ground segment elements, made of hardware, software, procedures, the specialists training and the simulation campaign. Then the operations execution, split in LEOP, Commissioning phase, Routine phase and decommissioning. The second part addresses the two parallel chains of health caring of the spacecraft and the productive chain, made of planning, execution, payload data acquisition and dissemination. Both parallel chains incorporate a variety of tasks that can embed AI algorithms. The identified tasks are preparation, planning, execution, monitoring, forecasting, diagnostic, optimization.

This presentation addresses several challenges in the space domain that have prompted the creation and dissemination of meticulously curated datasets and optimization problems, contributing to the broader academic community. An overview of significant challenges, including the Proba-V super resolution challenge [1], the data-driven “The OPS-SAT case”challenge [3], the collision avoidance challenge [4], and the pose estimation challenge [2], is provided. The talk delves into each challenge, offering brief insights into their objectives and methodologies, while also sharing select results achieved in these endeavors. Additionally, the application of Machine Learning (ML) inversion techniques within the domain of Geodesy for irregular solar system bodies is presented highlighting the utilization of ML methods to address challenges specific to Geodesy, showcasing results obtained through this innovative approach. The incorporation of ML in geodetic processes not only introduces a novel dimension to a traditional problem but also demonstrates its potential to yield meaningful insights on the internal structure of irregular bodies.

The talk aims to shed light on the interdisciplinary applications of ML techniques in the space domain, emphasizing the collaborative and innovative efforts that drive advancements in space-related research.

Over the past decade, Earth Observation (EO) has undergone a significant transformation thanks to the deep learning revolution and the increasing use of Artificial Intelligence (AI) to address various EO challenges. Although a standard framework for developing AI solutions to EO problems has been established, it still faces several challenges such as lack of labeled data, generative modeling, and integration of physics.

Foundational models offer a new perspective by integrating unsupervisedly learned knowledge in large models that can be adapted to various use-cases. The Phileo foundational model is presented as an example of desirable features, including global-scale training and the combination of pillar models with various pretext tasks. The European Space Agency has commissioned future European Foundational Models for EO and Society, as well as Climate, which will further advance this field.

Conventional modelling approaches are reaching the limit of computational capability as well as hard limits in power consumption. How can ML methods be best used to improve forecast skill, computational efficiency and data fusion in a highly distributed data centre structure? Should the focus be on Open Source foundation models that can be a community tool or hybrid model approaches or DTE types of approach? And what are the risks of one or two Big Tech companies developing a monopoly in the field and pushing not for profit organisations such as ECMWF out of the market?

Operating Earth observation satellite constellation raises many challenges for AI and Agent-based Approaches [1]. Here, we identify questions about planning observation tasks as to monitor physical parameters using taskable agile EO satellites. Notably,

1- For monitoring physical phenomena, each acquisition carried out by a satellite provides not only instantaneous information on the observed area, but also (a) information on neighboring areas due to spatial correlations and (b) information on the value of physical parameters in the future, due to temporal dependencies. Therefore, a difficulty lies in assessing the value of each observation, knowing that due to limited capacity, planning systems for satellite constellations must select observations from a set of candidate observations [2, 3]. 2- The satellites considered can be equipped with different sensors, and some satellites are even capable of carrying out observations in several modes (for example, “wide field observation” mode versus “targeted observation” mode). Managing this heterogeneity is also a challenge for evaluating the reward associated with each observation. For example, it is necessary to find a compromise between, on the one hand, continuously maintaining global knowledge on the value of physical parameters, and on the other hand, carrying out targeted observations on areas where phenomena have been detected. 3- The choice of a good observation strategy depends on the dynamics of the physical process being monitored. For example, monitoring deforestation does not require the same frequency of observation as monitoring illicit degassing from ships. Ideally, the automated planning system should be able to learn a good observation strategy from a global request to monitor a parameter, rather than waiting for basic observation requests formulated by users.

This working group concentrated on the use of AutoML methods in Space Operations and Earth Observation, as well as the presence of benchmarks.

From the discussion of this group, a number of short term and long-term opportunities were identified. Short-term opportunities include:

• $\mathrm{■}$

  Continuous engagement with users

• $\mathrm{■}$

  Increased collaboration between agencies and research facilities

• $\mathrm{■}$

  Make agencies (ESA, NASA, etc) visible to the research community

• $\mathrm{■}$

  Connect ESA and NASA to agree on some agreements to be competitive world-wide

Long-term opportunities include:

• $\mathrm{■}$

  International agreements on common approaches, standardization, and shared capabilities in AI/ML

• $\mathrm{■}$

  Space-based Autonomy for longer-term flights, system level control and onboard decisions

• $\mathrm{■}$

  Use of AI for space mining

The working group 2 discussed the following points:

• $\mathrm{■}$

  Verification and Validation (for software and products of ML pipelines)

• $\mathrm{■}$

  Open Source Pipelines and Benchmarks (especially in Operations)

• $\mathrm{■}$

  Smaller is better (produce smaller models, trade off between model accuracy and size)

• $\mathrm{■}$

  Edge computing for ground/space segments.

A summary of the discussion for each point follows below.

The working group 3 discussed the following points:

• $\mathrm{■}$

  The loop between planning, execution and monitoring should be closed.

• $\mathrm{■}$

  Anonymization (i.e. adding noise so parameters cannot be reversed-engineered) is something to be investigated.

• $\mathrm{■}$

  In foundation models, standardization and data availability are main issues.

• $\mathrm{■}$

  Existing foundation models could be used for on-board lossless compression or compression with loss (select relevant data to downlink).

• $\mathrm{■}$

  In anomaly detection, the end-user should be involved in the designing of the tool. The use of of AI should be planned from the early stages of mission development.

• $\mathrm{■}$

  In diagnostics, smart filtering, dependency analysis, and XAI are used, but not yet causality. Causal Inference remains a challenge in space operations. It will be very useful for diagnosing anomalies and increasing understanding.

The discussion of the working group 4 can be summarized as follows. Descriptions of the different types of data collected by different EO missions are needed. This would facilitate their use, reuse and combination. This is especially important for applying ML methods to these data and combinations thereof.

• $\mathrm{■}$

  Ground-truth data is sorely needed. The same holds for meta-data describing the ground-truth data, which is even more important, because in some cases the data might not be available. Meta-data are essential for finding data relevant to a problem at hand. In addition, it would be very important for transfer learning and learning foundation models. The decision on which foundation model is most relevant for a particular downstream task can be also taken much more competently if we have a description of the data at hand (geographical region, type of urban system/ ecosystem, type of data).

• $\mathrm{■}$

  Reuse of historical heterogeneous data (sensors and calibration data) in order to make value of that information for nowcasting/ forecasting is of primary importance and AI can provide methodologies and techniques for such “transfer”.

• $\mathrm{■}$

  Hindcasting is an interesting avenue for further work. If we have, e.g., both Landsat and LIDAR, for a recent period, we could learn to map forest cover (height and density) from Landsat. We could then get estimates of forest cover for the entire historical period (60 years), where we have Landsat data.

• $\mathrm{■}$

  Incrementally/Continually updating machine learning models with new EO and calibration data in order to avoid retraining the system from scratch is also a possible challenge. This is especially important in the context of EO missions that are acquiring systematically new information on the different areas of the Earth surface.

• $\mathrm{■}$

  Federated learning holds significant potential for EO applications. The decentralized training methodology would enable us to learn from massive amounts of data without the need of centralizing it. This aspect is particularly advantageous in EO, where data can be voluminous, diverse, and often sensitive w.r.t. privacy and security. Initial work on EO federated learning can be found in https://arxiv.org/pdf/2311.06141.pdf.

  Federated and transfer learning can be used to avoid issues with sharing data. When data cannot be shared, but models can, models can be sequentially (pre)trained and (fine)tuned. The data can stay in place, but models can move around and evolve.

• $\mathrm{■}$

  Semantic resources for EO need to be developed, e.g. controlled vocabularies/ ontologies for describing EO-related data, as well as EO-related machine learning tasks, to match to methods.

• $\mathrm{■}$

  Open publicly funded centre for AI. There is a high risk that one of the big tech companies creates a monopoly in the field of AI4EO and particularly by the development of very large foundation models. To avoid the risks associated with commercial imperatives that might drive such an approach, publicly funded foundation models are needed.