Containerized Spectral Analysis in a Federated Cloud-Edge Continuum: The AC3 Approach

The European AC3 project (Agile and Cognitive Cloud-edge Continuum management, https://ac3-project.eu/) aims to develop a novel architecture using a federated Cloud Edge Continuum infrastructure. Central to this effort is the Cloud Edge Continuum Computing Management (CECCM) that guides both the resources and the applications Life Cycle Management (LCM). Another important point of AC3 is that it includes the implementation of Artificial Intelligence and Machine Learning technologies to ensure an energy-efficient operation. The project consortium is composed of a total of 14 institutions from 9 European countries.

1. The aim of the AC3 project is to develop methods for run scientific applications across cloud and edge computing systems. In particular, Use Case 3 focuses on astronomy and deals with the computing needs of modern spectroscopic surveys, where very large sets of Integral Field Spectroscopy (IFS) data need to be processed.

1. The analysis of these datasets is difficult because: (i) they contain very large amounts of data, (ii) they require a lot of computing power, (iii) there is a need for effective tools to display the results, and (iv) the results must be carefully interpreted. To tackle these challenges, AC3 uses a modular architecture (see Figure 3) made of independent software components packaged in containers and distributed computing. This makes it easier to run the software and use the computing resources more efficiently. With tools that coordinate these containers (orchestration technologies), resources can be assigned depending on the amount of work, so large sets of data cubes can be processed in a scalable way.

Use Case 3 Goals:

•To demonstrate the CECCM’s capabilities to deploy and run astronomical software to process data cubes.

•To integrate scientific applications that will take advantage of hybrid cloud native infrastructures to optimize the computation process.

•To enable the whole astronomy community to accelerate the analysis of the novel data gathered from newer IFS instruments.

The implementation of Use Case 3 relies on a collection of containerized spectral analysis tools (named as "fitting code" in Figure 3). These tools include spectral synthesis codes designed to model observed galaxy spectra as combinations of Single Stellar Population (SSP) templates spanning a range of ages and metallicities. The fitting process also accounts for stellar kinematics, velocity broadening and instrumental effects, enabling derivation of the physical properties of stellar populations. The Cloud-Edge Continuum Computing Manager (CECCM) deploys and coordinates the application across the computing infrastructure. Figure 3 shows a simplified architecture diagram illustrating the integration of the Use Case 3 with the AC3 components. The main points to understand this diagram are discussed below:

Distributed processing workflow:

1. The microservices-based architecture (see components Cluster 1 and Cluster 2 in the bottom part of Figure 3) is organized around two main components: Producer and Consumer. The producer receives uploaded datasets (S3 Storage), prepares the analysis tasks and distributes them among the computing resources using RabbitMQ. The consumer (worker pods) run the scientific analysis using containerized versions of astronomical software (i.e, fitting code such as STARLIGHT, pPXF and STECKMAP). Because each processor works independently, many spectra can be analysed simultaneously, reducing the execution time for large datasets.

Parallel analysis of IFU datacubes:

1. The system is designed to process IFU data by decomposing each data cube into its individual spectra and distributing the analysis in parallel. This strategy makes efficient use of the computing resources. The approach is well suited to modern surveys that produce millions of spectra requiring stellar population analysis.

Scalable architecture:

1. A major advantage of the AC3 framework is its ability to adapt automatically to the computational workload. When the number of tasks increases, additional processing units can be deployed to maintain high throughput. Tests performed within the project demonstrate that this strategy can significantly reduce processing times, highlighting its potential for large-scale astronomical applications. Because each worker can execute any analysis task, the system can easily scale by launching additional workers, providing a flexible way to process very large spectroscopic datasets.

User interface and visualization:

1. The platform includes an intuitive graphical interface that enables astronomers to: (i) upload datasets, (ii) launch analyses, (iii) monitor job progress and (iv) retrieve results without interacting directly with the underlying computing infrastructure (see Figure 4).

The Graphical User Interface (GUI) allows astronomers to manage and visualize the data in a user-friendly environment. It also makes distributed processing accessible without requiring expertise in cloud computing or system administration.

1. Users can upload data files, organize datasets and launch spectral analyses. A progress panel displays the status of ongoing jobs, allowing users to track the execution in real time (see top panel of Figure 4).

1. The interface provides an interactive way to visualize the processed data. The GUI allows users to explore processed datasets through integration with Aladin Sky Atlas. For each object, different analysis products, such as stellar velocity maps (as shown in the bottom panel of Figure 4), can be visualized.

1. By combining distributed computing with spectral analysis tools, AC3 Use Case 3 provides a scalable framework for processing IFU datasets.

1. The platform accelerates analyses while allowing astronomers to focus on the scientific interpretation of stellar populations and galaxy evolution rather than on computational resource management.

Authors acknowledge the support of AC3, a project funded by the European Union’s Horizon Europe Research and Innovation programme under grant agreement No 101093129.