High Performance Visualization with VisIVO Across Cloud and HPC

Astrophysical observations and simulation codes on high-performance supercomputers generate petabytes of data [22]. Managing this data – storing, retrieving, and analyzing it – presents major challenges critical to scientific progress [17]. Pre-exascale systems enable new ways to scale High-Performance Computing (HPC) for Astrophysics and Cosmology. Researchers now need both powerful computation and interactive methods to visualize results [7]. VisIVO¹¹1VisIVO, https://visivo.readthedocs.io/, the Visualization Interface for the Virtual Observatory is a suite of tools for high-performance, multidimensional data analysis and visualization in astrophysics [11]. It was first built for distributed systems like grids and clouds [5], and later updated for the European Open Science Cloud (EOSC) [20, 21] using containerized science gateways.

VisIVO supports interactive and portable workflows [19]. Most recently, it has been integrated with the cloud-oriented InterActive Computing (IAC²²2IAC, https://jupyter.g100.cineca.it/) service [4], which is available on the Galileo 100 cluster (G100³³3Galileo 100 infrastructure: https://www.hpc.cineca.it/systems/hardware/galileo100/) [9], at Cineca. This IAC service enables users to access compute nodes on demand via Jupyter notebooks in a web browser. The IAC service fundamentally transforms the conventional batch-based HPC access model into a fully interactive, web-accessible environment. This platform enables users to initiate notebook sessions directly on compute nodes equipped with GPUs and high-speed interconnects, thereby facilitating interactive data analysis, real-time visualization, and dynamic simulation control [3]. By enhancing usability and reducing technical barriers, IAC addresses critical challenges in HPC accessibility and user engagement. The integration of VisIVO with IAC further empowers researchers to conduct 3D visualization workflows seamlessly within the HPC ecosystem, eliminating the need for complex command-line configurations or manual data transfer. This integration extends the scalability and computational power of HPC to Jupyter-based environments, promoting reproducible, accessible, and user-centric scientific workflows.

We embed VisIVO within IAC via lightweight Python wrappers and a custom Jupyter kernel that automatically loads modules, configures environment variables, and transparently calls the underlying C++ binaries [22]. This design hides system complexity and delivers consistent execution across distributed nodes. Additionally, we have developed cloud-oriented RESTful API services that expand VisIVO’s capabilities beyond the notebook interface. Built with lightweight Python frameworks, these APIs enable remote execution of visualization tasks and provide programmatic and web-based access for large-scale data rendering and analysis. This work complements the interactive IAC integration by enhancing automation and laying the foundation for future cloud–HPC hybrid environments.

The paper is organized as follows: Section 2 provides an overview of related works in interactive and HPC-driven visualization; Section 3 introduces the VisIVO Server modules and its architecture; Section 4 introduces the InterActive Computing (IAC) service; Section 5 describes the methodology used to integrate VisIVO into the IAC framework, while Section 6 focuses on the development of REST-based visualization APIs; Conclusions and future developments are discussed in Section 7.

Scientific visualization has progressed markedly at scale – especially in astrophysics and cosmology – where petascale simulations and surveys demand real-time analysis and visual insight. Although existing tools and platforms provide a solid foundation, just a few let you do data analysis and visualization in real-time, in one reproducible HPC workflow. Early pioneers in large-scale HPC visualization, VisIt [8] and ParaView [1] established powerful frameworks for parallel and remote rendering across distributed systems. Although both support HPC-friendly client–server models, they typically demand expert configuration and do not offer native web integration, which constrains usability for many domain scientists.

Interactive web-based platforms have become popular through science gateways and cloud-centric systems. For instance, [6] gateways have been shown to provide researchers access to sophisticated astrophysical analysis tools like VisIVO through web portals. Following this, the NEANIAS Visualization Gateway [21] was developed, incorporating VisIVO into the European Open Science Cloud (EOSC) using containerized services and Jupyter interfaces. This work demonstrated that deploying advanced astrophysical visualization on federated cloud resources is feasible, ensuring reproducibility and cross-platform portability. Nevertheless, these gateways generally do not deeply integrate with HPC schedulers or support real-time session control, operating primarily in batch mode or with limited interactivity.

As Jupyter Notebooks have become standard tools for reproducible scientific workflows, various projects have investigated combining them with high-performance computing resources. The work of [16] introduced Jupyter as a user interface for scientific computing, and newer efforts, such as the Fenix infrastructure [4], have adapted this approach specifically for neuroscience and brain studies. These platforms demonstrated that HPC systems could be accessed via notebook-driven web portals, though they primarily focused on static data analysis rather than interactive, GPU-powered visualization.

Here, combining VisIVO with Cineca’s InterActive Computing (IAC) service, as demonstrated in this study, addresses an important shortfall. This method enables interactive 3D rendering on HPC nodes and packages VisIVO’s functionality within a specialized Jupyter kernel via Python wrappers, making it easier for users to access high-performance GPU resources. Unlike [21], this solution removes the necessity for manual module management, SSH file transfers, or command-line interface rendering. Additionally, it offers live visualization through VisIVO Viewer and supports complex processing workflows by using pre-configured, scriptable notebooks. Spack [10] and Ansible [13] are commonly used tools for automating deployments in HPC software environments, and their application here to merge Python and C++ tools into unified notebook setups highlights an advanced degree of composability. This approach converts traditional static command-line workflows into interactive, reproducible sessions, thereby enhancing both the reproducibility and accessibility of scientific visualization methods in astrophysics. This method has been applied specifically to astrophysical simulations such as GADGET and to workflows involving density visualization and the examination of cosmological structures [22], but it might be generalized to any command-line-based backend tool. Additionally, it enables the use of cloud-based RESTful services and serverless visualization backends, simplifying HPC complexity without sacrificing efficiency.

VisIVO Server is a collection of software tools built to produce customized 3D visualizations using astrophysical datasets, handling very large amounts of data with no restrictions on data dimensions [23]. The primary modules available via the command line are:

• $\mathrm{■}$

  VisIVO Importer: transforms user datasets into an efficient internal format called VisIVO Binary Tables (VBT), supporting a wide range of formats from common data types like ASCII and CSV to specialized astronomy formats such as FITS, HDF5, and GADGET.

• $\mathrm{■}$

  VisIVO Filter: allows users to process VBT files by applying data transformations, performing filtering, and other modifications needed before visualization.

• $\mathrm{■}$

  VisIVO Viewer: creates interactive 3D graphics from VBTs, enabling users to visually analyze and explore datasets efficiently.

A standard VisIVO Server workflow progresses through three stages – data preparation, processing, and visualization (as shown in Figure 1).

As the demands of modern scientific workflows increase, there is a need for capabilities beyond traditional batch-based processing. Fenix⁴⁴4Fenix, https://fenix-ri.eu/ addresses this need for interactive access to large-scale computing resources by providing a federated set of infrastructure services that complement traditional batch-oriented HPC systems [2]. Fenix introduces Interactive Computing Services (IAC) as a core constituent of its architecture. IAC enables users to access a persistent, low-latency interactive environment, typically via web-based interfaces such as Jupyter. IAC has been developed with a focus on supporting neuroscience work in the Human Brain Project (HBP⁵⁵5Human Brain Project, https://humanbrainproject.eu/) and European Brain Research INfrastructure (EBRAINS⁶⁶6EBRAINS, https://ebrains.eu/) communities. This service lets researchers interact with data and run applications directly, facilitating rapid feedback and accelerating scientific progress. The development and management of IAC services were a collaborative effort between Cineca and E4, using the ICE4HPC⁷⁷7ICE4HPC, https://www.e4company.com/en/ice4hpc/ software suite. Currently, IAC is available on Cineca’s Galileo 100⁸⁸8Galileo 100 infrastructure: https://www.hpc.cineca.it/systems/hardware/galileo100/ supercomputing platform [22].

An interactive, browser-based interface to HPC delivers two key benefits.

1. 1.

   Users can launch sessions directly on compute nodes at https://jupyter.g100.cineca.it with near-instant startup, bypassing the traditional SSH, batch-queue workflow and gaining seamless access to graphical visualization tools that are cumbersome in command-line-only environments.

2. 2.

   It enables real-time control of running workflows – monitoring resource usage [15], inspecting intermediate results, and adjusting parameters on the fly – avoiding the unpredictable start times and slow feedback cycles inherent to batch processing.

The Interactive Access and Computing (IAC) framework delivers a secure, end-to-end interactive HPC experience by routing user authentication and resource requests from a local browser to a cloud-hosted frontend that isolates the web surface, which then forwards requests to Slurm to allocate a compute node and launch a Jupyter server with a VisIVO-enabled Python kernel, as shown in the UML diagram of Figure 2. Users work continuously in notebooks on that node – running heavy computations directly on HPC resources while monitoring usage and visualizing intermediate results in real time. Jupyter is chosen for its single-user server–client model, rich plugin ecosystem (e.g., monitoring tools, featured dashboards, and interface with other services via jupyter-server-proxy), and broad user familiarity. Security is enforced by executing orchestration on a dedicated VM in Cineca’s cloud, running HPC operations as standard user-space Slurm jobs (no root privileges required on the cluster side), and applying two-factor authentication alongside continuous monitoring, vulnerability assessments, and penetration testing.

Integrating a dedicated VisIVO Jupyter kernel into the IAC replaces the old ssh-and-module-load workflow on G100, eliminating command-line setup and extra steps to fetch PNG outputs. Users now launch an IAC session, select “VisIVO,” and run commands immediately; outputs are saved to the session and viewable in the web interface. This streamlines visualization, improves accessibility, and reduces friction – even though VisIVO, written in C++, isn’t a native Jupyter target – details of which are described next.

A second target for our codebase is pipeline-style automation, which is essential for high-throughput science (e.g., weather forecasting, tsunami alerts, satellite data management), where multiple servers interact in cascades that eventually trigger HPC jobs. Because initiating these interactions depends on site-specific HPC access policies, no standard method exists; in our earlier IAC use case, we overcame this by using Cloud–HPC coupling, which allows us to expose web services typically disallowed on HPC clusters. Extending that idea, we propose embedding standard web REST APIs into HPC workflows to provide secure, scalable, and reproducible automation from submission to result retrieval. REST makes services client-agnostic, so requests can be sent from the command line (e.g., curl) or any web front end without server-side changes. Similar needs arise in projects like Terabit [24, 25] (rapid earthquake simulations) and GAIA [17] (post-processing large astronomical datasets), where pipelines coordinate data flows from remote sources (telescopes, seismic sensors) through interconnected steps but still lack a general approach – REST APIs are a strong candidate standard. To prototype this, we used our VisIVO codebases to simulate a pipeline that automatically post-processes simulation data using the cloud-hosted VisIVO REST API.

This study describes how the VisIVO scientific visualization framework was integrated into Cineca’s InterActive Computing (IAC) service to create an interactive, scalable, and reproducible visualization environment directly on HPC systems. The integration transformed VisIVO from a command-line, batch-based tool into a web-accessible, browser-driven platform that supports complex 3D visualizations within Jupyter notebooks using GPU nodes. Python wrappers and a dedicated Jupyter kernel make it easier for users to interact and run VisIVO commands manual configuration or module loading. These updates speed up visualization, provide clear HPC access, and help scientists analyze data more quickly. Cloud-based RESTful APIs built with Flask now let users run VisIVO remotely through web services, which simplifies the backend and makes it easier to connect with other applications. Overall, this paper showcases how conventional visualization software can be adapted into modern HPC-native solutions that combine high performance with user-friendly accessibility. The framework was tested on real astrophysics workloads, such as the ones from the SPACE project, and handled them smoothly, efficiently scaling and reliably visualizing very large, complex datasets. These trials demonstrated its ability to connect traditional batch supercomputing with real-time, interactive analysis.

In the future, the next step is to exploit the cloud-hosted REST API with VAST’s multi-protocol shared storage so the web front end can read cluster-generated data directly – no data movement – possibly via VAST’s object-storage interface for extra flexibility. The REST API will require proper authentication, likely token-based. Once shared storage is active, a REST workflow similar to IAC can submit VisIVO jobs to the cluster’s batch system, moving computation to the data. This should boost throughput and I/O efficiency while keeping security tight by not exposing any web server from the cluster.

During the preparation of this work, the authors made use of OpenAI’s ChatGPT and Google’s Gemini for paraphrasing, rewording, and checking grammar and spelling. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.