Inverse Biophysical Modeling and Machine Learning in Personalized Oncology

Tumour invasion and progression may be viewed as collective phenomena emerging from the interplay of biological cells with their environment. Cell-based mathematical models in which cells are regarded as separate discrete entities can be used to decipher the rules of interaction. Here, we focus on the dynamics of glioma and breast cancer. We introduce lattice-gas cellular automaton models [1, 5] to analyse the role of phenotypic plasticity in cancer invasion, define spatial and non-spatial Moran processes to shed light on the size of the tumour originating niche, and adopt Markov chain models to investigate the origin of genetic heterogeneity in glioblastoma [2, 3, 4].

Inverse problems are omnipresent in any imaging related field and is as such a backbone in oncology, too. Here we focussed on the connections of machine learning to the particular inverse problem of image reconstruction but many concepts generalise to other inverse problems such as estimating parameters in PDEs. Solving inverse problems can be approached via variational regularization techniques which are dominant in the field of inverse problems in general. A drawback of these techniques is that they are dependent on a number of parameters which have to be set by the user. This issue can be approached by machine learning where we estimate these parameters from data. This is known as “Bilevel Learning” and has been successfully applied to many tasks, some as small-dimensional as learning a regularization parameter, others as high-dimensional as learning a sampling pattern in MRI. While mathematically appealing this strategy leads to a nested optimization problem which is computationally difficult to handle. We discussed several applications of bilevel learning for imaging [2, 1] as well as new computational approaches [1, 3].

Recent advances in medical imaging, coupled with the analysis capabilities offered by artificial intelligence, have led to significant progress in personalized oncology. Advanced MRI sequences in neuroimaging are now able to provide critical biophysical parameters for the study of tumor growth, response to therapies, and clinical decision-making. Furthermore, the integration of multi-parametric information, which would be otherwise infeasible, is now made possible through artificial intelligence. This presentation will introduce the collaborative efforts between the Biomedical Data Science Lab (Universitat Politècnica de València, UPV) and the MRI research and technology (Oslo University Hospital, OUH) to combine these two disciplines and make a real impact on clinical practice, particularly on high-grade glial tumors.

The OUH is improving its MRI protocol for neuro-oncology studies by incorporating advanced MRI sequences, such as Vessel Caliber MRI, Vessel Architectural Imaging, and MR Elastography. These sequences offer valuable information at the voxel level, such as vessel caliber size and density [1], vessel type dominance and microvascular efficiency [2], and tissue biomechanics by stiffness and viscosity [3, 8]. This enables researchers to gather a wider range of information on the brain’s blood vessels and tissue, providing a more comprehensive understanding of neuro-oncology.

To integrate all of the information gathered through advanced MRI sequences, processing pipelines and multi-parametric artificial intelligence models are being developed. The collaboration between the Oslo University Hospital (OUH) and the Universitat Politècnica de València (UPV) has led to the creation of AI systems that can accurately segment regions of interest [4], identify functional habitats [5], and analyze longitudinal series and growth dynamics [6], among others. An example of such a system is the publicly available ONCOhabitats platform developed by the UPV, which studies vascular heterogeneity in patients with high-grade glial tumors [7].

The success of these AI technologies in clinical practice depends on their integration into a relevant environment at the moment of decision-making. To achieve this, OUH’s models and associated pipelines are being integrated into a computation framework connected with the hospital PACS through the TrackGrowth, Chronos, and Progress research projects (see Acknowledgements). This setup allows for the direct evaluation of AI-based solutions in PACS by deploying hospital-approved software in the hospital interface.

Radiotherapy (RT) is a foundational component of clinical management for high-grade glioma (HGG) used to target residual and infiltrative disease following surgical resection. Variability in patient response to radiotherapy can depend on the tumor’s underlying sensitivity to treatment as well as the ability to accurately target the biologically relevant malignant tissue. To improve patient outcomes, RT treatment plans could be adapted for individual patients to target tumor sub-regions demonstrating treatment resilience and higher aggressive potential. Towards this goal, we developed a family of biologically-based mathematical models of HGG growth and response, which are initialized and calibrated using patient-specific multi-parametric magnetic resonance imaging (mpMRI) data [1, 2]. Our family of models is built upon a 3D, two-species model of enhancing and non-enhancing tumor that describes tumor cell proliferation, diffusion, and treatment response. Unique to our approach is the use of mpMRI collected weekly during RT which reports on both tumor extent and cellularity dynamics. Using patient imaging data collected before treatment onset and weekly up to mid-treatment, we identified patient-specific tumor growth and response parameters via a non-linear least squares optimization. These patient-specific model parameters were then used to forecast tumor growth and response dynamics at the remaining weekly imaging visits during RT. In an initial cohort of 13 patients, we observed that our computational framework was able to predict total tumor cell count with a Pearson correlation coefficient of 0.95 and concordance correlation coefficient of 0.91 at 1-month post-RT. Likewise, the forecasted total tumor volume agreed spatially with the observed tumor volume with Dice similarity coefficients greater than 0.73. At the individual voxel-level, the forecasted distribution of tumor growth was able to predict areas of significant increases or decreases in tumor cell with an accuracy, specificity, and sensitivity greater than 0.76. The results of this initial study demonstrates the ability for image-driven modeling to predict HGG response to RT that with further development may enable anticipatory adaption of RT.

Mathematical models that are used in science and engineering often need to be calibrated with respect to observational data. In the context of tumour modelling, for instance, image data can be used to estimate chemotaxis, consumption, and proliferation of a tumour [1]. Such parameter estimation problems are often referred to as “inverse problems”. Due to observational noise and complexity of models, inverse problems are usually difficult to solve and also ill-posed: a well-posed problem on the opposite is one, that has a solution, the solution is unique, and the solution depends continuously on the data. Well-posedness is important. Without existence, the problem has no solution and is, thus, not solvable. Uniqueness is required to prevent ambiguity between different solutions. The continuity assumption is a stability condition: the data is noisy, thus, we should hope that the influence of the noise on the parameter estimate is restricted in a certain sense.

The Bayesian approach to inverse problems gives a way to turn an ill-posed inverse problem into a well-posed problem. Here, we consider the calibration problem to be a statistical problem and model noise and unknown parameter as random variables. Through conditioning we are then able to incorporate the information from the data into the parameter. The conditioning can be achieved through Bayes’ formula.

As shown in [2], the resulting “Bayesian inverse problem” will be well-posed under very, very mild assumptions, allowing for parameter estimation in blackbox models and, e.g., with respect to data-driven prior models.

Brain tumour comprises a spectrum of malignant and benign entities. The complex pathophysiology of brain tumours poses challenges to effective clinical decision-making and treatment for patients. Multi-modal neuroimaging provides a non-invasive technique for probing brain tumours [5, 3, 13]. Based on neuroimaging, artificial intelligence (AI) offers an automated solution to optimise patient management, promising to accelerate precision neuro-oncology. Typically, the clinical applications of AI include tools for automatic diagnostics and guiding precise treatment. Together, these AI models promise to improve the overall efficiency of healthcare. Through engaging clinical domain knowledge, AI models can be tailored to the critical challenges in neuro-oncology, which could further advance our understanding of brain tumours and accelerate individualised and precise therapeutics.

Glioma is the most common malignant brain tumour in adults, characterised by remarkable heterogeneity and extensive invasion. To characterize tumour heterogeneity based on imaging, we designed novel radiological features to characterize tumour morphology and spatial heterogeneity [12]. Combined with machine learning methods, these features show robust performance in subtyping patients across diverse tissues and imaging modalities. The identified patient sub-groups show distinct molecular characteristics and prognostics. Advanced MRI techniques, e.g., perfusion and diffusion MRIs [4, 6], provide sensitive information for characterising tumour invasion over contrast-enhanced MRI. However, advanced MRI are typically in low resolution, which hinders full training labels for developing supervised models. To mitigate this challenge, we develop weakly supervised deep learning models that can identify the tumour invasion outside of contrast enhancement [2]. Further, glioma is considered a systematic disease, as it frequently spreads along white matter tracts into the whole brain. To characterize the tumour invasion globally, we developed an iterative tract-based spatial statistics method to quantify the structural connectivity of the brain and measure tract integrity in brain tumour patients [11]. Through comparing patients to healthy controls, we identified regional disrupted connectome in glioblastoma patients, which shows significance in predicting patient survival and indicating treatment targets [10]. Following this study, we introduced brain connectome into the AI model to better characterise glioma. Specifcially, we developed a multi-modal learning model, which leverages three encoders to extract features of focal tumour image, tumour geometrics and global brain network in predicting the isocitrate dehydrogenase (IDH) mutation, achieving higher performance over other state-of-the-art models [9].

In translating AI models into real-world practice, we need to tackle the challenges from heterogeneous clinical datasets, e.g., missing scans, and low image resolution. Therefore, we develop AI approaches to enhance image quality and standardisation [7, 1, 8]. For a trustworthy AI solution, we develop biophysics-informed deep learning models to enhance model explainability and generalisability. With these AI prototypes developed, we test the models in the real-world clinical setting, by connecting model development with the clinical system to obtain clinical and biological validations. We develop multi-centre imaging trials to validate the efficacy of imaging tools, where MR images are processed using reproducible and transparent pipelines. In the next step, we will test the imaging tools at scale through connections to large population data. Our vision is to transform the healthcare of brain tumour patients using image-based AI models.

Active surveillance (AS) is a feasible management option for low to intermediate-risk prostate cancer (PCa), which represents almost 70% of newly-diagnosed cases. During AS, patients have their tumor monitored via multiparametric magnetic resonance imaging (mpMRI), serum prostate-specific antigen (PSA), and biopsies [1]. If any of these data reveal tumor progression towards an increased clinical risk, the patient is prescribed a curative treatment. However, clinical decision-making in AS is usually guided by population-based protocols that do not account for the unique, heterogenous nature of each patient’s tumor. This limitation complicates the personalization of monitoring plans and the early detection of tumor progression, which constitute two unresolved problems in AS. To address these issues, we propose to forecast PCa growth using personalized simulations of an mpMRI-informed mechanistic model solved over the 3D anatomy of the patient’s prostate [1, 2, 3]. We describe PCa growth via the dynamics of tumor cell density with a diffusion operator, representing tumor cell mobility, and a logistic reaction term, which accounts for tumor cell net proliferation [1, 2]. Model calibration and validation rely on assessing the mismatch between model predictions of the tumor cell density map with respect to corresponding mpMRI-based estimates [2]. Our preliminary results on a cohort of seven patients show a median concordance correlation coefficient (CCC) and Dice score (DSC) of 0.55 and 0.82, respectively, for the spatial fit of tumor cell density during model calibration using two mpMRI datasets. Then, model validation at the date of a third mpMRI scan resulted in median CCC and DSC of 0.33 and 0.76, respectively. Thus, while further improvement and testing in larger cohorts are required, we believe that our results are promising for the potential use of our methods to personalize AS protocols and predict tumor progression.

We present a framework for diffeomorphic image registration termed CLAIRE [1, 6]. This algorithm is an integral part of some of our efforts to develop algorithms for the analysis of brain tumor imaging data [4, 5, 7, 8, 9]. Diffeomorphic image registration is a non-linear, ill-posed inverse problem that poses significant mathematical and computational challenges. Generally speaking, we seek a ${ℝ}^d$-diffeomorphism $y\in \text{diff}({ℝ}^d)$, $d\in \{2,3\}$ that establishes a point-wise spatial correspondence between two views (images) of the same scene. In our work, we consider a variational formulation governed by hyperbolic transport equations.

Our contributions are new algorithms and dedicated computational kernels to reduce the runtime. We study the performance of our solver in terms of rate of convergence, registration accuracy, and time-to-solution. We demonstrate that we can solve problems for clinically relevant data of sizes (${256}^3$ voxels; $\sim$50 million unknowns) in under 5 seconds (see table below). Our formulation and numerical algorithms are described in [6, 10, 11, 13]. Our parallel CPU implementation is discussed in [6, 12]. Our parallel GPU implementation is presented in [2, 3]. The integration of our registration algorithm with models of tumor progression is presented in [4, 5, 7, 8, 9]. We overview the computational performance of our framework for diffeomorphic image registration for an image of size ${256}^3$ in the table below. Compared to our CPU implementation we observe a speedup between 28$\times$ and 71$\times$ depending on the GPU and implementation (CLAIRE: standard implementation; CLAIRE^∗: additional intermediate variables kept in memory). We report from left to right the version of CLAIRE, the hardware CLAIRE is executed on, the memory use, the mismatch between the data after registration, the runtime in seconds and the speedup. We can see that the GPU implementation is significantly faster than our GPU implementation without sacrificing accuracy.

Artificial Intelligence approaches, and especially recent Deep Learning techniques, have shown to outperform conventional image processing algorithms in many medical image analysis scenarios.

This presentation will present Deep Learning approaches for Diffusion MRI Data for (1) diffusion signal augmentation [1], (2) free water correction [6, 2, 4] and (3) signal harmonization [5, 6, 7].

Finally, limitations of neuronal networks as well as current challenges and trends in Deep Learning will be discussed.

Real-world applicability of artificial intelligence (AI) in the clinical setting [39, 40, 41] is hampered by the i) lack of available diverse (training and validation) data affecting the robustness and generalizability of AI models towards unseen/unknown population groups, and ii) limitations on defining reproducible computational pipelines for local hardware resources at clinical sites.

The current paradigm towards sufficiently large and diverse data for training and validating AI models is via centralization of data from multiple institutions [29, 30, 32, 33, 31, 17, 6, 49, 50, 45, 46, 47, 48, 51]. However, this paradigm faces limitations when it comes to scale due to various legal, regulatory, cultural, and ownership concerns [8, 9]. Federated Learning (FL) offers an alternative paradigm to train robust AI models and a potential solution to the data sharing hurdles, as demonstrated in multiple simulated [8, 9, 2, 43] and real-world studies [1]. Furthermore, beyond training robust AI models, the evaluation of their effectiveness and durability over time on real-world patient data from large and diverse population demographics poses another challenge towards their clinical translation. Federated evaluation (FE) studies through persistent data registries and streamlined workflows may provide a solution on such performance evaluations, obviating the need of data sharing. Together, federated learning and evaluation form complementary mechanisms to generate meaningful clinical impact by enabling access to data silos in a way that is compatible with regulations and cultural concerns.

There have been numerous community-driven efforts to provide either common definitions towards results’ reproducibility [13, 14, 16, 29, 30, 32, 33, 31, 17, 49, 50, 45, 46, 47, 48, 51], or common benchmarking environments (i.e., challenges) for fair AI model evaluation [15]. Although a substantial number of closed-source and commercial solutions achieve clinical reproducibility [42], having widely available, community-driven, and well-documented open-source projects [18, 19, 20, 21, 22, 23, 34, 7] that focus on the reproducibility of research, while being driven by the clinical stakeholders would be critical towards ensuring that cutting edge scientific breakthroughs make it for clinical validation sooner. This further allows computational scientists to explore their methodological interests while allowing clinical partners to deploy these methods in an easy manner in their existing hardware infrastructure.

Our collaborative group has collectively produced open-source publicly-available software solutions to address this space. Starting with the largest real-world FL study to-date (the Federated Tumor Segmentation (FeTS) initiative)⁵⁵5www.fets.ai, which also describes the largest reported study on the rare cancer of glioblastoma, involving data of 6,314 patients from 71 institutions across 6 continents [1]. The tool used by the FeTS Initiative has been open-sourced as “The FeTS Tool” [4], which provides an end-to-end point solution for studies related to brain tumor boundary detection/segmentation. This solution includes all the required computational steps, starting from data curation, anonymization, brain extraction (also known as skull-stripping [35, 34]), to pre-processing, generation of baseline automated annotations leveraging methods considered state of the art [53, 54, 55], an interface to manually refine these automated annotations and sign off ground truth labels [18, 19, 20], as well as to allow a user to either train their AI model or join an existing FL study. Moreover, the FL component of the FeTS tool is enabled by the Open Federated Learning (OpenFL) library [11, 10], which is designed for general-purpose FL and being agnostic to use-case and framework. Further to the FeTS initiative, OpenFL has facilitated studies on the i) effect of cosmic radiation on astronauts by the Frontier Development Lab (FDL) of the National Aeronautics and Space Administration (NASA), and ii) prediction of respiratory distress syndrome and death for COVID-19 patients by the 11 sites of the Montefiore Health System in New York.

Building upon the collaborative network of the FeTS initiative, we further conducted the first-ever computational challenge in FL, which happened in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2021 and 2022 [5], and followed the principle of clinical trials [52]. The focus of the FeTS challenge was two-fold: i) the development of aggregation methods for FL, and ii) the federated evaluation of brain tumor segmentation algorithms in-the-wild, by circulating AI models on unseen data from multiple sites of the FeTS initiative collaborative network. The FeTS challenge was orchestrated by MedPerf [24], in which the challenge organizers initiated the design of the study, the collaborating sites registered information about their datasets, and the AI models of the challenge participants were described as independent experiments evaluated against these datasets. Finally, towards broader clinical workflows, we developed the Generally Nuanced Deep Learning Framework (GaNDLF) [12], which enables users to design and manage AI algorithms for multiple tasks and various data/organ/modality types, such as segmentation on brain tumor MRI [2, 1], breast mammograms [37, 36] & dynamic contrast enhanced MRI [6], as well as classification on histology whole slide images [3], RGB images [38], & breast mammograms [43]. The wide applicability and obtained results showcase the generalizability of GaNDLF. Additionally, GaNDLF offers automated post-training optimization of AI models [56, 44], allowing their execution/inference on consumer-grade computers without requiring specialized hardware, such as deep learning acceleration cards.

In conclusion, there is a need to i) assess the generalizability of AI models by capturing ample patient demographics, ii) address bias and inequities in AI, especially those related to underserved/underrepresented patient populations, and iii) on the continuous monitoring of AI models requiring further developments in automated quality control, monitoring of drift & bias, and model calibration. Towards fulfilling these goals, the open federated ecosystem consisting of GaNDLF [12], OpenFL [11], and MedPerf [24] provide a holistic end-to-end open-sourced federated learning and evaluation solution that supports multiple data types, and that be easily used by both experienced and novice researchers.

We present computational coupling of inverse tumor simulation and diffeomorphic image registration that allows to achieve two tasks that can be relevant for clinicians: (i) registration of a healthy statistical atlas brain to a patient brain with tumor in order to transfer labels and brain region boundaries; (ii) identification of tumor growth parameters such as diffusion and reaction rates or initial tumor. For both tasks, we have to solve a combined inverse problem involving image registration and the tumor model to ’move’ from an atlas image to a patient images with a tumor. We present various ways to achieve this by combining separate registration and tumor solvers in [1, 2]. More details on the single components are presented in [3] for the tumor growth inversion and in [4] for image registration. Both software components show very good scalability on high performance computing hardware such that we can solve problems at ${256}^3$ resolution in a couple of minutes.

For a glance at more general concepts for coupling of two or more computational components, refer to [5] and [1].

In my talk I will discuss the use of generative models for generalizable and interpretable analysis of brain tumor images. Specifically, I will highlight the fundamental differences that exist between analyzing tightly-standardized images in well-controlled group studies, vs. analyzing images acquired “in the wild”, i.e., as part of the clinical treatment of brain diseases. I will present our work on generative models that can naturally extrapolate beyond the narrow characteristics of manually labeled training data, and how these techniques are implemented within the well-known open-source software suite FreeSurfer. Specific attention will be paid to modeling lesions (such as white matter lesions or brain tumors) within whole-brain segmentation settings, and to leveraging the temporal consistency between follow-up scans in longitudinal data. Time permitting, I will also touch on the need for interpretable image prediction models, where the generative aspect encodes the causal effect of disease on brain shape. Such models are much easier to interpret and explain to clinicians than the “black box” discriminative methods that are often used for predicting diagnoses or disease scores from images.

The diffuse growth pattern of glioblastoma is one of the main challenges for improving patient survival. Computational tumor growth modeling has emerged as a promising tool to infer tumor cell distribution and thereby guide personalized therapy.

In [1], we performed clinical and biological validation of a novel, deep learning – based growth model [2], aiming to close the gap between the experimental state and clinical implementation. In more detail, we wanted to investigate how well this Fisher-Kolmogorov model correlates with (i) tumor biology, (ii) survival and – perhaps most importantly – (iii) location of tumor recurrence.

To answer these questions, we included a total of three data sets into our study. For (i) and (ii), we analysed 124 patients from The Cancer Genome Archive network and 397 patients from the UCSF glioma MRI data set for correlations between clinical data, genetic pathway activation maps (generated with PARADIGM; TCGA only), and infiltration (Dw) as well as proliferation ($\rho$) parameters stemming from a Fisher-Kolmogorov growth model adjusted to the patients’ preoperative images [2]. To address (iii), we correlated later tumor recurrence in an in-house data set with 30 glioblastoma patients with radiotherapy plans and growth model-derived tumor cell distribution.

Interestingly, we observed a significant correlation between 11 signaling pathways that are associated with proliferation, and the estimated proliferation parameter $\rho$. The parameter ratio Dw/$\rho$ (p<0.05 in TCGA) as well as the simulated tumor volume (p<0.05 in both TCGA and UCSF) were significantly inversely correlated with overall survival in Cox survival modeling. Depending on the cutoff value for tumor cell density, we observed a significant improvement of recurrence coverage without significantly increased radiation volume utilizing model-derived target volumes instead of standard radiation plans (example shown in figure 2).

Identifying a significant correlation between computed growth parameters, and clinical and biological data, we highlight the potential of tumor growth modeling for individualized therapy of glioblastoma. This holds promise to improve accuracy of personalized radiation planning in the near future. Future research directions include more complex growth models (e.g., including necrosis or mass effect), including imaging information for model calibration, and ultimately also going from global to local modeling, explicitly incorporating tumor heterogeneity.

Tumor simulations require complex multiscale models ranging from discrete agent-based models to continuum models, with various hybrid type models in between [1]. Extremely small scales require an agent-based formulation for the tumor and the capillaries, where only signaling molecules, drugs, and nutrients are described by continuous fields [2]. Larger tumors inside rat brains might be resolved with a continuous phase field approach, where still the capillary flow is described by 1D-models, and their growth is modeled by a rule-based algorithm [5]. On the macro scale, the capillaries might be further simplified to a porous medium, requiring only the resolution of larger vessels by 1D-models for breast tumors [7] or 2D surface sources in the lung [8]. Often a problem-dependent coupling is required to achieve a biologically meaningful value range, particularly for the pressure of the 1D blood flow where 0D models have to damp down oscillations.

Besides the choice between discrete and continuum approaches, there remains the question of which biophysical mechanisms are considered relevant for the application at hand and thus have to be modeled, often leading to increasingly complex models of various species. The tumor typically consists of necrotic, hypoxic, and proliferative cell species. For agent-based models, the latter might be further divided into the Q, G1, SG2 subspecies [2]. Further, matrix degenerative enzymes acting on the extra-cellular matrix might be added [3]. The nutrient field might be split up into various porous media resulting in double continuum models [7, 8]. For angiogenesis, vascular endothelial growth factors have to be included [2, 5]. Often mechanical deformations have to be considered [6], and, depending on the clinical therapy, one or more drug species have to be included [6, 8].

All these modeling choices lead to complex, heavily-coupled, nonlinear models, which pose mathematical challenges to the analysis of their well-posedness [3, 4], to the creation of stable numerical schemes and efficient decoupled solvers [5].

In a second, more applied and challenging step, these models have to be calibrated against real-world data [9] and verified against clinical measurements [8]. Here, the amount of data is often the bottleneck and requires a strong multidisciplinary effort to acquire, evaluate and interpret. Especially the derivation of generally accepted benchmark problems for model validation would be extremely valuable for future work.