GEMMA-FD: Zero-Shot Fault Detection in Heat Pumps Using Multimodal Language Models

We use multivariate time series data logged from an Austrian heat pump system. Each row in the CSV logs represents a 1-minute snapshot of multiple synchronized sensor readings. The dataset contains operational data of the heat pump system (AnlageA_JKj), annotated into four categories: betrieb_ok (normal operation), defrosting_issue, driver_temp_error, and overheating_control_issue. The dataset contains in total 130,351 samples. For model training and evaluation we split the data into training (70%, 91,245 samples), validation (15%, 19,553), and test (15%, 19,553) sets using a stratified sampling strategy. We further follow two different data pre-processing paths:

• $\mathrm{■}$

  XGBoost path: Each row is treated as an individual sample for supervised training and testing.

• $\mathrm{■}$

  GEMMA-3 path: Data is segmented into overlapping 4-hour windows. Each window is converted into a textual prompt and three visualizations.

All samples are transformed into structured natural language prompts (see Section 3.2). For GEMMA-3, the prompt is enriched with composite images (heatmap, time series, and histogram) to support multimodal reasoning.

The GEMMA-FD pipeline processes multivariate time series data logged from a real-world heat pump system. Sensor readings are stored as CSV files, where each row represents a 1-minute interval of synchronized measurements across multiple physical variables (e.g., temperatures, flow rates, compressor speed). We implement two processing pathways:

• $\mathrm{■}$

  A row-wise path for supervised learning (XGBoost) that treats each minute independently.

• $\mathrm{■}$

  A windowed path for zero-shot prompting (GEMMA-3) that aggregates sensor readings over a 4-hour sliding window to provide temporal context.

As illustrated in Algorithm 1 each sample (row or window) is transformed into a structured textual prompt. For GEMMA-3, additional multimodal visual summaries (heatmap, time series, histogram) are also generated and jointly passed to the model.

To enable large language models (LLMs) to reason about system state, we convert raw heat pump sensor data into structured textual summaries, using a sliding-window approach. Each window corresponds to a fixed temporal segment (e.g., 4 hours) of multivariate time-series data, which is then transformed into a prompt as follows:

• $\mathrm{■}$

  Sensor statistics: For each relevant sensor (e.g., compressor speed, flow rate, inlet/outlet temperatures), we compute the mean, minimum, and maximum over the window. These are presented in natural language.

• $\mathrm{■}$

  Trend detection: For selected sensors, we detect increasing or decreasing trends using the first and last sample in the window.

• $\mathrm{■}$

  Operating mode inference: If available, the categorical “wp-app-state_38” field is used to infer the dominant operational mode (e.g., heating, standby, defrost).

• $\mathrm{■}$

  Visual summary (multimodal input): A composite image is generated per window containing: (i) a z-score normalized heatmap, (ii) raw time-series plots, and (iii) sensor histograms. This image is passed to the LLM alongside the text.

• $\mathrm{■}$

  Instructions for classification: We append an instruction block asking the LLM to jointly analyze the image and text and classify the window into one of four predefined fault categories.

These design choices aim to emulate how human experts summarize system behavior and enable the LLM to leverage both symbolic and visual cues for anomaly detection. An example prompt is shown below:

Listing 1 presents a representative user prompt automatically generated using lightweight Python functions, while Listing 2 shows the corresponding system prompt provided to the model. This fully automated setup eliminates the need for manual annotation and enables scalable, LLM-based analysis of large, unlabeled datasets.

We use a locally hosted, open-weight multimodal language model gemma3:4b, available through the Ollama framework. This variant supports both text and image input and is based on the GEMMA-3 architecture released by Google DeepMind in 2025. Table 1 summarizes key model characteristics.

All predictions were performed using zero-shot prompting, without any fine-tuning or few-shot examples. Each prompt-image pair is sent independently to the model. The outputs are parsed to extract fault labels and free-text explanations.

We select gemma3:4b, an open-weight, multimodal LLM released by Google DeepMind [13] and served via the Ollama framework, to maximize reproducibility and local deployment feasibility. This 4B-parameter model supports both text and image inputs and is well-suited for vision-language inference tasks in constrained environments.

All experiments are conducted locally, with each sample consisting of a structured textual prompt and three visualizations as described in Section 3.2. The model receives a diagnostic system prompt specifying the output format; only the predicted label from the report function call is used for evaluation, parsed via regular expressions.

Evaluation is performed on a stratified sample of 300 test prompts. This setup reflects the label-scarce, zero-shot conditions of real-world fault detection. Our method is inspired in part by VisualTimeAnomaly [15], which highlights the value of multimodal visualizations for time series anomaly detection, and by Alnegheimish et al. [1], who emphasize the role of structured prompt engineering for LLM-based time series analysis.

To enrich the LLM’s context, each inference window is described by a structured natural language prompt and three visualizations: a z-score normalized heatmap of all sensor signals (Figure 2a), a time series plot (Figure 2b), and histograms of value distributions for each sensor (Figure 2c). All visual inputs are directly derived from real-world heat pump sensor data collected in Austria. The axis labels and variable names in the generated plots appear in German, as they reflect the naming conventions of the original control system and are passed to GEMMA-3 without translation to preserve semantic fidelity. These visuals capture temporal and statistical patterns, supporting robust zero-shot anomaly and fault detection.

For the zero-shot fault detection experiments, we evaluated GEMMA-3 using a stratified random sample of 300 test windows. To ensure class balance in the evaluation set, we performed stratified sampling across all annotated system states. Each sample consisted of a structured prompt and multimodal visualizations, as described previously.

The prediction workflow is as follows:

1. 1.

   Stratify and sample $N=300$ test windows from the full labeled dataset, preserving class proportions.

2. 2.

   For each sampled window, generate a structured prompt and corresponding visualizations.

3. 3.

   Run the llm_predict function, which queries GEMMA-3 via the Ollama inference framework with each prompt-image pair.

4. 4.

   Collect and store all raw LLM outputs for further analysis and reproducibility.

5. 5.

   Parse predicted fault labels from the structured LLM output using regular expressions.

6. 6.

   Save all predictions and explanations to CSV and TXT files for downstream analysis.

7. 7.

   Compute classification metrics (precision, recall, F1-score) and confusion matrix on the held-out samples.

8. 8.

   Export results as LaTeX tables and plots for inclusion in the paper.

All LLM experiments are conducted with the model running locally to ensure full reproducibility and data privacy. We restrict the evaluation to 300 test samples per run to manage computational demands and facilitate interpretability of individual LLM predictions.

For the supervised baseline, we train an XGBoost classifier using the full row-level dataset, where each row corresponds to a sensor snapshot and is labeled with one of the four classes: betrieb_ok, defrosting_issue, driver_temp_error, or overheating_control_issue. Individual CSV files for each class are parsed and transformed into prompt-label pairs using domain-informed prompt generation. After concatenating the class-specific data, we split the combined dataset into training, validation, and test partitions. To address class imbalance, the training set is balanced using random oversampling. Each prompt is then vectorized using term frequency-inverse document frequency (TF–IDF) features, allowing up to 2,000 unigrams and bigrams.

The XGBoost classifier (objective="multi:softprob") is trained on the balanced training data with hyperparameters set to 200 trees, a learning rate of 0.05, and a maximum tree depth of 6. After training, the model is evaluated on the held-out test partition, with classification metrics and confusion matrices generated to assess performance. This process is summarized in the following steps:

1. 1.

   Load and concatenate row-level CSV data for each fault class.

2. 2.

   Generate structured prompt-label pairs per sample.

3. 3.

   Stratify into train/validation/test splits.

4. 4.

   Apply random oversampling to balance the training set.

5. 5.

   Vectorize prompts with TF–IDF.

6. 6.

   Train the XGBoost classifier and evaluate on the test set.

7. 7.

   Report class-wise precision, recall, F1-score, and confusion matrix.

This workflow describes our supervised baseline for comparison with zero-shot LLM-based fault detection.

We report precision, recall, and F1-score for each class, as well as macro- and weighted averages to account for class imbalance and fault rarity, following Seliya et al. [10]. Confusion matrices visualize misclassifications.

GEMMA-3, evaluated on a stratified random sample of 300 windowed test prompts, achieved a macro-F1 score of 0.19 and a weighted F1 of 0.39 (Table 2), indicating reliable detection of the dominant class but limited sensitivity to rare faults. XGBoost, in contrast, reached a macro-F1 of 0.69 and a weighted F1 of 0.83 on the full test set (19,553 samples; Table 4), demonstrating robust, balanced performance across all fault categories. These results underscore the advantage of supervised learning with labeled data for comprehensive fault detection, while highlighting the generalization challenge faced by zero-shot LLM-based approaches.

We observe that GEMMA-3 achieves its highest recall on the normal class (betrieb_ok), suggesting that it captures expected system behavior more reliably than anomalous regimes, consistent with prior findings on LLM robustness to typical inputs [1].

A detailed examination of class-wise metrics reveals that GEMMA-3 achieves its highest recall for the majority class, betrieb_ok, but performs poorly on the minority fault classes. Specifically, GEMMA-3 attains a recall of 0.345 for betrieb_ok, while recall drops to zero for defrosting_issue and to 0.056 for driver_temp_error. The model achieves a moderate recall of 0.500 for overheating_control_issue, but with low precision. In contrast, the supervised XGBoost model shows high recall across all classes, including 0.818 for defrosting_issue, 0.934 for driver_temp_error, and 0.936 for overheating_control_ issue. This demonstrates that XGBoost not only identifies the majority class but also maintains strong detection performance for rare faults.

Analysis of the confusion matrix for GEMMA-3 (Figure 3(b)) shows that the model predominantly assigns samples to the majority class, betrieb_ok, resulting in frequent misclassification of fault cases as normal operation. Both defrosting_issue and driver_temp_error are rarely, if ever, correctly identified, with most of these cases assigned to the normal class. For overheating_control_issue, GEMMA-3 correctly identifies some instances but also produces a high number of false positives. In contrast, the confusion matrix for XGBoost (Figure 3(a)) is dominated by high values along the diagonal, indicating correct classification for all classes and minimal confusion between normal and fault types.

To further clarify the differences between approaches, we present a direct comparison of zero-shot LLM classification with GEMMA-3 and supervised learning with XGBoost. Both models are evaluated on identical input data and assessed using standard classification metrics. The following subsections provide detailed results for each method.

A detailed examination of the GEMMA-3 results (Tables 2 and 3) reveal three dominant error patterns:

• $\mathrm{■}$

  Majority class bias: GEMMA-3 disproportionately predicts the betrieb_ok (normal operation) class, even for samples labeled with faults. This leads to very low recall for rare fault types such as defrosting_issue (0.000) and driver_temp_error (0.056), reflecting a strong bias toward the majority class. This behavior is clearly visible in the confusion matrix (Figure 3(b)).

• $\mathrm{■}$

  Limited fault-specific detection: Fault samples are rarely classified with the correct fault type, even when statistical summaries contain values outside expected ranges. This suggests that GEMMA-3, operating under zero-shot conditions, struggles to associate specific numerical patterns with fault semantics.

• $\mathrm{■}$

  Overprediction of certain faults: For overheating_control_issue, the model exhibits moderate recall (0.500) but low precision (0.148). This pattern suggests that the model often predicts this fault incorrectly (false positives), indicating a tendency to overtrigger this class rather than true ambiguity at the decision boundary.

In contrast, the XGBoost baseline achieves high recall and precision for all classes, including minority faults (Table 4). Its confusion matrix (Figure 3(a)) is dominated by correct predictions, with minimal misclassification between fault types. This contrast highlights the challenge of zero-shot LLM-based fault detection – namely, that generalization from unlabeled normal data is insufficient for robust fault classification in complex cyber-physical systems.

These findings underscore the need for future work on:

• $\mathrm{■}$

  Enhancing prompt design with explicit fault indicators and temporal patterns,

• $\mathrm{■}$

  Applying few-shot learning to expose models to representative fault cases,

• $\mathrm{■}$

  Leveraging visual modality inputs (e.g., plots) more effectively for multimodal reasoning.

A direct comparison between GEMMA-3 and XGBoost highlights the fundamental trade-off between label-free generalization and supervised precision in heat pump fault detection.

• $\mathrm{■}$

  XGBoost consistently outperforms GEMMA-3, particularly on rare faults such as defrosting_issue and driver_temp_error, achieving high F1-scores across all classes – even when trained on natural language prompts originally designed for LLMs. This underscores the effectiveness of supervised learning when annotated data is available, regardless of input format.

• $\mathrm{■}$

  GEMMA-3 enables fully label-free fault detection, operating in a zero-shot regime using structured prompts and visualizations. However, it struggles with class imbalance and underperforms on minority fault types, limiting its current practical utility.

• $\mathrm{■}$

  LLM-based diagnostics remain promising, especially for low-resource or rapidly evolving environments. Hybrid strategies that integrate LLMs with few-shot exemplars, domain knowledge, or statistical post-processing may help overcome zero-shot limitations.

These results emphasize that while foundation models enable flexible deployment without labeled data, they are not yet reliable for safety-critical diagnostics. Improving multimodal prompt design and combining statistical learning with LLM reasoning are key directions for future research.

Out current framework already incorporates multimodal prompts – combining structured text with visual representations such as heatmaps, time series, and histograms. Future work could focus on leveraging these features more effectively. Enhancing prompt engineering with explicit fault indicators and temporal patterns, as well as introducing few-shot learning with annotated fault cases, may help address the low recall for rare fault types. In addition, hybrid approaches that combine LLM-based reasoning with supervised or statistical models could further improve robustness and accuracy. These directions aim to bridge the remaining gap between zero-shot generalization and reliable fault detection in complex, real-world cyber-physical systems.

In summary, supervised learning with XGBoost yields balanced detection across all fault types, while GEMMA-3’s zero-shot, prompt-based classification is largely limited to majority-class detection and fails to generalize to rare faults. These findings highlight the fundamental trade-off between the flexibility of zero-shot LLMs and the reliability of supervised models, motivating the improvements discussed in the next section.