Alignment Sets for Sensor Fusion Against Temporal Misalignment

Alignment constraints specify the intended synchronization at the level of individual sensor samples. Misalignment is how far the actual timestamp of a sample deviates from the intended synchronization specified by the alignment constraint. An individual alignment constraint describes where the time point $t_s$ of a sample $s$ should ideally be, by specifying the relationship of $t_s$ to other sample timestamps or fixed time points. Given a timestamp tuple $T$, one can then quantify how far $t_s$ deviates from this ideal case as a residual.

We first provide examples of common alignment scenarios, based on the motivational example from Section 2, and observe that earlier examples are special cases of later examples. We provide a unifying formal definition, showing how it captures all these scenarios.

A single alignment constraint specifies the intended timing of one sensor sample. However, for more complex fusion functions, a single alignment constraint is often insufficient to capture the intended synchronization. Multiple constraints together are needed to describe the intended synchronization. In the example from Section 2, both the relationship (i) between the two camera samples, and (ii) between the camera samples and the LiDAR sample are relevant for synchronization. Alignment sets combine multiple alignment constraints into a specification of how all sensor samples involved in a fusion function should ideally be aligned.

Formally, we define alignment sets as finite sets of alignment constraints over samples $𝒮$.

We call an alignment set ordered if it has no cyclic dependencies between samples.

We provide two examples of alignment sets for the camera-LiDAR fusion scenario from Section 2. The exact alignment requirements depend on the specific application. Depending on the application’s requirements, it is therefore possible that multiple different alignment sets are reasonable for the same fusion function. A key question is whether the perfect alignment is anchored to specific time points or whether only their relative timing matters.

For an alignment set $𝒜$, we now explain how to evaluate the misalignment function and the robustness against temporal misalignment constrained by the alignment set. We start from a general definition of a misalignment function.

The function $g$ can be any statistical functions or extremum functions. We give a bound on the time complexity of evaluating this misalignment measure $M_{𝒜}(T)$.

Reference-point-based temporal robustness as given by Kuhse et al. [16] is based on a single reference time point $t^{\star }$ and considers misalignments of the sensor samples with respect to this reference point. That is, given a reference time point $t^{\star }$, analogous to the $ϵ$-$\mathrm{\Delta }$-temporal robustness in Definition 9, the sensor fusion function $f$ is considered robust against temporal misalignment for a misalignment threshold $\mathrm{\Delta }$ and error threshold $ϵ$ if and only if:

where $L$ is a loss function that measures the difference between the fused output at time $t$ and the fused output at the reference time point $t^{\star }$.

We note that this loss function $L$ is different from our error function $E$ in Definition 9: $L$ directly compares two fusion outputs, while $E$ quantifies the error of a fusion result given timestamps $T$. Our $E$ is more flexible, as shown in the proof of Lemma 10, their loss function can be embedded into our error function by defining $E(T,f(T))=L(f(t^{\star },\mathrm{\dots },t^{\star }),f(T))$. We chose to generalize the error function as reference-point-based robustness and sample-based robustness (Section 5.2) used $L$ differently, comparing against a fusion at a single reference point and against multiple fusions in a range of reference points, respectively. The error function $E$ allows both of these situations to be captured as special cases, while also allowing for more general error functions that do not necessarily compare against a reference point fusion output (e.g., an error function that directly quantifies the error of the fusion output without comparing it to a reference point fusion output).

We show that alignment set based robustness is a strict generalization via proof by construction, showing that this definition can be captured by an alignment set with appropriate constraints, misalignment measure, and error function.

The proof is straightforward and therefore given in Appendix A.

Sample-based temporal robustness as given by Kuhse et al. [16] assumes time stamps for each sensor sample as given, evaluating robustness by considering possible reference points inside the range spanned by the sensor samples.

Formally, given $t_1,\mathrm{\dots },t_n$ as the timestamps of the sensor samples, two different notions of sample-based temporal robustness are defined, differing in strictness. Strong sample-based temporal robustness requires that the fusion output is close to the output at every reference point $t^{\star }$ in the range spanned by the sensor timestamps, while weak sample-based temporal robustness only requires that the fusion output is close to the output at some reference point $t^{\star }$ in this range. Strong sample-based temporal robustness means that the fusion output is stable across the entire range of possible reference points to represent all of them, while weak definition only requires that it reflects some point in the range. Formally, strong sample-based temporal robustness is defined as:

while weak sample-based temporal robustness is defined as:

In both cases, $L$ is the same kind of loss function as in Section 5.1.

In the global notion of sample-based temporal robustness, these definitions must hold for all possible timestamp combinations $T=(t_1,\mathrm{\dots },t_n)$ where the misalignment between the timestamps is bounded by a threshold $\mathrm{\Delta }$.

Following the same approach as for the reference-point-based definition, we show that alignment set based robustness is a strict generalization of sample-based robustness by showing that both strong and weak sample-based robustness can be captured by an alignment set with appropriate constraints, misalignment measure, and error function.

The proof of this lemma is given in the Appendix A.

Our evaluation focuses on showing how alignment sets can more precisely capture the temporal misalignment of the camera-LiDAR scenario from Section 2. This in turn leads to robustness estimates with tighter error bounds. In this section, we describe the fusion pipeline, evaluation scenarios for a single fusion, dataset, and metrics. The full-drive setup (Section 6.5) reuses the same fusion pipeline but defines its own scenarios.

The camera-LiDAR fusion we consider is based on the fusion algorithm provided in Autoware Universe’s perception stack [14]. Camera images are processed by YOLOv9 [28] into 2D bounding boxes, while LiDAR point cloud data is processed by a CenterPoint [32] network provided by Autoware, producing 3D bounding boxes. Camera and LiDAR often operate at different frequencies, e.g., the camera might operate at a higher frequency. To use all available information, multiple camera images are fused with one LiDAR scan. Each camera image is processed independently, the 2D bounding boxes are matched by bounding box overlap, and their coordinates averaged. These 2D bounding boxes are then fused with the 3D bounding boxes by projecting the latter into the camera plane and matching with the 2D bounding boxes by overlap. Only matched 3D bounding boxes are kept, with the class labels taken from the 2D detections. We run the inference of the YOLOv9 and CenterPoint models on an NVIDIA RTX 4090 GPU, with a single fusion taking about 56 ms.

Our full evaluation pipeline, including the sampling of misaligned timestamp tuples, misalignment computation, and aggregation of fusion errors, is implemented in custom Python code, with no external verification or analysis tool used.

We evaluate the temporal misalignment and robustness of this fusion pipeline under two configurations: In the first (LiDAR centered) the LiDAR sample is intended to be taken in the middle of the two camera samples, while in the second (LiDAR late) the LiDAR sample is intended to be taken at the same time as the second camera sample.

For these configurations, we consider three different scenarios:

• $\mathrm{■}$

  Synchronized: Samples are taken exactly at the intended time points according to the configuration.

• $\mathrm{■}$

  Jitter: Samples independently deviate from their intended time point by a random amount up to $\pm j$ ms, where $j$ is the maximum jitter.

• $\mathrm{■}$

  Uniform shift: All sensor samples are shifted by an identical constant offset of $d$ ms, where $d$ is the shift amount.

In all scenarios, we determine the timestamp misalignment for the reference-point-based definition from [16] and our alignment sets (Section 6.2), showing that the choice of alignment set significantly affects the reported misalignment and resulting robustness estimates.

The evaluation is performed on the KITTI dataset [12], which provides synchronized camera and LiDAR data at 10 Hz (i.e., every 100 ms) across 155 drives. We use the raw dataset, which contains full series of camera frames and LiDAR scans with their corresponding timestamps, allowing us to simulate various temporal misalignment scenarios by shifting timestamps and selecting appropriate samples, as in prior work [16, 11]. Since KITTI provides samples every 100 ms, both jitter and offset are simulated by selecting neighboring samples, so the applied jitter or offset is necessarily a multiple of 100 ms. For the full run in Section 6.5, we remove this restriction by interpolating data in steps of 5 ms.

For the LiDAR centered configuration, the LiDAR sample should lie at the midpoint between two camera frames. Since KITTI only provides samples for both sensors every 100 ms, no LiDAR sample exists at the midpoint (50ms) between consecutive camera frames. Thus, we pair camera frames that are 200 ms apart (i.e., separated by one frame, e.g., 0ms and 200ms, 100ms and 300ms, $\mathrm{\dots }$), fusing each pair with the LiDAR sample between them. For consistency, we keep the sampling of camera images the same for the LiDAR late configuration, but choose the LiDAR sample that aligns with the later camera sample.

We examine five ways to specify alignment. In addition to the reference-point-based definition, we consider four different alignment sets, namely one anchored and one relative alignment for both the LiDAR centered and LiDAR late configurations, capturing different synchronization requirements for the camera-LiDAR fusion setup.

Using the notation from Section 3, the sensor samples are $c_1=(S_{\text{cam}},1)$, $c_2=(S_{\text{cam}},2)$, and $l_1=(S_{\text{lidar}},1)$, with $\theta$ denoting the time difference between the two camera frames. Recall that $s\sim r+\theta$ means sample $s$ is intended to be taken at time $r+\theta$ (see Definition 2). For anchored alignment sets, we use $t^{\star }$ as a fixed reference time point serving as the anchor. Note that in the relative alignment sets, $c_1$ is unconstrained since only the relative timing between samples matters, not their absolute position.

Anchored: LiDAR centered.

All samples are anchored, the LiDAR sample is aligned in the middle between the two camera frames: ${𝒜}_{\text{anchored-inter}}=\{c_1\sim t^{\star },c_2\sim t^{\star }+\theta ,l_1\sim t^{\star }+\frac{\theta }{2}\}$

Anchored: LiDAR late.

All samples are anchored, the LiDAR sample is aligned with the second camera frame: ${𝒜}_{\text{anchored-late}}=\{c_1\sim t^{\star },c_2\sim t^{\star }+\theta ,l_1\sim t^{\star }+\theta \}$

Relative: LiDAR centered.

The second camera frame is relatively aligned to the first one, and the LiDAR sample is temporally centered relative to the two camera frames: ${𝒜}_{\text{relative-inter}}=\{c_2\sim c_1+\theta ,l_1\sim \frac{1}{2}{c}_1+\frac{1}{2}{c}_2\}$

Relative: LiDAR late.

The second camera frame is aligned to the first, and the LiDAR sample is aligned with the second camera frame: ${𝒜}_{\text{relative-late}}=\{c_2\sim c_1+\theta ,l_1\sim c_2\}$

For all alignment sets we use the maximum absolute misalignment across the constraints as the misalignment measure (Definition 7). That is, $g(m_1,\mathrm{\dots },m_{|𝒜|})={\max }_i|m_i|$ where $m_i$ is the misalignment of constraint $C_i$.

We evaluate the robustness estimates of all five specifications: the alignment sets presented in Section 6.2, in comparison to the reference-point-based temporal robustness definitions by Kuhse et al. [16].¹¹1We do not evaluate the sample-point-based definitions [16], as their error functions compare against multiple reference points (all points in the range spanned by the timestamps), making them not directly comparable to the other definitions, which all compare against a single timestamp tuple with no misalignment. A fair comparison would require a separate evaluation, where for each alignment set one would also compare against multiple timestamp tuples with no misalignment. For each misalignment specification and each misalignment threshold $\mathrm{\Delta }$ (from Definition 9), we evaluate the error of the fusion output using 77500 timestamp tuples (500 for each drive in KITTI), sampled from the entire KITTI dataset. Generating all these samples takes less than a second, with the fusion evaluation being the bottleneck, taking about one hour per alignment set and $\mathrm{\Delta }$ combination. Evaluating the fusion error of one individual sample takes about 0.05 seconds. Peak memory consumption is 3.3 GB.

As the KITTI dataset consists of only synchronized camera-LiDAR series, we first describe how we sample misaligned timestamp tuples from the synchronized KITTI data in Section 6.3.1, then explain how we evaluate the resulting fusion error in Section 6.3.2.

We consider several scenarios, showing the limitations of the reference-point-based specification of temporal misalignment and how different alignment sets best capture different synchronization requirements. First, Section 6.4.1 shows that in the synchronized scenario, the reference-point-based specification reports non-zero misalignment. Second, Section 6.4.2 shows that this also leads to overly pessimistic measures for the misalignment under a certain amount of jitter. Third, we consider the case of a uniform time shift in Section 6.4.3, showing the difference between the anchored and relative alignment sets.

For each scenario, we report the maximum misalignment $\mathrm{\Delta }$ under each specification. To translate these into errors, we use the robustness estimates from the evaluation described in Section 6.3. In Table 1, we give the full results on the KITTI dataset with threshold $\mathrm{\Delta }=100,200,300,400,500,600$ ms for all alignment sets and the reference-point-based specification, reporting the position error and F1 score for each case. For a given $\mathrm{\Delta }$, these results represent the expected error at that misalignment level. Hence, Table 1 shows the error corresponding to the maximum misalignment $\mathrm{\Delta }$ for that scenario and specification.

We note that the rows of Table 1 are not directly comparable across specifications, as the meaning of $\mathrm{\Delta }$ differs: for instance, $\mathrm{\Delta }=100$ ms for the reference-point specification means each sample is within 100 ms of a single reference point, while for an alignment set it means each constraint’s misalignment is at most 100 ms.

We evaluate alignment sets for a full trace on the longest KITTI drive (8.5 minutes, 5177 camera frames at 10 Hz, cameras downsampled to 5 Hz). Our aim is to show that alignment sets can be applied to a more complex scenario. We consider a 25 ms jitter and a potential 10 ms drift every 30 seconds. We use alignment sets in two manners: to give a robustness estimate for the entire drive akin to the previous fusion-level scenarios and to investigate the correlation between misalignment and fusion error.

To allow misalignments smaller than the 100 ms granularity, we resort to synthetic data. We interpolate camera frames using RIFE [13] and LiDAR bounding boxes by linear interpolation of their parameters, to achieve an effective 5 ms granularity. We verified by manual inspection that the interpolated data does not introduce significant artifacts.

We extend the alignment sets given in Section 6.2 to consider all fusions in a drive trace. For the anchored alignment sets, for each fusion $i$ we add a $t_i^{\star }$ and the same per-sample constraints as in Section 6.2, anchored to $t_i^{\star }$. We do the same for reference-point-based specification. For the relative alignment sets, we add constraints between samples used in the same fusion as before, but also add a constraint between the first camera sample $c_{i,1}=(S_{\text{cam}},2i-1)$ of each fusion to the second camera sample $c_{i-1,2}=(S_{\text{cam}},2i)$ of the previous fusion, to capture the intended relative timing between fusions: $c_{i,1}\sim c_{i-1,2}+\theta$.

We consider two scenarios: 1) Jitter: samples independently deviate from their intended time point by up to $\pm 25$ ms in $5$ ms steps, and 2) Jitter + Drift: the jitter for each sample is up to $\pm 25$ ms and every 30 seconds all following samples are shifted by 10 ms.

The first experiment extends the robustness evaluation to the entire drive trace. For each specification, we take $\mathrm{\Delta }$ to be the worst-case misalignment under the jitter-only scenario as reported by the specification measure (column $\mathrm{\Delta }$ in Table 5, compare Section 6.4.2). For anchored specifications, drift accumulates over time. Relative specifications only observe drift at the moment a drift step occurs: for fusions before or after the step the shift applies to every sample and cancels out. To model this, we modify our aggregation function $g$ to increase the effective misalignment threshold of affected constraints. We sample 1000 traces of the entire drive for each scenario and specification.

Table 5 shows the results, mirroring the findings of the fusion-level jitter and the uniform shift scenario. The alignment set (${𝒜}_{\text{anchored-inter}}$) gives the best error bounds, while the reference-point-based specification gives overly pessimistic estimates. The relative specifications are invariant to the uniform shift caused by drift, giving nearly identical results.

We also investigate how the misalignment measure $M_{𝒜}(T)$ correlates with the fusion error, by sampling 1000 traces for the scenarios from Experiment 1. Under each specification’s alignment set, we compute the misalignment for each fusion and its fusion error.

We note that calculating the misalignment of 1000 traces takes less than a second, but evaluating the fusion error under an alignment set spans the entire drive, requiring a considerable computation cost of in total 20 hours for 1000 traces. A single trace consists of 1294 individual fusions, resulting in $\approx$ 1.3 million fusions in total. On average, calculating the error of one individual fusion takes about 0.05 seconds. The sampling takes less than 10 seconds with a single trace consuming less than 1 MB. Peak memory consumption evaluating one trace is 3.3 GB.

We use two fusion error functions, capturing different application requirements. $E_{\text{unshifted}}$ compares against the fusion of the original, unjittered and undrifted samples. For the unshifted error function, the drift causes the error to continuously increase, even though fusions are locally correct. $E_{\text{shifted}}$ compares against the fusion of the samples at $(c_1,c_1+\theta ,c_1+\frac{\theta }{2})$, where $c_1$ is the actual (jittered and drifted) first camera timestamp. The two error functions correspond respectively to the use cases captured by anchored and relative alignment sets.

Table 6 reports Pearson ($r$) and Spearman ($\rho$) correlations for each combination of specification, scenario and error function [30]. Pearson’s $r=\frac{\text{cov}(X,Y)}{{\sigma }_X{\sigma }_Y}$ measures linear correlation and Spearman’s $\rho$ monotonic correlation by applying Pearson to the ranks of $X$ and $Y$. Both lie in in $[-1,1]$, with 0 indicating no correlation. The results mirror the findings of the previous experiments. The correctly matching alignment set gives the best correlation with the error function corresponding to its use case.