^{1, 2}

^{2}

^{1}

^{3}

^{4}

^{2}

^{2}

^{1}

^{2}

^{3}

^{4}

This paper aims to develop a general framework for accurately tracking and quantitatively characterizing multiple cells (objects) when collision and division between cells arise. Through introducing three types of interaction events among cells, namely, independence, collision, and division, the corresponding dynamic models are defined and an augmented interacting multiple model particle filter tracking algorithm is first proposed for spatially adjacent cells with varying size. In addition, to reduce the ambiguity of correspondence between frames, both the estimated cell dynamic parameters and cell size are further utilized to identify cells of interest. The experiments have been conducted on two real cell image sequences characterized with cells collision, division, or number variation, and the resulting dynamic parameters such as instant velocity, turn rate were obtained and analyzed.

Being the fundamental unit of life, cell is a key element in many biological processes. Researchers have realized the importance of studying cells motility, deformation or population dynamics, and cell to cell interactions in modern biology. Understanding the dynamical behaviors of cell of interest in living cells is essential to the fundamental studies for discovering effective medical therapy of diseases like cancer, AIDS, or any inflammatory diseases [

Over the past decade, a number of cell tracking algorithms have been proposed (see [

The first type of approaches is to run a cell detector based on texture, intensity, or gradient in every frame and then associate the detected cells between frames [

Inspired by work in [

This rest of the paper is organized as follows. Section

This section describes our proposed method in detail. In Section

The cell detection is a challenging job due to a high noise level in time-lapse microscopy images and wide ranging in intensity and shape. Image enhancement removes blurring and noise, increases contrast, and so forth. After the process of image enhancement, “a hybrid cell detection algorithm” is used to segment overlapping or adhesion cells. This combination method consists of threshold processing, holes filling, noise removal, image dilation, and shape and boundary constraint. The overview of the proposed detection method is given in Figure

A block diagram of the proposed cell detection method.

During the threshold process, individual pixels in an image are marked as “object” pixels if their values are greater than some threshold value (assuming an object to be brighter than the background) and as “background” pixels otherwise. Typically, an object pixel is given a value of “1” while a background pixel is given a value of “0.” Finally, a binary image is created. The key parameter in the threshold process is the choice of the threshold value. Since

An initial threshold (

Create two sets

Compute the average intensity of each set

Create the new threshold

Go back to Step

As shown in Figure

Illustration of the adhesion cell detection by proposed method. (a) Original image, (b) the contrast enhancement results on the whole image, (c) threshold process image, (d) hole-filled image, (e) noise removal image using median filter, and (f) result of our algorithm.

By a careful observation of image, the unexpected high frequency noise significantly affects the quality of image. A median filter is employed to perform noise removal. Median filter runs through the image pixel by pixel and replaces each pixel with the median of neighbor pixels, which is also known as a smoothing technique. To minimize the distortion of edges and remove noise effectively, the value of threshold should be considered carefully. If the threshold of median filter is set a bit low from the value, all the noise can be removed. Figure

After applying dilation filter, the boundaries of regions of foreground pixels which are typically bright pixels in the image are gradually enlarged. The areas of foreground pixels grow in size. Shape and boundary constraint is finally proposed and used to discriminate cells. If a component is either smaller than minimum, or bigger than maximum of cell size range, it will be removed as a noncell component; otherwise it will be kept as cell component. After setting up a bounding box to calculate the cells area, we continue to calculate the “width/height ratio,” and remove the component whose ratio is either bigger than 5.0 or less than 0.5; both are determined empirically.

Figure

In the field of cell tracking, each cell exhibits various behaviors, such as random moving fluctuation, collision, division, and shape deformation in different frames, as illustrated in Figure

An illustration of collision and division cells.

Collision cells

Division cells

The distribution of two elements of cell collision event.

The probability

The probability

The distribution of two elements of cell fission event.

The probability

The probability

The IMMPF framework [

Assume that there are total

In this section, to characterize cell dynamics and quantitative study multicell behaviors, we propose an augmented interacting multiple models particle filter (AIMMPF) tracking algorithm to accurately estimate the state vectors of multiple cells.

As observed in a series of cell image sequences, some cells exhibit unpredictable behaviors when cell collision and division occur, such as sudden change in motion speed, direction, and size of cell area. To deal with these uncertainties, we augment the state vectors in (

We also observe that a successful implementation of our proposed AIMMPF relies on two aspects, namely, the determination of turn rate

In terms of the turn rate variable

Another key issue in our proposed algorithm is how to determine the evolvement of cell state at each mode. Without loss of generality, we propose three interaction modes for cell tracking, namely, augmented variable nearly constant velocity mode for cell noninteracting (ACV), an augmented variable coordinate turn mode for cell collision (ACT1), and an augmented variable coordinate turn mode for cell division (ACT2).

If a given cell does not undergo collision or division, in other words, it moves smoothly with minimum appearance change, the following state transition model is adopted:

If one cell collides with the other in a given frame, its motion speed, direction, and area are assumed to vary accordingly. The current cell turn rate is fluctuant and unknown to us, but we can use (

In frame

To view our proposed cell interaction based AIMMPF tracking algorithm in a straight way, Figure

Tracking block of each cell (for simplicity, the superscript of target is omitted).

Three cell interaction modes

Mode update

State update

To track and discriminate simultaneously multiple cells, the correspondence and label management block is required for our proposed cell interacting based AIMMPF filter. The correspondence aims to introduce the measures to be associated, whereas identity management focuses on the strategy to discriminate and label each cell of interest. Therefore, we need to (1) define a dissimilarity measure between two cells in two consecutive frames and (2) design an appropriate identity management strategy.

If a cell moves in a smooth way, which means that the dynamics of the cell can be known a priori, the position of the cell in the next frame is predicted and further associated preferably with the available closest measurement (i.e., nearest neighbor method). In this way, the obtained distance difference is the smallest one, and such measure is denoted by

This function is independent of the direction of motion and allows nonsmooth trajectories. It can be seen that the above method is mainly dependent on the assumed dynamics of cell, and it often leads to tracking failures in a dense clutter environment or in the case of cell collision. Thus, we further introduce another measure, namely, area difference

The direct objective in multicell tracking is to discriminate and record the dynamic parameters and feature parameters of each cell in each frame for further biological process analysis. In our study, the cell division and collision are investigated. Thus the uncertainties in correspondence increase, which further lead to the difficulty in cell label management. Since our algorithm belongs to the type of probabilistic approach with reliable detection results, both the spatiotemporal and feature information could be utilized as the inputs to correspondence and label management block. In the case of cells collision and division, cells are easily merged in one frame and separated in another frame. As a result, one of them would be detected as a newly born cell for cell division and would be assigned a new label as a new track. In addition, in the case of cell collision, two cells would be merged as one cell, due to only one detection generated around the predicted cell state. To solve this problem, three cases are investigated, and related strategy is adopted as below.

If there is more than one measurement in the associated gate (

No cell collision or division.

Frame

Frame

Suppose that cells

Cell collision.

Frame

Frame

Frame

Without loss of generality, we give an example of cell division as illustrated in Figure

Cell fission.

Frame

Frame

The procedure of correspondence and label management scheme, which is appropriately embedded in our algorithm, is illustrated in Figure

The main framework of our proposed algorithm.

Flow chart of our algorithm

Correspondence and label block of cell

In this section, several experiments were conducted on two real image sequences to verify the effectiveness of our proposed method for cell tracking. These experiment data include various challenging scenarios, such as variation in cell dynamics and population, cell collision, and division in different frames. Our purpose is to estimate the position, velocity, turn rate of each cell from available cell detections. All experiments were implemented in MATLAB on a 1.7 GHz processor computer with 4G random access memory.

In our experiment, all cells are identified by rectangular blobs, and three modes in our proposed cell interacting AIMMPF are adopted, namely, ACV, ACT1, and ACT2.

This case includes cell collision and variation in population, and the Markov transition matrix between three modes is assumed constant and set empirically as

Figure

Multicell tracking with colliding and varying number of cells in different frames.

Original RGB image sequences

Tracking results of our proposed algorithm

Tracking results of IMMPF

The position estimate of each cell in

Due to lack of velocity ground-truth data, we evaluated the performance of our algorithm by comparison with manual tracking results. Instant velocity estimate of cell 1 per frame among three approaches is shown in Figure

Instant velocity estimate per frame using various methods.

Cell 2 in

Cell 2 in

Mean velocity (the track length is 30 frames) of all cells for our proposed algorithm versus IMMPF and manual tracking is shown in Figure

Mean velocity estimate of all cells in image using various methods.

Cells mean velocity in

Cells mean velocity in

Figure

Comparison of cells number estimates by various modes.

Our algorithm

IMMPF

Figure

Results of turn rate estimate using our proposed algorithm.

Cell 4

Cell 7

Mode probability of three modes.

Cell 4

Cell 7

The evolving curve of variables

In this case, the event of cell division is investigated and the corresponding performance of our algorithm is evaluated with comparison to other methods as well. The Markov transition matrix between three modes is assumed constant and set empirically as

As shown in Figure

Tracking results on cell division using various methods.

Original RGB image sequences

Tracking results of our proposed algorithm with detection image sequences

Tracking results with IMMPF with detection image sequences

The position estimate of each cell in

Instant velocity estimate for our proposed algorithm versus manual tracking is shown in Figure

Instant velocity estimate per time step using various methods.

Cell 1 in

Cell 1 in

Results of turn rate estimate using our proposed algorithm.

Cells 1 and 3

Cells 2 and 4

Model probability of three modes.

Cell 1

Cell 2

To get insight into tracking performance, we adopt one measure criterion, namely, percentage of tracked position (PAP) [

Performance comparison results using two methods.

Method | PTP (Scenario |
PTP (Scenario |
---|---|---|

IMMPF | 77.94% | 85.11% |

Our method | 89.71% | 91.49% |

Real-time tracking is required in our studied multicell tracking approach, so the total processing time must be in principle less than the interval between consecutive samplings. Computation time using our method is only 3.1129 s and 1.1930 s for Scenarios

In this paper, through introducing three types of interaction events among cells, namely, independence, collision, and division, an augmented interaction multiple models particle filter tracking algorithm has been presented for spatially adjacent cells with varying size. To reduce the ambiguity of correspondence and establish trajectories of interested cells, both cell topological feature and cell motion feature are used to manage data association problem. Simulation experiments on real image were carried out and the performance comparison has been reported. Our proposed algorithm can successfully track multiple cells of colliding, dividing, or cells of entering and/or leaving field of view, and so forth. Furthermore, it can provide accurate dynamic estimate of each cell, such as position, velocity, and turn rate.

The authors declared that they have no conflict of interests to this work.

This work is supported by National Natural Science Foundation of China (no. 61273312), the Natural Science Fundamental Research Program of higher education colleges in Jiangsu province (no. 14KJB510001), and the Project of Talent Peak of Six Industries of Jiangsu Province (no. DZXX-013, NY-021).