3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset

Introduction

Over the past decades, the use of zebrafish (Danio rerio) as an animal model has increased significantly due to its applicability within large-scale genetic screening . The zebrafish has been used as a model for studying human neurological disorders, drug addiction, social anxiety disorders, and more . Locomotion and behavioral analysis are often critical parts of neuroscientific and biological research, which have traditionally been conducted manually . However, manual inspection is subjective and limited to small-scale experiments. Therefore, tracking systems are getting increasingly popular due to their efficiency and objectivity. The majority of the solutions has been developed for terrestrial animals or fish in shallow water, and most studies have been based on 2D observations in scientific and commercial systems . However, observations in a single plane cannot capture all the relevant phenotypes of fish .

Estimating the 3D trajectories of multiple zebrafish accurately is difficult due to their erratic movement, visual similarity, and social behavior , see Figure 1. This may be one of the reasons why no commercial solution has been developed yet. Only few groups in the scientific community have addressed the problem, focusing mainly on stereo vision and monocular stereo using mirrors . However, no labeled datasets have been made publicly available within the field, which makes a fair comparison between the applied methods difficult. This ultimately hinders significant developments in the field as we have seen in other computer vision fields with common datasets. Therefore, our contributions are

a publicly available RGB 3D video dataset of zebrafish with 86,400 bounding box and point annotations.

A large part of 3D multi-object tracking methods are developed for LiDAR-based traffic datasets or RGB-D tracking . However, to the best of our knowledge, there exists no publicly available annotated RGB stereo dataset with erratic moving and similarly looking subjects like the one we propose.

Related Work

Multi-Object Tracking (MOT). Reliably tracking multiple objects is widely regarded as incredibly difficult. The interest in solving MOT has been steadily increasing since 2015 with the release of the MOT , UA-DETRAC , and KITTI challenges. Within the MOT challenges, the current focus is on either aiming to solve the association problem using deep learning , using techniques such as intersection-over-union based tracking , or disregarding tracking-specific models and utilizing the improvements within object detections .

Zebrafish Tracking. Vision-based tracking systems developed for studying animal behavior have traditionally been based on 2D due to simplicity and because the movement of most terrestrial animals can be approximated to a single plane. The majority of research in zebrafish tracking has followed this path by only allowing the fish to move in shallow water and assuming that motion happens in a 2D plane.

A 2D animal tracker, called idTracker presented by Perez-Escudero et al. in 2014 , uses thresholding to segment blobs and is able to distinguish between individual zebrafish based on intensity and contrast maps. In 2019, Romero-Ferrero et al. presented an updated version of idTracker, called idtracker.ai , which is the current state-of-the-art 2D tracker system based on convolutional neural networks (CNN) for handling occlusions and identifying individuals. The subjects are observed with a camera positioned above a tank with a water depth of 2.5 cm and the distance between camera and subjects is, therefore, approximately the same at all times. As stated by the authors, this simplifies the task compared to a real 3D tracking scenario.

However, as most aquatic species move in three dimensions, trajectories in 3D are required to thoroughly describe their behavior . The most frequently used acquisition method when dealing with studies of animal behavior in 3D is stereo vision . 3D tracking of zebrafish has been focused mainly on single subjects or small groups, as occlusion is a big hindrance for maintaining correct IDs due to their shoaling behavior . Furthermore, the visual appearance of the fish can change dramatically depending on the position and posture, which makes re-identification more complex compared to 2D.

The Track3D module from the commercial EthoVision XT is popular for tracking zebrafish in 3D, but is limited to a single individual . An early semi-automatic 3D tracking system was developed by Viscido et al. to investigate the relationship between individual members of fish schools. Initial 2D tracks were generated by a nearest neighbor algorithm followed by a step allowing the user to adjust and correct the proposed 2D trajectories, and subsequently triangulated to reconstruct the 3D trajectories.

Qian et al. have worked extensively with tracking of zebrafish and have developed a 2D tracking system with a top-view camera using an augmented fast marching method (AFMM) and the determinant of the Hessian . This was expanded to 3D tracking by extending the setup with a side-view camera. AFMM was utilized to generate a feature point based fish representation in each view followed by 2D tracklet construction based on motion constraints. 3D tracks were then constructed by associating the 2D tracklets with side-view detections using epipolar and motion consistency constraints . Liu et al. extended this method to better handle occlusions based on a set of heuristic methods and the epipolar constraint. A third camera was added in , and the feature point representation method was extended.

Cheng et al. utilized a similar three-camera setup, applying an iterative unsupervised learning method to train a CNN-based classifier to distinguish between the individual fish from a camera placed above the water tank. The classifier was trained on the head region of the fish during periods when all fish were visible at the same time. By iteratively retraining the classifier, they were able to generate 2D tracks from the top-view and reconstruct the 3D tracklets based on detections from the two other side-view cameras under epipolar and motion constraints.

Wang et al. also utilized a three-camera setup, using a Gaussian Mixture Model, a Gabor filter and an SVM-based method to detect the fish heads in the top- and side-views, respectively. The top-view detections are associated into 2D tracklets based on a cross-correlation method and by applying a Kalman filter; near linear movement is achieved by a frame rate of 100 FPS. The 2D tracklets are then constructed into 3D tracklets by associating the side-view detections under epipolar and motion constraints. In , Wang et al. proposed to model the top-view movement of the zebrafish through long short-term memory networks, which were used to improve the motion constraints in a new iteration of their 3D system . Lastly, Wang et al. used a CNN for re-identification of zebrafish heads from the top-view , although this has yet to be incorporated into a 3D tracking setup. None of the methods are able to track multiple zebrafish in 3D for more than a few seconds without ID swaps; this is still a difficult and unsolved problem.

Datasets. As in other MOT challenges, there is a mutual agreement that occlusion is what makes 3D tracking of zebrafish difficult. Nonetheless, only Wang et al. describe their recordings based on occlusion frequency; however, they do not define how it is measured. Qian et al. indicate their complexity based on the amount of fish, but only four occlusion events occur during their 15 seconds demo video with ten fish. For comparison, there are 66 occlusion events in our 15 seconds sequence with ten fish.

Proposed Dataset

The proposed 3D zebrafish dataset, 3D-ZeF, has been recorded from a top- and front-view perspective. This approach was taken to minimize events of total occlusion typical for side-by-side binocular setups. An example of the visual variation between the views is shown in Figure 2 together with an illustration of the experimental setup.

The setup used to record the proposed dataset was built entirely from off-the-shelf hardware, whereas previous methods have used specialized camera equipment. An illustration of the setup is shown in Figure 2. The two light panels are IKEA FLOALT of size $30\times 30$ cm with a luminous flux of 670 lumen and a color temperature of 4000K. The test tank is a standard glass aquarium of size $30\times 30\times 30$ cm with a water depth of 15 cm. The top and front cameras are GoPro Hero 5 and GoPro Hero 7, respectively. All the videos are recorded with a resolution of $2704\times 1520$, 60 FPS, 1/60 $s$ shutter speed, 400 ISO, and a linear field of view. However, the fish tank does not take up the entire image, therefore, the effective region of interest is approximately $1200\times 1200$ and $1800\times 900$ for the top- and front-view, respectively. Diffusion fabric was placed in front of the top light in order to reduce the amount of glare in the top-view. Semi-transparent plastic was attached to three out of four of the window panes in order to reduce reflections. Furthermore, the front camera was placed orthogonally to the water level, which reduced reflections from the water surface. Lastly, the pair-wise recordings have been manually synchronized using a flashing LED, which results in a worst case temporal shift of $\frac{1}{2\cdot\textsc{FPS}}$.

2 Dataset Construction

A total of eight sequences were recorded and divided into a training, validation, and test split. Each sequence consists of a pair of temporally aligned top- and front-view videos and the specifications of the three splits are shown in Table 1. In order to avoid data leakage, each split contains a unique set of fish. The training and validation set of fish were from the same cohort, whereas the fish in the test split were from a younger cohort. Therefore, the test set differs from the training and validation set, as the fish are smaller and behave socially different. This represent a real-life scenario where different cohorts need to be tracked, which has not generally been addressed within the field.

The zebrafish were manually bounding box and point annotated with consistent identity tags through all frames. The bounding boxes were tightly fitted to the visible parts of the zebrafish and the point annotations were centered on the head. If a set of fish touched, an occlusion tag was set for all involved bounding boxes. During occlusions, the bounding box was fitted to the visible parts of the fish and not where it was expected to be due to the extreme flexibility of the zebrafish. The pair-wise point annotations from the two views were triangulated into 3D positions using the method proposed by Pedersen et al. . The fish head was approximated during occlusions to ensure continuous 3D tracks.

It should be noted that the data was recorded in RGB. Zebrafish can change their body pigmentation based on their environment, stress level, and more . The changes in coloration can be important in behavioral studies and may even be valuable in solving the 3D tracking problem.

3 Dataset Complexity

Intuitively, a higher number of fish creates a more difficult tracking problem. However, this is only true to some extent as the main complexity factor is the number and level of occlusions, which depends on a combination of the social activity and amount of space rather than the number of individuals. Therefore, we have defined a range of metrics based on occlusion events to describe the complexity of the proposed sequences. An occlusion event is defined by a set of consecutive frames, where a fish is part of an occlusion. The events are measured from the perspective of the fish; if two fish are part of an occlusion it counts as two events.

The number of occlusion events indicates how often a fish is part of an occlusion, but, few long occlusions can be just as problematic as many short. The length of the occlusions and time between them are, therefore, important to keep in mind when evaluating the complexity of a recording. Due to our definition of occlusion events there are cases where fish are part of occlusions with only minor parts of their bodies. Therefore, the intersection between occlusions is measured as an indication of the general intersection level. The metrics that we provide as basis for the complexity level of our recordings are defined here: Occlusion Count (OC): the average number of occlusion events per second. Occlusion Length (OL): the average time in seconds of all occlusion events. Time Between Occlusions (TBO): the average time in seconds between occlusion events. Intersection Between Occlusions (IBO): a measure of how large a part of the fish that is part of an occlusion event. The intersection in a frame, $f$, for fish $i$ is given by

where $n_{\text{occ}}$ is the number of fish in an occlusion event, and bbj is the set of pixel coordinates in the bounding box of fish $j$. IBO is measured across all bounding boxes with an occlusion tag in a given frame, even for subjects that are not part of the same occlusion. Two examples are presented in Figure 3, where the IBOi,f is calculated from the perspective of the targets enclosed in yellow.

The blue area in the second example, represents the intersection with a subject that is not part of the same occlusion as the target. Additionally, the annotated bounding boxes enclose only the visible parts of the subjects. Thus, the actual intersection between the subjects can be higher if a large part of a fish is hidden. Nonetheless, the assumption is that a high IBO is an expression of heavy occlusion and vice versa. The IBO measure presented in Table 1 is an average between all fish in all frames. A single complexity measure is calculated per sequence, by combining the four proposed metrics by

Method

The pipeline of the proposed 3D tracker follows a modular tracking-reconstruction approach, where subjects are detected and tracked in each view before being triangulated and associated across views. This allows us to use the temporal information of the tracklets in the two views in the 3D association step in opposition to a reconstruction-tracking approach, where detections are triangulated before tracks are generated.

A consistent 2D point is needed in each view in order to create 3D trajectories. As the head is the only rigid part of the body the tracking point is chosen to be located between the eyes of the fish. We present two simple methods to find the head-point of the fish: a naive approach, that does not require training, and a CNN based approach.

Naive: A background image, $bg$, is initially estimated for each view by taking the median of $N_{bg}$ images sampled uniformly across the videos. Subsequently, the background is subtracted by calculating the absolute difference image, $fg=|im-bg|$. To locate the head of a fish in the top-view, the $fg$ is binarized using the intermodes bimodal threshold algorithm . The skeletonization approach of Zhang and Suen is applied, and the endpoints are analyzed to locate the head of the fish. In the front-view the $fg$ is binarized through the use of a histogram entropy thresholding method because the appearance of the fish cannot be approximated as bimodal. The head point is estimated as being either the center of the blob or one of the middle edge points of the blob along the minor axis of the detected bounding box. All three points are evaluated during the 3D reconstruction step, and the two points with the highest reprojection errors are discarded.

FRCNN-H: A Faster R-CNN model has been trained for each view. The bounding boxes have been extracted from all the head-point annotations in the training sequences in order to train a head-detector model for each view. The bounding boxes have static diameters of 25 and 50 pixels for the top-, and front-view, respectively. The head-points are determined as the center of the detected bounding boxes which have a minimum confidence of $c$.

See the supplementary material for more detailed information on the detectors.

2 2D Tracklet Construction

As zebrafish move erratically, it is difficult to set up a stable motion model. Therefore, we use a naive tracking-by-detection approach. The tracking is done by constructing a distance matrix between the detections in a frame and the last detections of current tracklets. The matrix is solved as a global optimization problem using the Hungarian algorithm . Tracklets are deliberately constructed in a conservative manner, where robustness is encouraged above length. A new detection is only assigned to a tracklet located within a minimum distance, denoted $\delta_{\text{T}}$ and $\delta_{\text{F}}$, for the top and front view respectively. If a tracklet has not been assigned a detection within a given amount of time, $\tau_{k}$, the tracklet is terminated.

3 2D Tracklet Association Between Views

The 2D tracklets from each view are associated into 3D tracklets through a graph-based approach. All 2D tracklets with less than a given number of detections, $\alpha$, are removed in order to filter out noisy tracklets. The 3D calibration and triangulation method from Pedersen et al. is used.

A directed acyclic graph (DAG) is constructed. Every node represents a 3D tracklet and consists of two 2D tracklets; one from each camera view. Each edge associates nodes, where the 3D tracklet is based on the same 2D tracklet from one of the views.

Create nodes: The graph nodes are constructed by processing each top-view tracklet and identifying all temporally intersecting front-view tracklets as given by

where $F_{\text{T}}$ and $F_{\text{F}}$ are the set of frames with detections in the top- and front-view tracklets, respectively, and $I$ is the set of frames with detections in both views. If $I=\emptyset$, the node is not created.

An example is presented in Figure 4, where both the blue and red tracklets in the top-view intersects with the three tracklets in the front-view. The outer and inner circles of the six nodes represent the top- and front-view tracklets, respectively. The number inside the nodes indicates the node weight, which is calculated as follows.

For each intersecting frame in $I$, denoted $f$, the 2D tracklets are triangulated. This results in a 3D point of the zebrafish head, $p_{f}$, with a reprojection error, $x_{f}$. For the Naive method where the head is not directly detected in the front-view, the top-view 2D point is triangulated with the three estimated points to find the match resulting in the smallest reprojection error. Therefore, $p_{f}$ represents the point with the smallest reprojection error. To penalize large reprojection errors, the complimentary probability from the exponential cumulative distribution function (CDF), $\Phi$, is utilized. The exponential CDF is chosen as it approximately models the reprojection error of the ground truth training data. The set of weights for all valid 3D points, $V$, can be described by the following set-builder notation

where $\lambda_{err}$ is the reciprocal of the mean of the training data reprojection error, and $A$ states whether $p_{f}$ is within the water tank. The per-frame weights in $V$ are combined into a single weight, $W$, for the entire node by

and the node is added to the DAG given that $W\neq 0$. This weighting scheme considers both the reprojection error and the ratio of frames with valid 3D points compared to the set of all frames $I$. The median function is used instead of the mean function in order to counteract that a few extreme outliers skew the weight.

Connect nodes: The nodes in the DAG should be connected to all other nodes building on one of the same 2D tracklets, as long as the 2D tracklets in the other view do not overlap temporally, as illustrated in Figure 4. This is done by constructing the set of node pairs, $P$, from the set of nodes in the DAG, $N$. Each element of $N$, denoted $n$, consists of the 2D tracklets, $t_{\text{F}}$ and $t_{\text{T}}$, the 3D tracklet, $t$, and the node weight, $W$. Nodes $n_{i}$ and $n_{j}$ are considered a pair if $t_{i,\text{T}}=t_{j,\text{T}}$ or $t_{i,\text{F}}=t_{j,\text{F}}$, if the 2D tracklets in the other view do not temporally overlap, and if $t_{i}$ starts earlier in time than $t_{j}$. This is necessary in order to avoid assigning multiple detections to the same frame.

This can be represented by the set-builder notation

where $O$ assesses whether $t_{i}$ starts before $t_{j}$, and $T$ ensures that the 2D tracklets in $n_{i}$ and $n_{j}$ do not temporally overlap, where $n=\{t_{\text{T}},t_{\text{F}},t,W\}$.

For each node pair in $P$, the weight, $E$, of the directed edge from $n_{i}$ to $n_{j}$ is based on:

$s$, the speed of the fish as it moves between the last detection in $t_{i}$ and the first detection in $t_{j}$.

$t_{d}$, the temporal difference between $t_{i}$ and $t_{j}$.

$W_{i}$ and $W_{j}$, the weights of the nodes.

The edge weight is calculated as the complimentary probability of the CDF of the exponential distribution, $\Phi$. The exponential distribution is chosen as it approximately models that of the speed of the zebrafish. $E$ is calculated by

where $\tau_{p}$ is an empirically chosen value, and $\lambda_{s}$ is the reciprocal of the sum of the mean and standard deviation of the measured speed in the training data. In case a node is not present in any node pairs, the node will be assigned to the DAG, but it will have no edges. The DAG is therefore a disconnected graph.

3.2 Graph Evaluation

4 3D Tracklet Association

The final 3D tracks are constructed from the 3D tracklets in a greedy manner. A set of tracklets equal to the amount of fish present, $N_{\text{fish}}$, is used as initial main tracklets. The remaining tracklets, denoted gallery tracklets, are assigned one by one to a single main tracklet, until no more tracklets can be assigned.

The set of $N_{\text{fish}}$ in the main tracks is selected by finding the stable tracklets that are temporally concurrent in time and span long time intervals. For each tracklet, the set of other temporally concurrent tracklets is considered. In this set, all possible combinations of size $N_{\text{fish}}$ are investigated. If all tracklets in the set overlap temporally, the set is saved as a valid tracklet set. The valid tracklet set with the highest median temporal overlap is used to construct $N_{\text{fish}}$ full 3D tracks. This is done by using the greedy association scheme described in the following sections. No 3D tracks are created if no valid combination of size $N_{\text{fish}}$ is identified.

4.2 Greedy Association

A greedy association algorithm is used when each gallery tracklet is associated with a single main tracklet. The greedy part of the algorithm concerns the way that gallery tracklets are chosen; all gallery tracks are ranked in ascending order by the shortest temporal distance to any main tracklet. If the gallery tracklet overlaps temporally with all main tracklets, it is relegated to the end of the list. When the gallery tracklet has been associated with a main track, the remaining gallery tracks are re-ranked, and the process repeated. In this way, the main tracklets are “grown” into full tracks. The gallery tracklet assignment is based on minimizing the cost of assignment. The cost is based on a set of distance measures, which are determined from two cases.

In the first case at least one main tracklet does not temporally overlap with the gallery tracklet. In this case, the association process is based on the spatio-temporal distances between the gallery tracklet and main tracklets. All temporally overlapping main tracklets are not considered.

In the second case the gallery tracklet overlaps temporally with all main tracklets. As the spatio-temporal distances between the main and gallery tracklet is no longer measurable, a different set of distance values are used: The internal spatio-temporal distances, the amount of intersecting frames, i.e. frames with a detection in both the main and gallery tracklets, and the ratio of intersecting frames compared to the total amount of detections in the gallery tracklet. The internal spatio-temporal distances are determined through the construction of a DAG, where each node is a detection in a frame, and the edge weights are the spatial distances between the temporally previous nodes. The final path is the one minimizing the spatial distance traveled. An example of a graph is shown in Figure 6. The distances are calculated as the mean of the values when the graph switches from a detection in the gallery tracklet to the main tracklet and vice versa.

Association: The distance measures are consolidated into a single assignment decision through a global cost scheme. Each distance value is normalized across valid main tracklets into the range $[0;1]$ and sum to 1. The final cost of assigning the gallery tracklet to a main tracklet, is obtained by calculating the mean of the normalized distance values. The gallery tracklet is associated with the main tracklet with the smallest cost, unless all main tracklet costs are located within a small margin, $\beta$, of each other, in which case the gallery tracklet is discarded. $\beta$ directly enforces a margin of confidence in the assignment, in order to not assign a gallery traklet based on inconclusive cost values.

Evaluation

The metrics used in the MOT challenges and the Mean Time Between Failures (MTBF) proposed by Carr and Collins are utilized to measure the performance of the system on the proposed dataset. The MOT challenge metrics consist of the CLEAR MOT metrics , the mostly tracked/lost metrics , and the identification-based metrics .

The final 3D tracks are evaluated based on a subset of the MOT challenge metrics and the monotonic MTBF metric. The primary metric used is the multiple object tracking (MOTA) metric. The detected and ground truth tracklets are compared using the detected and annotated head points. A detection is only associated with a ground truth tracklet if it is within a distance of 0.5 cm.

The performance of the system is evaluated with two different detection modules: Naive, and FRCNN-H. The results are compared with a hypothetical tracker, called Oracle, which tracks perfectly at all times except during occlusions. This provides an upper bound on the performance if occlusions are not handled in any way. The full set of metrics, system parameters, and results can be found in the supplementary material.

Results for all sequences compared to data complexity is shown in Figure 7, and metrics for the test sequences are shown in Table 2. It is clear that the FRCNN-H outperforms the Naive method on the training and validation splits; it even outperforms the Oracle tracker in three out of four cases. This is likely due to the method being able to detect some of the fish heads during occlusions. However, the superior performance is only seen on the two splits where the fish are from the same cohort. On the test set the FRCNN-H fails to generalize, whereas the Naive method still manages to track the fish.

It should be noted that the poor performance of the Naive method on Tst1, is suspected to be due many short tracks from erratic movement, which the pipeline with the used parameter settings does not handle well.

It has not been possible to make a fair comparison with the other 3D zebrafish tracking methods mentioned in Section 2. Previous systems have been analyzed in terms of ID swaps, fragments, precision, and recall for the generated 2D and 3D tracks. However, there is no exact description of how these metrics are calculated. The evaluation protocol is further limited by not including a statement on the maximum allowed distance between estimated and ground truth tracks leading to uncertainty on the accuracy of the metrics.

Furthermore, the evaluated sequences are not described in terms of complexity, even though occlusion is repeatedly stated as a major hindrance in 3D zebrafish tracking. The only common complexity indication of the datasets is the number of fish, even though it is not representative. An example of this is the tracking demo video of Qian et al. with ten fish and only four occlusion events during 15 seconds. Wang et al. describes their dataset on basis of an occlusion probability but do not explain how it is measured.

There are currently no publicly available annotated data and the previous systems are evaluated on seemingly simplified cases of the problem. Furthermore, the used evaluation protocols are lacking details in such a manner that it is not possible to determine under which conditions the metrics have been calculated. This, along with inaccessible codebases, severely limits the reproducibility of the results, and makes it impossible to ensure identical evaluation procedures. Therefore, it simply does not make sense to compare the proposed system to the other methods under the current circumstances.

Conclusion

Zebrafish is an increasingly popular animal model and behavioral analysis plays a major role in neuroscientific and biological research. However, it is tedious and subjective to manually describe the complex 3D motion of zebrafish. Therefore, 3D zebrafish tracking systems are critically needed to conduct accurate experiments on a grand scale. The significant development experienced in other fields of MOT has not yet translated to 3D zebrafish tracking. The main reason being that no dataset has been made publicly available with ground truth annotations. Therefore, we present the first publicly available RGB 3D zebrafish tracking dataset called 3D-ZeF.

3D-ZeF consists of eight stereo sequences with highly social and similarly looking subjects demonstrating complex and erratic motion patterns in three dimensions that are not seen in common MOT challenges. A complexity measure based on the level of occlusions has been provided for each sequence to make them comparable to future related datasets. The proposed dataset is annotated with 86,400 bounding boxes and points; the latter used for estimating ground truth 3D tracks based on the head position of the fish. Different cohorts of zebrafish are used in the training, validation, and test splits to avoid data leakage; a problem that has never been addressed within the field.

The proposed Naive method scores a MOTA between 25% and 80% across the entire dataset, which correlates well with the complexity measure of the recordings. The open-source modular based system provides a baseline and stepping stone for further development within the field of 3D zebrafish tracking and understanding.

References

Appendix A Content

In these supplementary materials we provide: examples of the visual difference between the fish in the three splits, a more detailed explanation of the detectors, details on the benchmark pipeline parameters, a more detailed presentation of the relation between the proposed dataset complexity measures and tracking metrics, and the full set of tracking metrics for each step in the proposed benchmark pipeline.

Appendix B Fish Examples

The visual appearance of the fish varies both within and between the splits as unique groups of fish have been used for each split. However, the two groups of fish used in the train and validation sets are from the same cohort, whereas the zebrafish in the test split are from a cohort of younger and smaller fish. We present a fish from each of the three splits in Figure 8 captured at the approximately same position in the water tank. The resolution of the zebrafish from the test split is significantly smaller compared to the fish in the training and validation splits, mainly due to the physical size of the fish.

Appendix C Naive Object Detection

The Naive object detection algorithms presented briefly in the paper are described in more details in this section. The methods are inspired by Qian et al. and their work on tracking zebrafish in 2D.

Initially the background image, without any fish, is estimated by taking the median of $N$ images sampled uniformly across the video. In the tests presented in the paper a set of 80 images were used to create the background image for each recording. All further processing is restricted within a defined region of interest based on the boundaries of the water tank and the level of water, in order to limit the processing time.

The video is downsampled by a factor of $2$, in order to further decrease the processing time. Background subtraction is applied by calculating the absolute difference image, $|im-bg|$, choosing the max value across all color channels, and normalizing the image into the range $[0;255]$. The resulting image is filtered with a $5\times 5$ median filter to reduce noise.

C.2 Top-View Detection

The top-view is thresholded based on the assumption of the image histogram being bimodal, due to a near uniform bright background, and darker zebrafish. Therefore the intermodal approach of Prewitt et al. can be utilized. The threshold is set to the middle point between the two modes in the image histogram. If the histogram is not bimodal, the frame is filtered with a $3\times 3$ mean filter, until its histogram is bimodal.

The fish are detected by applying a skeletonization based approach. BLOBs are initially detected and filled, and the skeletonization approach of Zhang and Suen is applied. The skeleton is analyzed in order to find the skeleton keypoints: head, tail, and junctions. This is done by convolving the frame with the kernel in Equation (8). All end points of the skeleton have a value of 116, 117, 118, or 131, while junctions have a value of 148, 149, 150, or 151. Values, like 132, that can represent both an end point or an arbitrary point on the skeleton are not considered.

Each keypoint is assigned a weight, $w$, consisting of the smallest eigenvalue of the covariance matrix of the BLOB coordinates extracted from a $20\times 20$ window around the keypoint. The smallest eigenvalue is used, as it is assumed the variance will be smallest along the width of the head. This way the head keypoints will be assigned a larger $w$ than the tail keypoints, as the width of the zebrafish head is usually larger than the width of the zebrafish tail. The $w$ of the junction points are reduced by a factor of $2.5$, as they are usually larger than endpoint keypoints.

Keypoints too close to each other are removed by applying non-maximal suppression (NMS). Each keypoint is assigned a bounding box of size $w\times w$, and kept if the bounding box does not overlap with any other keypoint by more than NMS${}_{\text{thresh}}$% of the area of the keypoint in focus. In case there is an overlap, only the keypoint with the largest $w$ is kept. NMS${}_{\text{thresh}}$ was set to 50 in the conducted tests.

Finally, all keypoints with $w<1$ are discarded. Per skeleton, the two keypoints furthest apart are determined, and the keypoint with largest $w$ is kept. Furthermore, half of the additional extra keypoints with the largest $w$ values are also kept in order to handle crossings and occlusions. Every keypoint ideally represent a zebrafish head and they are all kept as individual detections.

C.3 Front-View Detection

The front frame cannot be thresholded based on the assumption of a bimodal distribution of the image histogram, as the stripes of the zebrafish results in a non-uniform appearance. Instead the frame is thresholded by finding the point maximizing the total entropy of the image . The entropy is modeled as background and foreground entropy for each non-zero bin.

For bin $k$ the background entropy, $b_{k}$, is calculated by

where $h$ is the normalized image histogram and $h_{c}$ is the cumulative histogram of $h$. The foreground entropy, $w_{k}$, is calculated by

resulting in the two entropy sets $B=\{b_{0},b_{1},..,b_{255}\}$ and $W=\{w_{0},w_{1},..,w_{255}\}$ which are combined into

and the threshold, $t$, is then finally determined by

The BLOBs in the thresholded image are found through simple Connected Component Analysis (CCA). Only the $2N_{\text{fish}}$ largest BLOBs are kept, as long as the area of the BLOBs are larger than a predefined threshold. This allows for potential reflections being detected alongside the true detection of the fish. Otherwise, if only the $N_{\text{fish}}$ largest BLOBs are considered for further analysis, it is possible that reflections are considered at the expense of real fish.

As it is not known where the head of the fish is, two points are saved as proxies for the head, together with the center-of-mass. If the width of the BLOB bounding box is larger than the height, the proxy points are $(\min(x),\mu(y))$ and $(\max(x),\mu(y))$, where x and y are the coordinates of the BLOB pixels, and $\mu$ is the arithmetic mean function. Otherwise, the proxy points are $(\mu(x),\min(y))$ and $(\mu(x),\max(y))$. The center-of-mass is further used for constructing 2D tracklets, as it is the most stable of the three proxy points.

Appendix D FRCNN-H Object Detection

Two Faster R-CNN networks were trained to detect the zebrafish heads in each view, respectively. The official PyTorch implementation of Faster R-CNN with a ResNet50 based Feature Pyramid Network backbone was utilized in both cases. The model was pretrained on the COCO train2017 dataset. The network was fine-tuned for 30 epochs, using stochastic gradient descent with momentum. A learning rate of 0.005, momentum of 0.9, weight decay of 0.0005, and batch size of 8 was used. The learning rate was “warmed up” during the first epoch, linearly interpolating the learning rate from 0.001 to 0.005. Each model was trained on an RTX 2080TI.

Appendix E Pipeline Parameters

The full system pipeline has a set of parameters, which needs to be manually set. The parameters were chosen based on empirical investigation on the training data and they are shown in Table 3. The front-view distance threshold, $\delta_{F}$, has two different values, depending on the method applied. For the Naive method it is set to $0.5$, as the distance is measured in standard deviations, whereas for the FRCNN-H method it is set to $15$, where the distance is measured in pixels.

Furthermore, during the association of 2D tracklets between views, a set of exponential distributions are utilized. These distributions are parameterized using the mean of the reprojection error of the ground truth training annotations and the average speed of the zebrafish in the training split. This builds on the assumption that these parameters generalize across the different splits. As shown in Table 4, the average speed of the zebrafish for each split are within $0.1\frac{cm}{s^{2}}$, which can be seen as a negligible difference, whereas the mean reprojection error is $2-4$ pixels larger in the training and validation splits than in the testing split. The cause of this difference is hypothesized to be the larger amount and duration of occlusions in the training and validation splits, which can introduce some uncertainty during the annotation phase. The training split has the largest reprojection error, which means the utilized exponential distribution penalize larger reprojection errors less harshly. However, as the mean training reprojection error is still low, it is still deemed fitting for the task.

Appendix F Tracking Metrics and Results

When evaluating tracking tasks, a wide suite of metrics are often used, each expressing different properties of the task. Commonly, the MOTA metric is used as a representative metric for the overall performance of the tracker, as it incorporates false positives and negatives, as well as the amount of identity swaps. However, as shown by Carr and Collins the MOTA metric is not guaranteed to be robust at all times. We evaluate the generated tracks using the collection of metrics from the MOT challenge, consisting of the CLEAR MOT , mostly tracked/lost metric , and identity based metrics of Ristani et al. , as well as using the MTBF metric proposed by Carr and Collins . A description of each of the used metrics are presented in Table 5. Each step of the pipeline has been evaluated according to the previous mentioned metrics. Specifically the following steps have been evaluated:

2D tracklets after 2D Tracklet Construction (for top-view see Table 6 and for front-view see Table 7)

2D tracklets after 2D Tracklet Association Between Views (for top-view see Table 8 and for front-view see Table 9)

3D tracklets after 2D Tracklet Association Between Views (see Table 10)

3D tracks after 3D Tracklet Association (see Table 11)

For the 2D tracklets, a distance threshold of 20 px is used.

Appendix G Occlusions

The two views are not equal when it comes to occlusions. Due to the social behavior of zebrafish they tend to swim in the same horizontal layer of the water column. This is indicated by the graph in Figure 9, where the occlusion lengths are displayed with respect to the proposed complexity measure, $\Psi$. It should be noted that $\Psi$ is calculated per view and not as a mean between the two, as in the main article. Data points are presented for all the recordings from both views, except Tst1 as it only contains a single fish and therefore has a complexity measure of zero. The dashed lines illustrate the mean occlusion lengths and shows that there is a significant difference between the two views. In general, the occlusion events seems longer in the front-view recordings.

The MOTA of the hypothetical oracle tracker is plotted against the proposed complexity measure in Figure 10. MOTA is calculated for the top- and front-view 2D tracklets as presented in Table 6 and Table 7, respectively. The tracking performance of the oracle tracker correlates well with the complexity in both views. Notice the significant difference in performance of the tracker in the two views; the mean scores are MOTA${}_{\text{top}}=86.0\%$ and MOTA${}_{\text{front}}=75.5\%$ as illustrated by the two dashed lines.