Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras

Introduction

Estimating 3D structure and camera motion from video is a key problem in computer vision. Traditional approaches to this problem rely on identifying the same points in the scene in multiple consecutive frames and solving for a 3D structure and camera motion that is maximally consistent across those frames . But such correspondences between frames can only be established for a subsets of all pixels, which leaves the problem of estimating depth underdetermined. As commonly done with inverse problems, the gaps are filled with assumptions such as continuity and planarity.

Rather than specifying these assumptions manually, deep learning is able to obtain them from data. Wherever information is insufficient to resolve ambiguities, deep networks can produce depth maps and flow fields by generalizing from prior examples they have seen. Unsupervised approaches allow learning from raw videos alone, using similar consistency losses as traditional methods but optimizing them during training. At inference, the trained networks are able to predict depth from a single image and egomotion from pairs or longer sequences of images.

As research in this direction got traction , it became clear that object motion is a major obstacle because it violates the assumption that scene is static. Several directions have been proposed to address the issue , including leveraging semantic understanding of the scene through instance segmentation . Occlusions have been another limiting factor, and lastly, in all prior work in this direction, the intrinsic parameters of the camera had to be given.

This work addresses these problems and, as a result, reduces supervision and improves the quality of depth and motion prediction from unlabeled videos. First, we show that a deep network can be trained to predict the intrinsic parameters of the camera, including lens distortion, in an unsupervised manner, from the video itself (see Fig. 1). Second, we are the first in this context to address occlusions directly, in a geometric way, from the predicted depth as it is. Lastly, we substantially reduce the amount of semantic understanding needed to address moving elements in the scene: Instead of segmenting every instance of a moving object and tracking it across frames , we need a single mask that covers pixels that could belong to a moving object. This mask can be very rough, and in fact can be a union of rectangular bounding boxes. Obtaining such a rough mask is a much simpler problem and can be solved more reliably with existing models than instance segmentation.

In addition to these qualitative advances, we conduct an extensive quantitative evaluation of our method and find that it establishes a new state of the art on multiple widely used benchmark datasets. Pooling datasets together, a capability which is greatly advanced by our method, proves to enhance quality. Finally, we demonstrate for the first time that depth and camera intrinsics prediction can be learned on YouTube videos, which were captured with multiple different cameras, each with unknown and generally different intrinsics.

Related work

Estimating scene depth is an important task for robot navigation and manipulation and historically much research has been devoted to it including large body of research on stereo, multi-view geometry, active sensing and so on . More recently learning-based approaches for depth prediction have taken center stage based on learning of dense prediction . In these, scene depth is predicted from input RGB images and the depth estimation function is learned using supervision provided by a sensor, such as a LiDAR. Similar approach is used for other dense predictions e.g. surface normals .

Unsupervised learning of depth, where the only supervision is obtained from the monocular video itself and no depth sensors are needed, has also been popularized recently . Garg et al. introduced joint learning of depth and ego-motion. Zhou et al. demonstrated a fully differentiable such approach where depth and ego-motion are predicted jointly by deep neural networks. Techniques were developed for the monocular and binocular settings. It was shown that depth quality at inference is improved when stereo inputs are only used during training, unlike methods that relied on stereo disparity at inference too . Other novel techniques include the use of motion

Learning from images or videos in the wild.

Learning depth from images in the wild is also an active research field, mostly focusing on single or multi-view images . It is especially hard as shown by Li et al for internet photos, due to the diversity of input sources and no knowledge of the camera parameters . Our work makes a step in addressing this challenge by learning intrinsics for videos in the wild.

Occlusion aware learning.

Multiple approaches have been proposed for handling occlusions in the context of optical flow . These approaches are disconnected from geometry. Differentiable mesh rendering, has recently been attracting increasing attention , begins to adopt a geometric approach to occlusions. In the context of learning to predict depth and egmotion, occlusions were addressed via a learned explainability mask , by penalizing the minimum reprojection loss between the previous frame or the next frame into the middle one, and through optical flow . In the context of unsupervised learning of depth from video, we are the first to propose a direct geometric approach to occlusions via a differentiable loss.

Learning of intrinsics.

Learning to predict the camera intrinsics has mostly been limited to strongly supervised approaches. The sources of groundtruth varies: Workman et al. use focal lengths estimated employing classical 1D structure from motion. Yan et al. obtain the focal length based on EXIF. Bogdan et al. synthesize images from panoramas using virtual cameras with known intrinsics, including distortion. To our knowledge, our approach is the only one that learns the camera intrinsics in an unsupervised manner, directly from video, jointly with depth, ego-motion and object motion.

Preliminaries

Our method extends prior techniques in the domain of simultaneous learning of depth and motion . Similarly to prior work, the backbone of our method is the equation that ties together two adjacent video frames using a depth map and the camera matrix:

and $p$ and $p^{\prime}$ are pixel coordinates in homogeneous form before and after the transformation represented by the rotation matrix $R$ and the translation vector $t$. $z$ and $z^{\prime}$ are the respective depths, and $f_{x},f_{y},x_{0},y_{0}$ are the camera intrinsics.

Using $z$, $R$ and $t$ as predicted by deep networks, Eq. 1 is used to warp one video frame onto the other. The result is then compared to the actual other frame, and the differences constitute the main component of the training loss. The premise is that through penalizing the differences, the networks will learn to correctly predict $z$, $R$ and $t$.

Method

In this work we propose simultaneous learning of depth, egomotion, object motion, and camera intrinsics from monocular videos. To accomplish that, we design a motion-prediction network which predicts camera motion, motion of every pixel with respect to the background, and the camera intrinsics: focal lengths, offsets and distortion. A second network predicts depth maps. Through imposing consistency across neighboring frames as a loss, the networks simultaneously learn to predict depth maps, motion fields and the camera intrinsics. We introduce a loss that demands consistency only for pixels that are unoccluded in both frames, where occlusion estimation is done geometrically, based on the depth maps themselves as they are learned. The motion fields are regularized with the help of a mask that indicates pixels that might belong to moving objects, obtained from a pretrained segmentation or an object detection network.

$r_{y}$ and $r_{x}$ are the rotation angles along the respective axes in radians and $w$ and $h$ are the image width and height respectively.

2 Learning object motion

Eq. 1 can propagate frame inconsistency losses to $z$, $R$ and $t$ at every pixel. However without further regularization, the latter trio remain greatly underdetermined. While continuity of $z$, $R$ and $t$ is a powerful regularizer, we found that further regularization helps significantly. In particular, we impose constancy of $R$ throughout the image, and allow $t$ to deviate from a constant value only at pixels that are designated as ‘‘possibly mobile". This mask can be obtained from a pretrained segmentation model. Unlike in prior work , instance segmentation and tracking are not required, as we need a single ‘‘possibly mobile" mask. In fact, we show that a union of bounding boxes is sufficient (see Fig. 2). In addition, an $L1$ smoothing operator is applied on $t$.

3 Occlusion-aware consistency

When the camera moves relatively to a scene, and / or objects move, points in the scene that were visible in one frame may become occluded in another, and vice versa. In pixels that correspond to these points, cross-frame consistency cannot be enforced by a loss. Given a depth map and a motion field in one frame, one could actually detect where occlusion is about to occur, and exclude the occluded areas from the consistency loss.

While detecting occluded pixels is doable, it requires some sort of reasoning about the connectivity of the surface represented by the depth map, and z-buffering. Keeping the mechanism differentiable and efficient, to be suitable for a training loop, may pose a challenge.

We therefore take a different approach, as illustrated in Fig. 3. For each pixel $(i,j)$ in the source frame, the predicted depth $z_{ij}$ and the camera intrinsic matrix are used to obtain the respective point in space, $(x_{ij},y_{ij},z_{ij})$. The point is moved in space according to the predicted motion field. In particular, the depth changes to $z^{\prime}$. The new spatial location is reprojected back onto the camera frame, and falls at some generally-different location ($i^{\prime},j^{\prime}$) on the target frame. $i^{\prime}$ and $j^{\prime}$ are generally non-integer. Therefore obtaining the depth on the target frame at ($i^{\prime},j^{\prime}$), $z^{t}_{i^{\prime},j^{\prime}}$, requires interpolation.

Occlusions happen at ($i^{\prime},j^{\prime}$) where $z^{\prime}$ becomes multivalued. At such points, color and depth consistency should be applied only to the visible branch of $z^{\prime}$, that is, the branch where $z^{\prime}$ is smaller. If the source and target frames are nearly consistent, the visible branch will be close to target depth at ($i^{\prime},j^{\prime}$), $z^{t}_{i^{\prime},j^{\prime}}$. The way we propose to detect that is to include in the losses only points $(i^{\prime},j^{\prime})$ where $z^{\prime}_{i^{\prime},j^{\prime}}\leq z^{t}_{i^{\prime},j^{\prime}}$. In other words, only if a transformed pixel on the source frame lands in front of the depth map in the target frame, do we include that pixel in the loss. This scheme is not symmetrical with respect to interchanging the source and target frames, which is why we always apply it in a symmetrized way: We transform the source onto the tagret, calculate the losses, and then switch the roles of source and target. Fig. 3 illustrates the method. This way of applying losses can be invoked for many types of loss, and we shall refer to it in this paper as ‘‘occlusion-aware" loss or penalty.

4 Networks, losses and regularizations

The motion estimation network is a UNet architecture inspired by FlowNet . A stack of convolutions with stride 2 (the‘‘encoder"), with average pooling in the last one, forms a bottleneck of 1024 channels with a 1x1 spatial resolution. Two 1x1 convolutions with 3 output channels each predict the global rotation angles ($r_{0}$) and the global translation vector ($t_{0}$). The latter two represent the movement of the entire scene with respect to the camera, due to camera motion. Each of the intrinsic parameters is predicted by a 1x1 convolution stemming from the bottleneck, with softplus activations for the focal lengths and distortions. The next layers progressively refine (that is, increase the spatial resolution of) the translation, from a single vector to a residual translation vector field $\delta t(x,y)$, by a factor of 2 in the height and width dimension each time.

It is here where we utilize the foreground mask $m(x,y)$ described in Sec. 4.2: The translation field is expressed as the sum of the global translation vector plus the masked residual translation:

$m(x,y)$ equals one at pixels that could belong to mobile objects and zero otherwise.

Losses

Given a pair of frames, we apply an occlusion-aware L1 penalty for each of the RGB color channels and the depths. For the motion fields, we demand cycle consistency: The rotation and translation at pixel $(i,j)$ of the source frame must form the opposite transform of the ones at $(i^{\prime},j^{\prime})$ of the target frame, and vice versa. Occlusion awareness is invoked here too.

Structural similarity (SSIM) is a crucial component of the training loss, and occlusion awareness as defined above is harder to enforce here, because SSIM involves the neighborhood of each pixel. It is possible that $z^{\prime}\leq z^{t}$ holds for only part of the pixels in the neighborhood. The solution we found here is to weigh the structural similarity loss by a function that falls off where the depth discrepancy between the frames is large compared to the RMS of depth discrepancy.

Randomized layer normalization

Initially our depth prediction network had batch normalization. However we repeatedly observed it leading to anomalous behavior:

Eval accuracy was consistently better when running inference at the ‘‘training mode" of batch normalization. That is, instead of long-term averaged means and variances, the ones obtained from the image itself during inference were usedEven at batch size 1, there would still be means and variances over the spatial dimensions., rendering batch normalization more similar to layer normalization .

As we increased the batch size at training, the eval accuracy was consistently worse and worse, no matter how we scaled the learning rate.

These two observations led us to try replacing batch normalization by layer normalization, but the eval quality degraded. Our next postulate was that while batch normalization is actually acting like layer normalization, the other other items in the batch serve as a source of noise on top of that. To test this theory we replaced batch normalization by layer normalization with Gaussian noise applied on the layer means and variances.

Indeed all eval metrics exhibited significant improvements compared to batch normalization. Moreover, now the eval metrics started to slightly improve when the batch size at training increased (accompanied with a linear increase of the learning rate ), rather than significantly degrading. The best results were obtained with multiplicative noise. While it has been observed in the past that noise can act as a regularizer, for example with dropout or when applied to the gradients , we are not aware of prior work where it is applied with layer normalization.

Experiments

In this section, we evaluate our method on depth prediction, odometry estimation, and the recovery of camera intrinsics across a range of diverse datasets, which will be described next.

The KITTI dataset is collected in urban environments and is the main benchmark for depth and ego-motion estimation. It is accompanied with a LIDAR sensor which is used for

Cityscapes

The Cityscapes dataset is a more recent urban driving dataset, which we use for both training and evaluation. It is a more challenging dataset with many dynamic scenes. With a few exceptions it has not been used for depth estimation evaluation. We use depth from the disparity data for evaluation on a standard evaluation set of 1250 .

EuRoC Micro Aerial Vehicle Dataset

The EuRoC Micro Aerial Vehicle (MAV) Dataset is a very challenging dataset collected by an aerial vehicle indoors. While the data contains a comprehensive suite of sensor measurements, including stereo pairs, IMU, accurate Leica laser tracker ground truth, Vicon scene 3d scans, and camera calibration, we only used the monocular videos for training. Since the camera has significant lens distortion, this is an opportunity to test our method for learning lens distortions (see later sections).

YouTube8M videos

To demonstrate that depth can be learned on videos in the wild from unknown cameras, we collected videos from the YouTube8M dataset . From the 3079 videos in YouTube8M that have the label ‘‘quadcopter", human raters selected videos that contain significant amount of footage from a quadcopter. Naturally, the videos were taken with different unknown cameras, with varying fields of view and varying degrees of lens distortion. The YouTube8M IDs will be made public.

2 Depth

Table 1 summarizes the evaluation results on the KITTI Eigen partition of a model trained on KITTI. The metrics are the ones defined in Zhou et al. . Only the best methods and the first three metrics are displayed in Table 1, the rest are given in the Appendix. As seen in the results, we obtain best state-of-the-art result. More importantly, we observed that learned intrinsics, rather than given ones consistently help performance, as seen too in later experiments.

Cityscapes

Table 2 summarizes the evaluation metrics of models trained and tested on Cityscapes. We follow the established protocol by previous work, using the disparity for evaluation . Since this is a very challenging benchmark with many dynamic objects, very few approaches have evaluated on it. As seen in Table 2, our approach outperforms previous ones and benefits from learned intrinsics.

Cityscapes + KITTI

Being able to learn depth without the need for intrinsics opens up the opportunity for pooling videos from any data sources together. Figure 5 shows the results of pooling Cityscapes and KITTI datasets and evaluating on either one, in this experiment the intrinsincs are assumed unknown and are learned. As seen jointly training improves the depth results even further than the best depth models on each dataset. This is a key result which demonstrates the impact of our proposed method to leverage data sources of potentially unlimited size.

Cityscapes + KITTI: ablation experiments

Table 3 summarizes the results of ablation experiments we ran in order to study the impact of each of the techniques described in this paper on the end results. In order to reduce the number of combinations of results, in all experiments the training set was Cityscapes and KITTI mixed together. Each model was evaluated on both Cityscapes and KITTI separately.

Using a union of bounding boxes as a ‘‘possibly-mobile" mask, as depicted in Fig. 2 is found to be as good as using segmentation masks, which makes our technique more broadly applicable. Object motion estimation is shown to play a crucial role, especially on Cityscapes, which is characterized by more complex scenes with more pedestrians and cars. Randomized layer noralizarion is shown to be superior to standard batch normalization, and at last -- accounting for occlusions significantly improves the quality of depth estimation, especially on Cityscapes, which has richer scenes with more occlusions. Figure A1 visualizes the type of artifacts that occlusion-aware losses reduce.

EuRoC MAV Dataset

We further use the EuRoC MAV Dataset to evaluate depth. We selected also a very challenging out-of-sample evaluation protocol in which we trained on the ‘‘Machine room" sequences and tested on the ‘‘Vicon Room 2 01, which has 3D groundtruth. Table 4 reports the results. The Appendix details how depth ground truth was generated from the provided Vicon 3D scans.

Figure 7 visualizes some of the learned depth. The EuRoC results were obtained from a network trained on the entire EuRoC dataset. Intrinsics are learned, and as seen later are very close to the real ones. No other information is used as input.

3 YouTube Videos

To demonstrate that depth can be learned on collections of videos in the wild for which the camera parameters are not known, and differ across videos, traained our model on the YouTube8M videos described in Sec. 5.1. Figure 8 visualizes the results. We note that this dataset is very challenging as it features objects at large ranges of depth.

4 Camera intrinsics evaluation

In order to evaluate our method with respect to learning camera intrinsics, two separate questions can be asked. First, how good is the supervision signal that cross-frame consistency provides for the camera intrinsics. Second is how good a deep network is in learning them and generalizing to test data.

To estimate the supervision signal for the intrinsics, we represented each intrinsic parameter as a separate learned variable. The model was trained on the EuRoc dataset, since it was captured with the same camera throughout. Each of the 11 subsets of EuRoC was trained in a separate experiment, until the convergence of the intrinsics variables, yielding 11 independent results. The resulting sample mean and standard deviation for each intrinsic parameter are summarized in Table 5. All parameters agree with groundtruth within a few pixels. Since the groundtruth values were not accompanied by tolerances, it is hard to tell if the differences are within tolerance or not.

Learning and generalization

Prior work has shown that deep networks can learn and generalize camera intrinsics in a strongly supervised setting. In our setting, the camera intrinsics and motion are predicted by the same network and are thus correlated. In other words, the losses imposed on the motion / intrinsics network only impose correctness of the intrinsics within the limits of Eq. 3. This is the downside of having the same network predict both quantities, the advantage of which is reduced computational cost.

We evaluated our model’s predictions of the intrinsic parameters on the KITTI odometry series. The model was trained on the union of the Cityscapes and KITTI training sets. Figure 9 shows the scatter plot of the predicted $f_{x}$ as function of the predicted $r_{y}$. The predictions fall within the limits imposed by Eq. 3. While Table 5 indicates a high-quality supervision signal for the intrinsics, Fig. 9 shows that in the setting where the intrinsics and motion are predicted by the same network, the latter learns them ‘‘in aggregate", only to the extent needed for the depth-prediciton network to learn depth.

5 Odometry

We evaluated our egomotion prediction on the KITTI sequences 09 and 10. The common 5-point Absolute Trajectory Error (ATE) metric measures local agreement between the the estimated trajectories and the respective groundtruth. However assessing the usefulness for a method for localization requires evaluating its accuracy in predicting location. A common metric for that is average relative translational drift $t_{rel}$ - the distance between the predicted location and the groundtruth location divided by distance traveled and averaged over the trajectory. Table 6 summarizes both metrics, demonstrating the improvements our method achieves on both.

When evaluating for odometry, the most naive way is to calculate the inference of egomotion for every pair of adjacent frame. That leads to the red ‘‘learned intrinsics" curve in Fig. 10. However it is also possible to make an inference-time correction if we know the intrinsics of the camera at inference time. In that case, one can leverage the fact that for small errors in the rotation angles and focal lengths, $r_{x}f_{y}$ and $r_{y}f_{x}$ are approximately constant. Therefore if the network predicted $r^{\prime}_{y}$ and $f^{\prime}_{x}$ for a given pair of images, and we know the true focal length $f_{x}$, we can correct our estimate of $r_{y}$ to $r^{\prime}_{y}f^{\prime}_{x}/f_{x}$. This is the procedure invoked in generating the ‘‘Learned and corrected intrinsics" curve in Fig. 10, and the respective entry in Table 6. The trajectories and metrics obtained with learned itrinsics with inference time correction and with given intrinsics are similar. Both notably improve prior art, which is especially prominent for localization, as the $t_{rel}$ metric indicates.

Conclusions

This work addresses important challenges for unsupervised learning of depth and visual odometry through geometric handling of occlusions, a simple way of accounting for object motion, and a novel form of regularization. Most importantly, it takes a major step towards leveraging the vast amounts existing unlabeled videos for learning depth estimation: Through unsupervised learning camera intrinsic parameters, including lens distortion, it enables for, the first time, learning depth from raw videos captured by unknown cameras.

Acknowledgements

We Thank Roland Siegwart for the permission to use EuRoC dataset, Sergey Ioffe for fruitful discussions, and Kurt Konolige for consultation and critical reading of the manuscript.

References

A Appendix

according to Eq. (1) the training loss will not be affected. Proving and analyzing this statement is the subject of this section.

A.3 Tolerances

Mathematically, $R=I$ provides no supervision signal for $K$, whereas $R\neq I$ provides a complete supervision siglal for it. Practically, it is clear that the closer $R$ is to $I$, the weaker the supervision on $K$ is. Here we would like to quantify this relation.

For simplicity, suppose we have only rotation around $y$ and no translation. Eq. (1) then reduces to

where $\Delta p_{x}=p^{\prime}_{x}-p_{x}$. $\Delta p_{x}$ is where the supervision signal comes from.

The worst case of Eq. A7 is where $p_{x}=w/2$:

As long as the absolute value of the right hand side of Eq. A8 is $\ll 1$, the worst case error in pixel coordinates is much less than one pixel. This gives the condition

A similar relation can be derived for $\delta f_{y}$. This is how Eq. (3) was established.

A.4 Full tables of metrics for depth estimation

The numbers in Table 1 and 2, as well as in Fig. 5, are given for only part of the metrics commonly published for depth estimation. In this section we give the rest of the metrics, for completeness. Tables A1, A2 and A3 provide the full set of numbers for the former ones, respectively.

A.5 Generating depth groundtruth for the EuRoC dataset

In the EuRoC dataset, the Vicon Room 2 series has pointclouds that were obtained from merging depth sensor readings. In addition, there is groundtruth for the position and orientation of the camera at given timestamps, as well as the intrinsics. For every frame, we reprojected the point clouds onto the camera using the intrinsics and extrinsics. To address occlusions, each point was given some finite angular width. If two 3D points were projected onto close enough locations on the image plane, and their depth ratio was greater than a certain threshold, only the one closer to the camera was kept. Finally, the rendered depth maps were made more uniform by introducing a uniform grid in projective space and keeping at most one point in a each cell. The code performing the above transformation will be released. An example of the result is shown in Fig. 6.

A.6 Intrinsics transformation on the EuRoC dataset

The intrinsics of cam0 in the EuRoC set are $(752,480)$ for the width and height, $458.654$, $457.296$ for the focal lengths in the $x$ and $y$ direction respectively, and $367.215$, $248.375$ for $x_{0}$ and $y_{0}$ respectively. The radial distortion coefficients are -0.28340811 and 0.07395907, and the higher-order coefficients are small. In our experiments, we first center-cropped the images to (704, 448). This does not change the focal lengths nor the distortion coefficients, and changes $x_{0}$ and $y_{0}$ to 343.215, 232.375 respectively. Next, we resized the images to (384, 256), which multiplies all $x$-related parameters by $284/704$, and all $y$-related parameters by $256/448$. The results are in the last column of Table 5.

A.7 Odometry

The KITTI Sequence 10 is shown in Figure A2. Tables A4 and A5 extend Table 6 with more metrics.