Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework

Introduction

Modern intelligent systems rely more and more on visual information as input. However, in an outdoors setting, visual input quality and in turn, system performance, could be seriously degraded by atmospheric turbulences . One such turbulence, rain streaks, degrade image contrast and visibility, obscure scene features, and could be misconstrued as scene motion by computer vision algorithms. Rain removal is therefore vital to ensure the robustness of outdoor vision-based systems.

There are two categories of methods for rain removal – image-based methods, which rely solely on the information of the processed frame, and video-based methods, which also utilize temporal clues from neighboring frames. Due to the lack of temporal information, image-based methods face difficulties in recovering from torrential rain with large and opaque occlusions.

To properly utilize temporal information, video-based methods require scene content to be aligned throughout consecutive frames. However, this requirement is challenging due to two factors – motion of the camera and dynamic scene content, i.e., presence of moving object. Previous works tackle these two issues separately. Camera motion-induced scene content shifts can be reversed using global frame alignment . However, the granularity of global alignment is too large when scene depth range is large; parts of scene content will be poorly aligned. Scene content shifts due to scene object motions could cause the scene to be misclassified as rain. One solution is to identify and exclude these pixels. This approach, however, is unable to remove rain that overlaps moving objects.

In this paper, we propose a novel and elegant framework that simultaneously solves both issues for the video-based approach – rain removal based on robust SuperPixel (SP) Alignment between video frames followed by detail Compensation in a CNN framework (SPAC-CNN). First, the target video frame is segmented into SPs and each SP is aligned with its temporal neighbors. This step simultaneously aligns both the scene background and moving objects without prior assumptions about moving objects. Scene content is also much better aligned at a SP level granularity. An intermediate derain output can be obtained by averaging the aligned SPs, which unavoidably introduces blurring. We restore the rain free details to the intermediate output by extracting the information from the aligned SPs using a convolutional neural network (CNN).

Extensive experiments show that our proposed algorithm achieves up to 5dB reconstruction advantage over state-of-the-art rain removal methods. Visual inspection shows that rain is much better removed, especially for heavy and opaque rainfall regions over highly dynamic scene content. Fig. 1 illustrates the advantage of our proposed algorithm over existing methods in a challenging video sequence. The contribution of this work can be generalized as follows:

We propose a novel spatial-temporal content alignment algorithm at SP level, which can handle fast camera motion and dynamic scene contents in one framework. This mechanism greatly outperforms existing scene motion analysis methods that models background and foreground motion separately.

The strong local properties of SPs can robustly counter heavy rain interferences, and facilitate much more accurate alignment. Owing to such robust alignment, accurate temporal correspondence could be established for rain occlusions such that heavily occluded backgrounds could be truthfully restored. This greatly outperforms image-based derain methods in which recovery of large and opaque rain occlusions remain the biggest challenge.

We propose a set of very efficient spatial-temporal features for the compensation of high frequency details lost during the deraining process. An efficient CNN network is designed, and a synthetic rain video dataset is created for training the CNN.

Related Work

Rain removal based on a single image is intrinsically a challenging one, since it only relies on visual features and priors to distinguish rain from the background. Local photometric, geometric, and statistical properties of rain have been studied in . Li et al. models background and rain streaks as layers to be separated. Under the sparse coding framework, rain and backgrounds can be efficiently separated either with classified dictionary atoms , or via discriminative sparse coding . Convolutional Neural Networks have been very effective in both high-level vision tasks and low-level vision applications for capturing signal characteristics . Hence, different network structures and features were explored for rain removal, such as the deep detail network , and the joint rain detection and removal model . Due to the lack of temporal information, heavy and opaque rain is difficult to be distinguished from scene structures. Full recovery of a seriously occluded scene is almost impossible.

The temporal information from a video sequence provides huge advantage for rain removal . True rain pixels are separated from moving object pixels based on statistics of intensity values or chromatic values , on geometric properties of connected candidate pixels , or on segmented motion regions . Kim’s work compensates for scene content motion by using optical flow for content alignment. Ren et al. decomposes a video into background, rain, and moving objects using matrix decomposition. Moving objects are derained by temporally aligning them using patch matching, while the moving camera effect is modeled using a frame transform variable. Temporal derain methods can handle occlusions much better than image-based methods; however, these methods perform poorly for complex dynamic scenes shot from fast moving cameras.

Proposed Model

Throughout the paper, scalars are denoted by italic lower-case letters, 2D matrices by upper-case letters, 3D tensors, functions, and operators by script letters.

Given a target derain video frame $I_{0}$, we look at its immediate past and future neighbor frames to create a sliding buffer window of length $n_{t}$: $\{I_{i}|i=[-\frac{n_{t}-1}{2},\frac{n_{t}-1}{2}]\}$. Here, negative and positive $i$ indicate past and future frames, respectively. We only derain the Y luminance channel. The derain output is used to update the history buffer (Fig. 2). Such history update mechanism ensures cleaner derain for heavy rainfall scenarios.

The system diagram for the proposed SPAC-CNN rain removal algorithm is shown in Fig. 2. The algorithm can be divided into two parts: first, video content alignment is carried out at SP level, which consists two SP template matching operations that produce two output tensors: the optimal temporal match tensor $\mathcal{T}_{0}$, and the sorted spatial-temporal match tensor $\mathcal{T}_{1}$. An intermediate derain output $X_{\text{avg}}$ is calculated by averaging the slicesA slice is a two-dimensional section of a higher dimensional tensor, defined by fixing all but two indices . of the tensor $\mathcal{T}_{1}$. Second, these two tensors will be prepared as input features to a CNN to compensate the high frequency details lost in $X_{\text{avg}}$ caused by mis-alignment blur. The detail of each component will be explained in this section.

One of the most important procedure for video-based derain algorithms is the estimation of content correspondence between video frames. With accurate content alignment, rain occlusions could be easily detected and removed with information from the temporal axis.

The popular solution to compensate camera motion between two frames is via a homography transform matrix estimated based on global consensus of a group of matched feature points . Due to the reasons analyzed in Sec. 1, perfect content alignment can never be achieved for all pixels with a global transform at whole frame level, especially for dynamic scenes with large depth range.

The solution naturally turns to pixel-level alignment, which faces no fewer challenges: first, feature points are sparse, and feature-less regions are difficult to align; more importantly, rain streak occlusions will cause serious interferences to feature matching at single pixel level. Information from larger areas are required to overcome rain interferences. This lead us to our final solution: to decompose images into smaller depth consistent units.

The concept of SuperPixel (SP) is to group pixels into perceptually meaningful atomic regions . Boundaries of SP usually coincide with those of the scene contents. Comparing Fig. 3(a) and (b), the SPs are very adaptive in shape, and are more likely to segment uniform depth regions compared with rectangular units. We adopt SP as the basic unit for content alignment.

1.2 Optimal Temporal Matching for Rain Detection

Pixels within the same SP are very likely to belong to the same object and possess identical motion between adjacent frames. Therefore, we can approximate the SP appearance in their adjacent frames based on its appearance in the current frame via linear translations.

Searching for the reference SP is done by template matching of the target SP at all candidate locations in $\mathcal{B}_{k}$. A match location is found at frame $I_{t^{\prime}}$ according to:

A visual demonstration of $\hat{M}_{\text{rain}}$, $M_{\text{e}}$, and $M_{\text{rain}}$ is shown in Fig. 4(a), (b), and (c), respectively. In our implementation, $\epsilon_{\text{rain}}$ is set to $0.012$ while $\epsilon_{\text{edge}}$ is set to $0.2$.

1.3 Sorted Spatial-Temporal Template Matching for Rain Occlusion Suppression

The second round of template matching will be carried out based on the following cost function:

The rain-free matching template $M_{\text{RSP}}$ is calculated as:

The slices of $\mathcal{T}_{1}(\cdot,\cdot,t)$ are expected to be well-aligned to the current target SP $\mathcal{P}_{k}$, and is robust to interferences from the rain. Since rain pixels are temporally randomly and sparsely distributed within $\mathcal{T}_{1}$, when $n_{st}$ is sufficiently large, we can get a good estimation of the rain free image through tensor slice averaging, which functions to suppress rain induced intensity fluctuations, and bring out the occluded background pixels:

Fig. 5 gives a visual example of $X_{\text{avg}}$ and its calculation flow. We can see that all rain streaks have been suppressed in $X_{\text{avg}}$ after the averaging.

2 Detail Compensation for Mis-Alignment Blur

The averaging of $\mathcal{T}_{1}$ slices provides a good estimation of rain free image; however, it creates noticeable blur due to un-avoidable mis-alignment, especially when the camera motion is fast. To compensate the lost high frequency content details without reintroducing the rain streaks, we propose to use a CNN model for the task.

Note that the feature $\mathcal{F}_{1}$ itself is already a reasonable derain output. However its quality is greatly limited by the correctness of the rain mask $M_{\text{rain}}$. For false positiveFalse positive rain pixels refer to background pixels falsely classified as rain; false negative rain pixels refer to rain pixels falsely classified as background. rain pixels, $X_{\text{avg}}$ will introduce content detail loss; for false negative pixels, rain streaks will be added back from $X_{k}$. This calls for more informative features.

2.2 Temporal Consistency Feature

The temporal consistency feature is designed to handle false negative rain pixels in $M_{\text{rain}}$, which falsely add rain streaks back to $\mathcal{F}_{1}$. For a correctly classified and recovered pixel (a.k.a. true positive) in Eq. (9), intensity consistency should hold such that for the collocated pixels in the neighboring frames, there are only positive intensity fluctuations caused by rain in those frames. Any obvious negative intensity drop along the temporal axis is a strong indication that such pixel is a false negative pixel.

2.3 High Frequency Detail Feature

In order to facilitate the network training, we limit the mapping range between the input features and regression output by removing the low frequency component ($X_{\text{avg}}$) from these input features. Pixels in $X_{k}$ but outside of the SP $\mathcal{P}_{k}$ is masked out with $M_{\text{SP}}$:

The final input feature set is $\{\hat{\mathcal{F}}_{1},~{}\hat{\mathcal{F}}_{2},~{}\hat{\mathcal{F}}_{3}\}$. The feature preparation process is summarized in Fig. 5.

2.4 CNN Structure and Training Details

For the CNN training, we minimize the $\mathcal{L}_{2}$ distance between the derain output and the ground truth scene:

here $\hat{X}$ denotes the ground truth clean image. We use stochastic gradient descent (SGD) to minimize the objective function. Mini-batch size is set as 50 for better trade-off between speed and convergence. The Xavier approach is used for network initialization, and the ADAM solver is adpatoed for system training, with parameter settings $\beta_{1}=$ 0.9, $\beta_{2}=$ 0.999, and learning rate $\alpha=$ 0.0001.

To create the training rain dataset, we first took a set of 8 rain-free VGA resolution video clips of various city and natural scenes. The camera was of diverse motion for each clip, e.g., panning slowly with unstable movements, or mounted on a fast moving vehicle with speed up to 30 km/h. Next, rain was synthesized over these video clips with the commercial editing software Adobe After Effects , which can create realistic synthetic rain effect for videos with adjustable parameters such as raindrop size, opacity, scene depth, wind direction, and camera shutter speed. This provides us a diverse rain visual appearances for the network training.

We synthesized 3 to 4 different rain appearances with different synthetic parameters over each video clip, which provides us 25 rainy scenes. For each scene, 21 frames were randomly extracted (together with their immediate buffer window for calculating features). Each scene was segmented into approximately 300 SPs, therefore finally we have around 157,500 patches in the training dataset.

Performance Evaluation

We set the sliding video buffer window size $n_{t}=5$. Each VGA resolution frame was segmented into around 300 SPs using the SLIC method . The bounding box size was $n_{x}=$ 80, and the spatial-temporal buffer $\mathcal{B}_{k}$ dimension was $n_{\text{s}}\times n_{\text{s}}\times n_{\text{t}}=$ 30$\times$30$\times$5. MatConvNet was adopted for model training, which took approximately 54 hours to converge over the training dataset introduced in Sec. 3.2.4. The training and all subsequent experiments were carried out on a desktop with Intel E5-2650 CPU, 56GB RAM, and NVIDIA GeForce GTX 1070 GPU.

To quantitatively evaluate our proposed algorithm, we took a set of 8 videos (different from the training set), and synthesized rain over these videos with varying parameters. Each video is around 200 to 300 frames. All subsequent results shown for each video are the average of all frames.

To test the algorithm performance in handling cameras with different motion, we divided the 8 testing scenes into two groups: Group a consists of scenes shot from a panning and unstable camera; Group b from a fast moving camera (with speed range between 20 to 30 km/h). Thumbnails and the labeling of each testing scene are shown in Fig. 7.

Four state-of-the-art methods were chosen for comparison: two image-based derain methods, i.e., discriminative sparse coding (DSC) , and the deep detail network (DDN) ; one video-based method via matrix decomposition (VMD) . The intermediate derain output $X_{\text{Avg}}$ is also used as a baseline (abbr. as SPAC-Avg).

Rain fall introduces edges and textures over the background. To evaluate how much of the modifications from the derain algorithm contributes positively to only removing the rain pixels, we calculated the rain streak edge precision-recall (PR) curves. Absolute difference values were calculated between the derain output against the scene ground truth. Different threshold values were applied to retrieve a set of binary maps, which were next compared against the ground truth rain pixel map to calculate the precision recall rates.

Average PR curves for the two groups of testing scenes by different algorithms are shown in Fig. 10. As can be seen, for both Group a and b, SPAC-CNN shows consistent advantages over SPAC-Avg, which proves that the CNN model can efficiently compensate scene content details and suppress influences from rain streak edges.

Video-based derain methods (i.e., VMD and SPAC-CNN) perform better than image-based methods (i.e., DSC and DDN) for scenes in Group a. With slow camera motion, temporal correspondence can be accurately established, which brings great advantage to video-based methods. However, with fast camera motion, the performance of VMD deteriorates seriously for Group b data: rain removal is now at the cost of background distortion. Image-based methods show its relative efficiency in this scenario. However, SPAC-CNN still holds advantage over image-based methods at all recall rates for Group b data, which shows its robustness for fast moving camera.

1.2 Scene Reconstruction PSNR/SSIM

We calculated the reconstruction PSNR/SSIM between different state-of-the-art methods against the ground truth, and the results are shown in Table 1. The F-measure for rain streak edge PR curves are also listed for each data.

As can be seen, SPAC-CNN is consistently 5 dB higher than SPAC-Avg for both Groups a and b. SSIM is also at least 0.06 higher. This further validates the efficiency of the CNN detail compensation network.

Video based methods (VMD and SPAC-CNN) show great advantages over image-based methods for Group a data (around 2dB and 5dB higher respectively than DSC). For Group b, image-based methods excel VMD, however SPAC-CNN still hold a 3dB advantage over DDN, 4dB over DSC.

1.3 Feature Evaluation

We evaluated the roles different input features play in the final derain PSNR over two testing data a1 and b4. Three baseline CNNs with different combinations of features as input were independently trained for this evaluation. As can be seen from the results in Table. 2, combination of the three features $\hat{\mathcal{F}}_{1}+\hat{\mathcal{F}}_{2}+\hat{\mathcal{F}}_{3}$ provides the highest PSNR. $\mathcal{F}_{1}$ proves to be the most important feature. Visual inspection on the derain output show both $\hat{\mathcal{F}}_{2}$+$\hat{\mathcal{F}}_{3}$ and $\hat{\mathcal{F}}_{1}$+$\hat{\mathcal{F}}_{3}$ leaves significant amount of un-removed rain. Comparing the last two columns, it can be seen that $\hat{\mathcal{F}}_{3}$ works more efficiently with a1 than b4, which makes sense since the high frequency features are better aligned for slow cameras, which led to more accurate detail compensation.

2 Visual Comparison

We carried out visual comparison to examine the derain performance of different algorithms. Fig. 8 shows the derain output for the testing data a.3, b.1, and b.4. Two consecutive frames are shown for b.1 and b.4 to demonstrate the camera motion. As can be seen, image-based derain methods can only handle well light and transparent rain occlusions. For those opaque rain streaks that cover a large area, they fail unavoidably. Temporal information proves to be critical in truthfully restoring the occluded details.

It is observed that rain can be much better removed by video-based methods. However the VMD method creates serious blur when the camera motion is fast. The derain effect for SPAC-CNN is the most impressive for all methods. The red dotted rectangles showcase the restored high frequency details between SPAC-CNN and SPAC-Avg.

Although the network has been trained over synthetic rain data, experiments show that it generalizes well to real world rain. Fig. 9 shows the derain results. As can be seen, the advantage of SPAC-CNN is very obvious under heavy rain, and robust to fast camera motion.

3 Execution Efficiency

We compared the average runtime between different methods for deraining a VGA resolution frame. Results are shown in Table 3. As can be seen SPAC-Avg is much faster than all other methods. SPAC-CNN is much faster than video-based method, and it’s comparable to that of DDN.

Discussion

For SPAC-CNN, the choice of SP as the basic operation unit is key to its performance. When other decomposition units are used instead (e.g., rectangular), matching accuracy deteriorates, and very obvious averaging blur will be introduced especially at object boundaries.

Although the SP template matching can only handle translational motion, alignment errors caused by other types of motion such as rotation, scaling, and non-ridge transforms can be mitigated with global frame alignment before they are buffered (as shown in Fig. 2) . Furthermore, these errors can be efficiently compensated by the CNN.

When camera moves even faster, SP search range $n_{s}$ needs to be enlarged accordingly, which increases computation loads. We have tested scenarios with camera speed going up to 50 km/h, the PSNR becomes lower due to larger mis-alignment blur, alignment error is also possible as showcased in blue rectangles in Fig. 9. We believe a re-trained CNN with training data from such fast moving camera will help improve the performance.

Conclusion

We have proposed a video-based rain removal algorithm that can handle torrential rain fall with opaque streak occlusions from a fast moving camera. SP have been utilized as the basic processing unit for content alignment and occlusion removal. A CNN has been designed and trained to efficiently compensate the mis-alignment blur introduced by deraining operations. The whole system shows its efficiency and robustness over a series of experiments which outperforms state-of-the-art methods significantly.

Acknowledgment

The research was partially supported by the ST Engineering-NTU Corporate Lab through the NRF corporate lab@university scheme.

References