EAST: An Efficient and Accurate Scene Text Detector

Introduction

Recently, extracting and understanding textual information embodied in natural scenes have become increasingly important and popular, which is evidenced by the unprecedented large numbers of participants of the ICDAR series contests and the launch of the TRAIT 2016 evaluation by NIST .

Text detection, as a prerequisite of the subsequent processes, plays a critical role in the whole procedure of textual information extraction and understanding. Previous text detection approaches have already obtained promising performances on various benchmarks in this field. The core of text detection is the design of features to distinguish text from backgrounds. Traditionally, features are manually designed to capture the properties of scene text, while in deep learning based methods effective features are directly learned from training data.

However, existing methods, either conventional or deep neural network based, mostly consist of several stages and components, which are probably sub-optimal and time-consuming. Therefore, the accuracy and efficiency of such methods are still far from satisfactory.

In this paper, we propose a fast and accurate scene text detection pipeline that has only two stages. The pipeline utilizes a fully convolutional network (FCN) model that directly produces word or text-line level predictions, excluding redundant and slow intermediate steps. The produced text predictions, which can be either rotated rectangles or quadrangles, are sent to Non-Maximum Suppression to yield final results. Compared with existing methods, the proposed algorithm achieves significantly enhanced performance, while running much faster, according to the qualitative and quantitative experiments on standard benchmarks.

Specifically, the proposed algorithm achieves an F-score of 0.7820 on ICDAR 2015 (0.8072 when tested in multi-scale), 0.7608 on MSRA-TD500 and 0.3945 on COCO-Text , outperforming previous state-of-the-art algorithms in performance while taking much less time on average (13.2fps at 720p resolution on a Titan-X GPU for our best performing model, 16.8fps for our fastest model).

The contributions of this work are three-fold:

We propose a scene text detection method that consists of two stages: a Fully Convolutional Network and an NMS merging stage. The FCN directly produces text regions, excluding redundant and time-consuming intermediate steps.

The pipeline is flexible to produce either word level or line level predictions, whose geometric shapes can be rotated boxes or quadrangles, depending on specific applications.

The proposed algorithm significantly outperforms state-of-the-art methods in both accuracy and speed.

Related Work

Scene text detection and recognition have been active research topics in computer vision for a long period of time. Numerous inspiring ideas and effective approaches have been investigated. Comprehensive reviews and detailed analyses can be found in survey papers . This section will focus on works that are mostly relevant to the proposed algorithm.

Conventional approaches rely on manually designed features. Stroke Width Transform (SWT) and Maximally Stable Extremal Regions (MSER) based methods generally seek character candidates via edge detection or extremal region extraction. Zhang et al. made use of the local symmetry property of text and designed various features for text region detection. FASText is a fast text detection system that adapted and modified the well-known FAST key point detector for stroke extraction. However, these methods fall behind of those based on deep neural networks, in terms of both accuracy and adaptability, especially when dealing with challenging scenarios, such as low resolution and geometric distortion.

Recently, the area of scene text detection has entered a new era that deep neural network based algorithms have gradually become the mainstream. Huang et al. first found candidates using MSER and then employed a deep convolutional network as a strong classifier to prune false positives. The method of Jaderberg et al. scanned the image in a sliding-window fashion and produced a dense heatmap for each scale with a convolutional neural network model. Later, Jaderberg et al. employed both a CNN and an ACF to hunt word candidates and further refined them using regression. Tian et al. developed vertical anchors and constructed a CNN-RNN joint model to detect horizontal text lines. Different from these methods, Zhang et al. proposed to utilize FCN for heatmap generation and to use component projection for orientation estimation. These methods obtained excellent performance on standard benchmarks. However, as illustrated in Fig. 2(a-d), they mostly consist of multiple stages and components, such as false positive removal by post filtering, candidate aggregation, line formation and word partition. The multitude of stages and components may require exhaustive tuning, leading to sub-optimal performance, and add to processing time of the whole pipeline.

In this paper, we devise a deep FCN-based pipeline that directly targets the final goal of text detection: word or text-line level detection. As depicted in Fig. 2(e), the model abandons unnecessary intermediate components and steps, and allows for end-to-end training and optimization. The resultant system, equipped with a single, light-weighted neural network, surpasses all previous methods by an obvious margin in both performance and speed.

Methodology

The key component of the proposed algorithm is a neural network model, which is trained to directly predict the existence of text instances and their geometries from full images. The model is a fully-convolutional neural network adapted for text detection that outputs dense per-pixel predictions of words or text lines. This eliminates intermediate steps such as candidate proposal, text region formation and word partition. The post-processing steps only include thresholding and NMS on predicted geometric shapes. The detector is named as EAST, since it is an Efficient and Accuracy Scene Text detection pipeline.

A high-level overview of our pipeline is illustrated in Fig. 2(e). The algorithm follows the general design of DenseBox , in which an image is fed into the FCN and multiple channels of pixel-level text score map and geometry are generated.

One of the predicted channels is a score map whose pixel values are in the range of $$. The remaining channels represent geometries that encloses the word from the view of each pixel. The score stands for the confidence of the geometry shape predicted at the same location.

We have experimented with two geometry shapes for text regions, rotated box (RBOX) and quadrangle (QUAD), and designed different loss functions for each geometry. Thresholding is then applied to each predicted region, where the geometries whose scores are over the predefined threshold is considered valid and saved for later non-maximum-suppression. Results after NMS are considered the final output of the pipeline.

2 Network Design

Several factors must be taken into account when designing neural networks for text detection. Since the sizes of word regions, as shown in Fig. 5, vary tremendously, determining the existence of large words would require features from late-stage of a neural network, while predicting accurate geometry enclosing a small word regions need low-level information in early stages. Therefore the network must use features from different levels to fulfill these requirements. HyperNet meets these conditions on features maps, but merging a large number of channels on large feature maps would significantly increase the computation overhead for later stages.

In remedy of this, we adopt the idea from U-shape to merge feature maps gradually, while keeping the up-sampling branches small. Together we end up with a network that can both utilize different levels of features and keep a small computation cost.

A schematic view of our model is depicted in Fig. 3. The model can be decomposed in to three parts: feature extractor stem, feature-merging branch and output layer.

The stem can be a convolutional network pre-trained on ImageNet dataset, with interleaving convolution and pooling layers. Four levels of feature maps, denoted as $f_{i}$, are extracted from the stem, whose sizes are $\frac{1}{32}$, $\frac{1}{16}$, $\frac{1}{8}$ and $\frac{1}{4}$ of the input image, respectively. In Fig. 3, PVANet is depicted. In our experiments, we also adopted the well-known VGG16 model, where feature maps after pooling-2 to pooling-5 are extracted.

In the feature-merging branch, we gradually merge them:

For RBOX, the geometry is represented by 4 channels of axis-aligned bounding box (AABB) $\mathbf{R}$ and 1 channel rotation angle $\theta$. The formulation of $\mathbf{R}$ is the same as that in , where the 4 channels represents 4 distances from the pixel location to the top, right, bottom, left boundaries of the rectangle respectively.

For QUAD $\mathbf{Q}$, we use 8 numbers to denote the coordinate shift from four corner vertices $\{p_{i}\,|\,i\!\!\in\!\!\{1,2,3,4\}\}$ of the quadrangle to the pixel location. As each distance offset contains two numbers $(\Delta x_{i},\Delta y_{i})$, the geometry output contains 8 channels.

3 Label Generation

Without loss of generality, we only consider the case where the geometry is a quadrangle. The positive area of the quadrangle on the score map is designed to be roughly a shrunk version of the original one, illustrated in Fig. 4 (a).

For a quadrangle $\mathbf{Q}=\{p_{i}|i\in\{1,2,3,4\}\}$, where $p_{i}=\{x_{i},y_{i}\}$ are vertices on the quadrangle in clockwise order. To shrink $\mathbf{Q}$, we first compute a reference length $r_{i}$ for each vertex $p_{i}$ as

We first shrink the two longer edges of a quadrangle, and then the two shorter ones. For each pair of two opposing edges, we determine the “longer” pair by comparing the mean of their lengths. For each edge $\langle p_{i},p_{(i\bmod 4)+1}\rangle$, we shrink it by moving its two endpoints inward along the edge by $0.3r_{i}$ and $0.3r_{(i\bmod 4)+1}$ respectively.

3.2 Geometry Map Generation

As discussed in Sec. 3.2, the geometry map is either one of RBOX or QUAD. The generation process for RBOX is illustrated in Fig. 4 (c-e).

For those datasets whose text regions are annotated in QUAD style (e.g., ICDAR 2015), we first generate a rotated rectangle that covers the region with minimal area. Then for each pixel which has positive score, we calculate its distances to the 4 boundaries of the text box, and put them to the 4 channels of RBOX ground truth. For the QUAD ground truth, the value of each pixel with positive score in the 8-channel geometry map is its coordinate shift from the 4 vertices of the quadrangle.

4 Loss Functions

where $L_{\text{s}}$ and $L_{\text{g}}$ represents the losses for the score map and the geometry, respectively, and $\lambda_{\text{g}}$ weighs the importance between two losses. In our experiment, we set $\lambda_{\text{g}}$ to 1.

In most state-of-the-art detection pipelines, training images are carefully processed by balanced sampling and hard negative mining to tackle with the imbalanced distribution of target objects . Doing so would potentially improve the network performance. However, using such techniques inevitably introduces a non-differentiable stage and more parameters to tune and a more complicated pipeline, which contradicts our design principle.

To facilitate a simpler training procedure, we use class-balanced cross-entropy introduced in , given by

where $\mathbf{\hat{Y}}=F_{\text{s}}$ is the prediction of the score map, and $\mathbf{Y^{*}}$ is the ground truth. The parameter $\beta$ is the balancing factor between positive and negative samples, given by

This balanced cross-entropy loss is first adopted in text detection by Yao et al. as the objective function for score map prediction. We find it works well in practice.

4.2 Loss for Geometries

One challenge for text detection is that the sizes of text in natural scene images vary tremendously. Directly using L1 or L2 loss for regression would guide the loss bias towards larger and longer text regions. As we need to generate accurate text geometry prediction for both large and small text regions, the regression loss should be scale-invariant. Therefore, we adopt the IoU loss in the AABB part of RBOX regression, and a scale-normalized smoothed-L1 loss for QUAD regression.

For the AABB part, we adopt IoU loss in , since it is invariant against objects of different scales.

where $\hat{\mathbf{R}}$ represents the predicted AABB geometry and $\mathbf{R}^{*}$ is its corresponding ground truth. It is easy to see that the width and height of the intersected rectangle $|\mathbf{\hat{R}}\cap\mathbf{R^{*}}|$ are

where $d_{1}$, $d_{2}$, $d_{3}$ and $d_{4}$ represents the distance from a pixel to the top, right, bottom and left boundary of its corresponding rectangle, respectively. The union area is given by

Therefore, both the intersection/union area can be computed easily. Next, the loss of rotation angle is computed as

where $\hat{\theta}$ is the prediction to the rotation angle and $\theta^{*}$ represents the ground truth. Finally, the overall geometry loss is the weighted sum of AABB loss and angle loss, given by

Where $\lambda_{\theta}$ is set to $10$ in our experiments.

Note that we compute $L_{\text{AABB}}$ regardless of rotation angle. This can be seen as an approximation of quadrangle IoU when the angle is perfectly predicted. Although it is not the case during training, it could still impose the correct gradient for the network to learn to predict $\mathbf{\hat{R}}$.

We extend the smoothed-L1 loss proposed in by adding an extra normalization term designed for word quadrangles, which is typically longer in one direction. Let all coordinate values of $\mathbf{Q}$ be an ordered set

and $P_{\mathbf{Q}}$ is the set of all equivalent quadrangles of $\mathbf{Q}^{*}$ with different vertices ordering. This ordering permutation is required since the annotations of quadrangles in the public training datasets are inconsistent.

5 Training

The network is trained end-to-end using ADAM optimizer. To speed up learning, we uniformly sample 512x512 crops from images to form a minibatch of size 24. Learning rate of ADAM starts from 1e-3, decays to one-tenth every 27300 minibatches, and stops at 1e-5. The network is trained until performance stops improving.

6 Locality-Aware NMS

To form the final results, the geometries survived after thresholding should be merged by NMS. A naïve NMS algorithm runs in $O(n^{2})$， where $n$ is the number of candidate geometries, which is unacceptable as we are facing tens of thousands of geometries from dense predictions.

Under the assumption that the geometries from nearby pixels tend to be highly correlated, we proposed to merge the geometries row by row, and while merging geometries in the same row, we will iteratively merge the geometry currently encountered with the last merged one. This improved technique runs in $O(n)$ in best scenariosConsider the case that only a single text line appears the image. In such case, all geometries will be highly overlapped if the network is sufficiently powerful. Even though its worst case is the same as the naïve one, as long as the locality assumption holds, the algorithm runs sufficiently fast in practice. The procedure is summarized in Algorithm 1

In fact, there is a subtle difference that we are ”averaging” rather than ”selecting” geometries, as in a standard NMS procedure will do, acting as a voting mechanism, which in turn introduces a stabilization effect when feeding videos. Nonetheless, we still adopt the word ”NMS” for functional description.

Experiments

To compare the proposed algorithm with existing methods, we conducted qualitative and quantitative experiments on three public benchmarks: ICDAR2015, COCO-Text and MSRA-TD500.

ICDAR 2015 is used in Challenge 4 of ICDAR 2015 Robust Reading Competition . It includes a total of 1500 pictures, 1000 of which are used for training and the remaining are for testing. The text regions are annotated by 4 vertices of the quadrangle, corresponding to the QUAD geometry in this paper. We also generate RBOX output by fitting a rotated rectangle which has the minimum area. These images are taken by Google Glass in an incidental way. Therefore text in the scene can be in arbitrary orientations, or suffer from motion blur and low resolution. We also used the 229 training images from ICDAR 2013.

COCO-Text is the largest text detection dataset to date. It reuses the images from MS-COCO dataset . A total of 63,686 images are annotated, in which 43,686 are chosen to be the training set and the rest 20,000 for testing. Word regions are annotated in the form of axis-aligned bounding box (AABB), which is a special case of RBOX. For this dataset, we set angle $\theta$ to zero. We use the same data processing and test method as in ICDAR 2015.

MSRA-TD500 is a dataset comprises of 300 training images and 200 test images. Text regions are of arbitrary orientations and annotated at sentence level. Different from the other datasets, it contains text in both English and Chinese. The text regions are annotated in RBOX format. Since the number of training images is too few to learn a deep model, we also harness 400 images from HUST-TR400 dataset as training data.

2 Base Networks

As except for COCO-Text, all text detection datasets are relatively small compared to the datasets for general object detection, therefore if a single network is adopted for all the benchmarks, it may suffer from either over-fitting or under-fitting. We experimented with three different base networks, with different output geometries, on all the datasets to evaluate the proposed framework. These networks are summarized in Tab. 2.

VGG16 is widely used as base network in many tasks to support subsequent task-specific fine-tuning, including text detection . There are two drawbacks of this network: (1). The receptive field for this network is small. Each pixel in output of conv5_3 only has a receptive field of 196. (2). It is a rather large network.

PVANET is a light weight network introduced in , aiming as a substitution of the feature extractor in Faster-RCNN framework. Since it is too small for GPU to fully utilizes computation parallelism, we also adopt PVANET2x that doubles the channels of the original PVANET, exploiting more computation parallelism while running slightly slower than PVANET. This is detailed in Sec. 4.5. The receptive field of the output of the last convolution layer is 809, which is much larger than VGG16.

The models are pre-trained on the ImageNet dataset .

3 Qualitative Results

Fig. 5 depicts several detection examples by the proposed algorithm. It is able to handle various challenging scenarios, such as non-uniform illumination, low resolution, varying orientation and perspective distortion. Moreover, due to the voting mechanism in the NMS procedure, the proposed method shows a high level of stability on videos with various forms of text instancesOnline video: https://youtu.be/o5asMTdhmvA. Note that each frame in the video is processed independently..

The intermediate results of the proposed method are illustrated in Fig. 6. As can be seen, the trained model produces highly accurate geometry maps and score map, in which detections of text instances in varying orientations are easily formed.

4 Quantitative Results

As shown in Tab. 3 and Tab. 4, our approach outperforms previous state-of-the-art methods by a large margin on ICDAR 2015 and COCO-Text.

In ICDAR 2015 Challenge 4, when images are fed at their original scale, the proposed method achieves an F-score of 0.7820. When tested at multiple scales At relative scales of 0.5, 0.7, 1.0, 1.4, and 2.0. using the same network, our method reaches 0.8072 in F-score, which is nearly 0.16 higher than the best method in terms of absolute value (0.8072 vs. 0.6477).

Comparing the results using VGG16 network, the proposed method also outperforms best previous work by 0.0924 when using QUAD output, 0.116 when using RBOX output. Meanwhile these networks are quite efficient, as will be shown in Sec.4.5.

In COCO-Text, all of the three settings of the proposed algorithm result in higher accuracy than previous top performer . Specifically, the improvement over in F-score is 0.0614 while that in recall is 0.053, which confirm the advantage of the proposed algorithm, considering that COCO-Text is the largest and most challenging benchmark to date. Note that we also included the results from as reference, but these results are actually not valid baselines, since the methods (A, B and C) are used in data annotation.

The improvements of the proposed algorithm over previous methods prove that a simple text detection pipeline, which directly targets the final goal and eliminating redundant processes, can beat elaborated pipelines, even those integrated with large neural network models.

As shown in Tab. 5, on MSRA-TD500 all of the three settings of our method achieve excellent results. The F-score of the best performer (Ours+PVANET2x) is slightly higher than that of . Compared with the method of Zhang et al. , the previous published state-of-the-art system, the best performer (Ours+PVANET2x) obtains an improvement of 0.0208 in F-score and 0.0428 in precision.

Note that on MSRA-TD500 our algorithm equipped with VGG16 performs much poorer than that with PVANET and PVANET2x (0.7023 vs. 0.7445 and 0.7608), the main reason is that the effective receptive field of VGG16 is smaller than that of PVANET and PVANET2x, while the evaluation protocol of MSRA-TD500 requires text detection algorithms output line level instead of word level predictions.

In addition, we also evaluated Ours+PVANET2x on the ICDAR 2013 benchmark. It achieves 0.8267, 0.9264 and 0.8737 in recall, precision and F-score, which are comparable with the previous state-of-the-art method , which obtains 0.8298, 0.9298 and 0.8769 in recall, precision and F-score, respectively.

5 Speed Comparison

The overall speed comparison is demonstrated in Tab. 6. The numbers we reported are averages from running through 500 test images from the ICDAR 2015 dataset at their original resolution (1280x720) using our best performing networks. These experiments were conducted on a server using a single NVIDIA Titan X graphic card with Maxwell architecture and an Intel E5-2670 v3 @ 2.30GHz CPU. For the proposed method, the post-processing includes thresholding and NMS, while others should refer to their original paper.

While the proposed method significantly outperforms state-of-the-art methods, the computation cost is kept very low, attributing to the simple and efficient pipeline. As can be observed from Tab. 6, the fastest setting of our method runs at a speed of 16.8 FPS, while slowest setting runs at 6.52 FPS. Even the best performing model Ours+PVANET2x runs at a speed of 13.2 FPS. This confirm that our method is among the most efficient text detectors that achieve state-of-the-art performance on benchmarks.

6 Limitations

The maximal size of text instances the detector can handle is proportional to the receptive field of the network. This limits the capability of the network to predict even longer text regions like text lines running across the images.

Also, the algorithm might miss or give imprecise predictions for vertical text instances as they take only a small portion of text regions in the ICDAR 2015 training set.

Conclusion and Future Work

We have presented a scene text detector that directly produces word or line level predictions from full images with a single neural network. By incorporating proper loss functions, the detector can predict either rotated rectangles or quadrangles for text regions, depending on specific applications. The experiments on standard benchmarks confirm that the proposed algorithm substantially outperforms previous methods in terms of both accuracy and efficiency.

Possible directions for future research include: (1) adapting the geometry formulation to allow direct detection of curved text; (2) integrating the detector with a text recognizer; (3) extending the idea to general object detection.

References