An end-to-end TextSpotter with Explicit Alignment and Attention

Introduction

The goal of text spotting is to map an input natural image into a set of character sequences or word transcripts and corresponding location. It has attracted increasing attention in the vision community, due to its numerous potential applications. It has made rapid progress riding on the wave of recent deep learning technologies, as substantiated by recent works . However, text spotting in the wild still remains an open problem, since text instances often exhibit vast diversity in font, scale and orientation with various illumination affects, which often come with a highly complicated background.

Past works in text spotting often consider it as two individual tasks: text detection and word recognition, which are implemented sequentially. The goal of text detection is to precisely localize all text instances (e.g., words) in a natural image, and then a recognition model is processed repeatedly through all detected regions for recognizing corresponding text transcripts. Recent approaches for text detection are mainly extended from general object detectors (such as Faster R-CNN and SSD ) by directly regressing a bounding box for each text instance, or from semantic segmentation methods (e.g., Fully Convolutional Networks (FCN) ) by predicting a text/non-text probability at each pixel. With careful model design and development, these approaches can be customized properly towards this highly domain-specific task, and achieve the state-of-the-art performance . The word recognition can be cast into a sequence labeling problem where convolutional recurrent models have been developed recently . Some of them were further incorporated with an attention mechanism for improving the performance . However, training two tasks separately does not exploit the full potential of convolutional networks, where the convolutional features are not shared. It is natural for us to make a more reliable decision if we clearly understand or recognize the meaning of a word and all characters within it. Besides, it is also possible to introduce a number of heuristic rules and hyper-parameters that are costly to tune, making the whole system highly complicated.

Recent Mask R-CNN incorporates an instance segmentation task into the Faster R-CNN detection framework, resulting in a multi-task learning model that jointly predicts a bounding box and a segmentation mask for each object instance. Our work draws inspiration from this pipeline, but has a different goal of learning a direct mapping between an input image and a set of character sequences. We create a recurrent sequence modeling branch for word recognition within a text detection framework, where the RNN based word recognition is processed in parallel to the detection task.

However, the RNN branch, where the gradients are back-propagated through time, is clearly much more difficult to optimize than the task of bounding box regression in detection. This naturally leads to significant differences in learning difficulties and convergence rates between two tasks, making the model particularly hard to be trained jointly. For example, the magnitude of images for training a text detection model is about $10^{3}$ (e.g., 1000 training images in the ICDAR 2015 ) , but the number is increased significantly by many orders of magnitude when a RNN based text recognition model is trained, such as the 800K synthetic images used in . Furthermore, simply using a set of character sequences as direct supervision may be too abstractive (high-level) to provide meaningful detailed information for training such an integrated model effectively, which will make the model difficult to convergence. In this work, we introduce strong spatial constraints in both word and character levels, which allows the model to be optimized gradually by reducing the search space at each step.

Contributions In this work, we present a single-shot textspotter capable of learning a direct mapping between an input image and a set of character sequences or word transcripts. We propose a solution that combines a text-alignment layer tailed for multi-orientation text detection, together with a character attention mechanism that explicitly encodes strong spatial information of characters into the RNN branch, as shown in Fig. 1. These two technologies faithfully preserve the exact spatial information in both text instance and character levels, playing a key role in boosting the overall performance. We develop a principled learning strategy that allows the two tasks to be trained collaboratively by sharing convolutional features. Our main contributions are described as follows.

Firstly, we develop a text-alignment layer by introducing a grid sampling scheme instead of conventional RoI pooling. It computes fixed-length convolutional features that precisely align to a detected text region of arbitrary orientation, successfully avoiding the negative effects caused by orientation changing and quantization factor of the RoI pooling.

Secondly, we introduce a character attention mechanism by using character spatial information as an addition supervision. This explicitly encodes strong spatial attentions of characters into the model, which allows the RNN to focus on current attentional features in decoding, leading to performance boost in word recognition.

Thirdly, both approaches, together with a new RNN branch for word recognition, are integrated elegantly into a CNN detection framework, resulting in a single model that can be trained in an end-to-end manner. We develop a principled and intuitive learning strategy that allows the two tasks to be trained effectively by sharing features, with fast convergence.

Finally, we show by experiments that word recognition can significantly improve detection accuracy in our model, demonstrating strong complementary nature of them, which is unique to this highly domain-specific application. Our model achieves new state-of-the-art results on the ICDAR2015 in end-to-end recognition of multi-orientation texts, largely outperforming the most recent results in , with improvements of F-measure from $(0.54,0.51,0.47)$ to $(0.82,0.77,0.63)$ in terms of using a strong, weak and generic lexicon. Code is available at https://github.com/tonghe90/textspotter

Related work Here we briefly introduce some related works on text detection, recognition and end-to-end wordspotting.

Scene text detection Recently, some methods cast previous character based detection into direct text region estimation , avoiding multiple bottom-up post-processing steps by taking word or text-line as a whole. Tian et al. modified Faster-RCNN by applying a recurrent structure on the convolution feature maps of the top layer horizontally. The methods in were inspired from . They both explored the framework from generic objects and convert to scene text detection by adjusting the feature extraction process to this domain-specific task. However, these methods are based on prior boxes, which need to be carefully designed in order to fulfill the requirements for training. Methods of direct regression for inclined bounding boxes, instead of offsets to fixed prior boxes, have been proposed recently. EAST designed a fully convolutional network structure which outputs a pixel-wise prediction map for text/non-text and five values for every point of text region, i.e., distances from the current point to the four edges with an inclined angle. He et al. proposed a method to generate arbitrary quadrilaterals by calculating offsets between every point of text region and vertex coordinates.

Scene text recognition With the success of recurrent neural networks on digit recognition and speech translation, a lot of works have been proposed for text recognition. He et al. and Shi et al. treat text recognition as a sequence labeling problem by introducing LSTM and connectionist temporal classification (CTC) into a unified framework. proposed an attention-based LSTM for text recognition, which mainly contains two parts: encoder and decoder. In the encoding stage, text images are transformed into a sequence of feature vectors by CNN/LSTM. Attention weights, indicating relative importance for recognition, will be learned during the decoding stage. However, these weights are totally learned by the distribution of data and no supervision is provided to guide the learning process.

End-to-end wordspotting End-to-end wordspotting is an emerging research area. Previous methods usually try to solve it by splitting the whole process into two independent problems: training two cascade models, one for detection and one for recognition. Detected text regions are firstly cropped from original image, followed by affine transforming and rescaling. Corrected images are repeatedly precessed by recognition model to get corresponding transcripts. However, training errors will be accumulated due to cascading models without sharable features. Li et al. proposed a unified network that simultaneously localizes and recognizes text in one forward pass by sharing convolution features under a curriculum strategy. But the existing RoI pooling operation limits it to detect and recognize only horizontal examples. Busta et al. brought up deep text spotter, which can solve wordspotting of multi-orientation problem. However, the method does not have sharable feature, meaning that the recognition loss of the later stage has no influence on the former localization results.

Single Shot TextSpotter by Joint Detection and Recognition

In this section, we present the details of the proposed textspotter which learns a direct mapping between an input image and a set of word transcripts with corresponding bounding boxes of arbitrary orientations. Our model is a fully convolutional architecture built on the PVAnet framework . As shown in Fig. 2, we introduce a new recurrent branch for word recognition, which is integrated into our CNN model in parallel with the existing detection branch for text bounding box regression. The RNN branch is composed of a new text-alignment layer and a LSTM-based recurrent module with a novel character attention embedding mechanism. The text-alignment layer extracts precise sequence feature within the detected region, preventing encoding irrelevant texts or background information. The character attention embedding mechanism regulates the decoding process by providing more detailed supervisions of characters. Our textspotter directly outputs final results in one shot, without any post-processing step except for a simple non-maximum suppression (NMS).

Network architecture Our model is a fully convolutional architecture inspired by , where a PVA network is utilized as backbone due to its significantly low computational cost. Unlike generic objects, texts often have a much larger variations in both sizes and aspect ratios. Thus it not only needs to preserve local details for small-scale text instances, but also should maintain a large receptive field for very long instances. Inspired by the success in semantic segmentation , we exploit feature fusion by combining convolutional features of conv5, conv4, conv3 and conv2 layers gradually, with the goal of maintaining both local detailed features and high-level context information. This results in more reliable predictions on multi-scale text instances. Size of the top layer is $\frac{1}{4}$ of the input image for simplicity.

Text detection This branch is similar to that of , where a multi-task prediction is implemented at each spatial location on the top convolutional maps, by adopting an Intersection over Union (IoU) loss described in . It contains two sub-branches on the top convolutional layer designed for joint text/non-text classification and multi-orientation bounding boxes regression. The first sub-branch returns a classification map with an equal spatial size of the top feature maps, indicating the predicted text/non-text probabilities using a softmax function. The second sub-branch outputs five localization maps with the same spatial size, which estimate five parameters for each bounding box with arbitrary orientation at each spatial location of text regions. The five parameters represent the distances of the current point to the top, bottom, left and right sides of an associated bounding box, together with its inclined orientation. With these configurations, the detection branch is able to predict a quadrilateral of arbitrary orientation for each text instance. The feature of the detected quadrilateral region is then feed into the RNN branch for word recognition via a text-alignment layer which is described below.

We create a new recurrent branch for word recognition, where a text-alignment layer is proposed to precisely compute fixed-size convolutional features from a quadrilateral region of arbitrary size. The text-alignment layer is extended from RoI pooling which is widely used for general objects detection. The RoI pooling computes a fixed-size convolutional features (e.g., $7\times 7$) from a rectangle region of arbitrary size, by performing quantization operation. It can be integrated into the convolutional layers for in-network region cropping, which is a key component for end-to-end training a detection framework. However, directly applying the RoI pooling to a text region will lead to a significant performance drop in word recognition due to the issue of misalignment.

First, unlike object detection and classification where the RoI pooling computes global features of a RoI region for discriminating an object, word recognition requires more detailed and accurate local features and spatial information for predicting each character sequentially. As pointed out in , the RoI pooling performs quantizations which inevitably introduce misalignments between the original RoI region and the extracted features. Such misalignments have a significant negative effect on predicting characters, particularly on some small-scale ones such as ‘i’, ‘l’.

Second, RoI pooling was designed for a rectangle region which is only capable of localizing horizontal instances. It will make larger misalignments when applied to multi-orientation text instances. Furthermore, a large amount of background information and irrelevant texts are easily encoded when a rectangle RoI region is applied to a highly inclined text instance, as shown in Fig. 3. This severely reduces the performance on RNN decoding process for recognizing sequential characters.

Recent Mask R-CNN considers explicit per-pixel spatial correspondence by introducing RoIAlign pooling . This inspires current work that develops a new text-alignment layer tailored for text instance which is a quadrilateral shape with arbitrary orientation. It provides strong word-level alignment with accurate per-pixel correspondence, which is of critical importance to extract exact text information from the convolutional maps, as shown in Fig. 3.

Specifically, given a quadrilateral region, we first build a sampling grid with size of $h\times w$ on the top convolutional maps. The sampled points are generated with equidistant interval within the region, and the feature vector ($\textbf{v}_{p}$) for a sampled point ($p$) at spatial location $(p_{x},p_{y})$, is calculated via a bilinear sampling as follows,

Where $\textbf{v}_{pi}$ refers to four surrounding points of point $p$, $g(m,n)$ is the bilinear interpolation function and $p_{ix}$ and $p_{iy}$ refer to the coordinates of point $p_{i}$. As presented in , an appealing property of the bilinear sampling is that gradients of the sampled points can be back-propagated through the networks, by using Eq. 2.

Grid sampling, by generating a fixed number of sampling points (e.g., $w=64$, $h=8$ in our experiments), provides an efficient way to compute fixed-size features from a quadrilateral region with arbitrary size and orientation. The bilinear sampling allows for exacting per-pixel alignment, successfully avoiding the quantization factor.

2 Word Recognition with Character Attention

Word recognition module is built on the text-alignment layer, as shown in Fig. 2. Details of this module is presented in Fig. 4, where the input is fixed-size convolutional features output from the text-align pooling layer with size of $w\times h\times C$, where $C$ is the number of convolutional channels. The convolutional features are fed into multiple inception modules and generate a sequence of feature vectors, e.g., 64 $\times$C-dimensional features, as shown in Fig. 4. In the next part, we will briefly introduce attention mechanism and three strategies to enhance attention alignment.

where $h^{d}_{t}$ is the hidden state vector of the decoder at time $t$, computed by:

$g_{t}$ is the context vector, which is calculated as a weighted sum of the input sequence: $g_{t}=\sum_{j=1}^{w}\alpha_{t,j}h^{e}_{j}$. The decoder is ended until it encounters an end-of-sequence ($EOS$). The attention vector is calculated by $\alpha_{t,j}=softmax(e_{t,j})$, where $e_{t,j}=z(h^{d}_{t-1},h^{e}_{j})$ is an alignment factor measuring matching similarity between the hidden state and encoding features $h^{e}_{j}$. However, these attention vectors are learned automatically in the training process without an explicit guidance, giving rise to misalignment problem which severely reduces recognition performance, as shown in Fig.5. To address this problem, we propose new attention alignment and enhancement methods that explicitly encode strong attention of each character.

2.2 Attention Alignment and Enhancement

We introduce a new method which enhance the attention of characters in word recognition. We develop character-alignment mechanism that explicitly encodes strong character information, together with a mask supervision task which provides meaningful local details and spatial information of character for model learning. Besides, an attention position embedding is also presented. It identifies the most significant spot from the input sequence which further enhances the corresponding text features in inference. These technical improvements are integrated seamlessly into a unified framework that is end-to-end trainable. Details of each module are described as follows.

Attention alignment To deal with misalignment issue raised by existing implicit attention models, we propose an attention alignment which explicitly encodes spatial information of characters, by introducing an additional loss as supervision.

Specifically, assuming that $p_{t,1},p_{t,2},...,p_{t,w}$ are central points in each column of the sampling grid. At $t$-th time step, these central points can be calculated by Eq. 5,

Ideally, $\delta_{t}$ should close to the center of current character, $y_{t}$. Without supervision, it is likely to result in misalignment and therefore incorrect sequence labels. Intuitively, we can construct a loss function to describe whether the attention points is focusing on the right location.

where $k_{t}$ is ground truth (GT) coordinates, and $\bar{w}_{t}$ is the GT width of current character, $y_{t}$. Both of them are projected onto the axis of text orientation. $T$ is the number of characters in a sequence. Notice that the distance between the prediction and GT should be normalized by character width, which we found is useful for model convergence.

Position embedding Position embedding was first introduced in , aiming to make the model ‘location aware’ by encoding a one-hot coordinate vector. This is equivalent to adding a varying bias terms. It is difficult to directly apply it to our task, as the size of the feature maps changes according to the size of input image. Instead, we generate a one-hot vector from the attention vector, $u_{k}=\mathop{\arg\min}_{j}\alpha_{t,j}$, which is a fixed-size binary vector (e.g., 64-D). Then, we directly concatenate the one-shot vector with the context vector ($g_{t}$), which forms a new feature representation with additional one-hot attention information. Then the decoder computed in Eq. 3 can be modified as,

Finally, by integrating all these modules into a single model, we obtains an overall loss function including four components,

3 Training Strategy

Training our model in an end-to-end manner is challenging due to a number of difficulties. First, largely different nature of them, e.g., significant differences in learning difficulties and convergence rates. Second, the extremely unbalanced distribution of image data. Our methods require character-level bounding boxes for generating character coordinates and masks. These detailed character annotations are not provided in the standard benchmarks, such as the ICDAR2013 and ICDAR2015 . Although Gupta et al. developed a fast and scalable engine to generate synthetic images of text, providing both word-level and character-level informations, there is still a large gap between realistic and synthesized images, making the trained model difficult to generalize well to real-world images.

We fill this gap by developing a principled training strategy which includes multiple steps. It is able to train multiple tasks collaboratively in our single model, allowing for excellent generalization capability from the synthesized images to real-world data.

Step Two: For the next 80k iterations, we open the detection branch, but still use the GT bounding boxes for the text-align layer, as the detector performs poorly at first, which will be harmful to the already trained recognition branch. The learning rate is set to $2\times 10^{-4}$. During the next 20k iterations, sampling grid are generated from the detection branch. The model is trained end-to-end in this stage.

Step Three: About 3,000 real-world images from the ICDAR 2013 , ICDAR 2015 and Multi-lingual http://rrc.cvc.uab.es/?ch=8&com=introduction datasets are utilized in the next 60k iterations. To enhance generalization ability, data augmentation is employed. We re-scale the images by keeping aspect ratio unchanged, followed by random rotation ranging from $-20^{\circ}$ to $20^{\circ}$, and random cropping 800$\times$800 patches for training. To utilize the character-level supervision, we set the batch size to 4, where an image from synthetic dataset is included. The learning rate remained at $2\times 10^{-4}$. The whole system is implemented by Caffe , with TITAN X GPUs.

Experiments

In this section, we first briefly introduce the datasets we use and the evaluation protocols, followed by thorough comparison of the proposed method with the state-of-the-art along with comprehensive ablation experiments.

Datasets The ICDAR2013 dataset focuses more on horizontal text instances, which contains 229 images for training and 233 images for testing with word-level annotation.

The ICDAR2015 dataset is collected by Google glasses, which has 1,000 images for training and 500 images for testing. Different from previous datasets which are well-captured horizontal English text, it contains texts with more scales, blurring, and orientation.

Multi-lingual scene text datasethttp://rrc.cvc.uab.es/?ch=8&com=introduction is built for developing script-robust text detection methods, which contains about 9,000 images with 9 different kinds of transcriptions. We choose about 2000 of them, identified with ‘Latin’, to train the end-to-end task.

Detection There are two standard protocols for evaluating detection results: DetEval and ICDAR2013 standard . The main difference between the two protocols is that the latter one stress more on individual words while the former can achieve high score even when many words are connected into a line.

End-to-end for detection and recognition The criterion has been used in competition: the evaluation of the results will be based on a single IoU criterion, with a threshold of 50%, and correct transcription. Besides, three dictionaries are also provided for testing reference, i.e., ‘strong’, ‘weak’ and ‘generic’. ‘Strong’ lexicon has 100 entries for every image, and most words appeared in that image are included. ‘Weak’ lexicon contains all the words that appeared in the testing dataset. ’Generic’ lexicon has 90K words. One thing should be noticed that the length of all the words in dictionaries are greater than 3 with symbols and numbers excluded. There are two protocols for evaluation: end-to-end and word-spotting. End-to-end needs to recognize all the words precisely, no matter whether the dictionary contains these strings. On the other hand, word-spotting only examine whether the words in the dictionary appear in images, making it less strict than end-to-end for ignoring symbols, numbers and words whose length is less than 3.

2 Text-alignment vs. RoI Pooling

We first compare the proposed text-alignment with standard RoI pooling. To make fair comparison, the detection part is fixed with ground truth and recognition performance is evaluated on ICDAR2015, which contains text instances of multi-orientation. Due to encoding background information and irrelevant text instances, RoI pooling results in mis-alignment and inaccurate representation of feature sequences. As shown in Tab. 2, the accuracy of recognition with proposed method surpasses standard RoI pooling by a large margin, boosting from 60.7% to 67.6%. All results are evaluated without referring to any lexicon in single scale.

3 Character Attention

Different from traditional attention-based recognition models, where attention weights are automatically learned, we propose a method to regulate the learning process to prevent mis-alignment in the decoding stage. To demonstrate the effectiveness of our proposed method, we conduct two experiments with the detection part fixed. The first one is on VGG synthetic data , where we select 600K for training and 200K for testing. The accuracies of character-level and word-level are evaluated. The method with supervision has accuracy of 0.95 and 0.88 on two protocols, comparing to 0.93 and 0.85 on traditional attention-based method. The other experiment is tested on ICDAR2015 dataset. As is shown in Fig. 5, the proposed method give more accurate character localization than attentional LSTM, leading to about 2% boosting in accuracy.

4 Joint Training vs. Separate Models

We believe that text detection and recognition are not two standalone problems, but highly correlated where each task can benefit from the training of the other. Joint training of two tasks in a unified framework avoids error accumulations among cascade models. As shown in Tab. 3, the task of recognition greatly enhances the performance of detection in terms of recall and precision, leading to a 3% improvement on F-Measure (noting: the detection performances are achieved without referring to recognition results). As can be seen from Fig. 6, joint training makes it more robust to text-like background and complicated text instances. We also provide a comparison with other detection approaches, indicating that our method achieved new state-of-the-art performance on ICDAR2013 and ICDAR2015 datasets.

5 Proposed Method vs. State-of-the-art Methods

End-to-end results on some extremely challenging images are presented in Fig. 7. As can be seen in Fig. 7, our method can correctly detect and recognize both small text instances and those with large inclined angles.

ICDAR2015 The effectiveness to multi-orientation texts is testified on ICDAR2015 dataset. Our method achieved an F-measure of 0.82, 0.77 and 0.63 respectively in terms of referencing ‘Strong’, ‘Weak’ and ‘Generic’ lexicon under the end-to-end protocol, which surpasses the state-of-the-art performance of 0.54, 0.77 and 0.63 by a large margin.

ICDAR2013 The dataset is well-captured for horizontal text instances. The result is shown in Tab. 1, which is comparable to the state-of-the-art result .

Conclusion In this paper we have presented a novel framework that combines detection and recognition in a unified network with sharable features. The model can directly output detection and recognition results of multi-orientation text instances.

We have proposed a novel text-alignment layer that can extract precise sequence information without encoding irrelevant background or texts. We also improve the accuracy of traditional LSTM by enhancing the attention of characters during the decoding process. Our proposed method achieves state-of-the-art performance on two open benchmarks: ICDAR2013 and ICDAR2015 and outperforms previous best methods by a large margin.

Acknowledgments C. Shen’s participation was in part supported by an ARC Future Fellowship.

References