Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale

Introduction

As mobile apps become prevalent in people’s daily life, there has been a growing research interest in developing various machine learning applications based on mobile screens, e.g., UI component detection (Zhang et al., 2021), screen embedding (Li et al., 2021a), widget captioning (Li et al., 2020b), icon annotation (Zang et al., 2021) and screen summarization (Wang et al., 2021), which enhance the interactive capability and accessibility of mobile phones. These tasks usually rely on the screen layoutWe use ”layout” and ”view hierarchy” interchangeably in this paper., i.e., the tree structure representation underlying the UI that contains the objects on the screen. Each object comes with a set of attributes such as its bounding box and type, which are often used either as input signal or output labels. For example, the widget captioning task (Li et al., 2020b) takes in the cropped pixels of the objects based on their bounding boxes and their attributes including types from the view hierarchy, and outputs a brief caption describing the functionality of the object.

However, UI layouts in existing datasets are often noisy. We have observed two major issues with these UI layouts. First, they are typically captured from the rendering tree or the accessibility tree of the UI at runtime. Similar to taking a screenshot, the results can be dynamic and out of sync. As a result, there can be invisible objects with bounding boxes with no visual correspondences on the screen, misaligned objects with bounding boxes only partially covering the rendered objects, or objects in the background that are grayed out and not clickable. We refer to these objects in a layout as invalid objects in the paper. In the Rico dataset (Deka et al., 2017) we analyzed and annotated, about 37.4% of the screens contain invalid objects. Second, the types of the objects in these captured view hierarchies can be either too generic, e.g., the View class in Androidhttps://developer.android.com/reference/android/view/View, or too app-specific, e.g., ColombiaNativeAdView, to be meaningful. Such types convey little information about the object for machine learning or data science tasks. In fact, the number of different types is virtually unlimited, which makes rule-based approaches hard to implement and extremely challenging to generalize. In the Rico dataset alone, we counted $9,331$ different types (view classes) in 59,555 view hierarchies. For comparison, the heuristic approach proposed in (Liu et al., 2018) only handles and maps $46$ such types to a semantic label.

The invalid objects and the problematic object types are not helpful as either input signal or output labels, and might even harm the model performance. For example, there has been recent interest in developing screen recognition/parser models (Zhang et al., 2021; Wu et al., 2021) using the screenshot as input only. The objects in the view hierarchy can be potentially used as labels for these visual models. However, the invisible/misaligned objects are wrong labels to the visual models, as they don’t match the rendered objects on the screen visually. Furthermore, the too generic object types (e.g., View) might include semantically different objects, which might confuse the visual models. The too specific object types would be too sparse for the models to learn and generalize. Similar problems exist when the objects are used as input features to other UI models (e.g., widget captioning (Li et al., 2020b), language grounding (Li et al., 2020a)). The invisible/misaligned objects would result in invalid visual features and the too generic/specific object types would convey little or too sparse information for learning. Traditionally, these issues are addressed by employing human labelers to annotate a screen layout (Zhang et al., 2021) to acquire clean layouts. However, the process is expensive because there are often tens of objects and containers on each screen to label.

To address this issue, we develop the CLAY pipeline (see Figure 1) that employs deep learning models for automatically correcting a raw UI layout, by removing invalid objects and assigning a meaningful type to each object in the layout. Our pipeline consists of two steps: invalid object detection and object type recognition. Our invalid object detection module identifies and filters out the invalid objects. For object type recognition, we develop a multi-class classification model that is trained to assign to each object a meaningful type label, e.g., Drawer or Toolbar. To train and evaluate these models, we create a new dataset of 59,555 human-labeled screen layouts, based on the public Rico dataset (Deka et al., 2017). We design the invalid object detection model based on ResNet (He et al., 2015). The multi-class classification models are based on two popular model architectures, namely Graphical Neural Networks (GNNs) (Gilmer et al., 2017) and Transformer (Carion et al., 2020), combined with a ResNet backbone. We demonstrate that the models achieved high accuracy and outperform a heuristic-based baseline on the test set, which show a great potential for denoising mobile UI datasets at scale. In summary, the paper makes the following contributions.

We identify the two main problems with existing mobile UI datasets: invalid objects and objects with generic or app-specific types. We propose a denoising task for mobile UI layouts, which can improve a dataset at scale.

We create the large CLAY dataset of 59,555 screen layouts based on screenshots and raw layouts in Rico (Deka et al., 2017). Each object is either flagged as invalid, or labeled with a meaningful type. The datasetThe dataset and codes are released at https://github.com/google-research/google-research/tree/master/clay. can be used as a common base for future model development and evaluation.

We design a two-phase approach (the CLAY pipeline) for the denoising task. The first phase is a ResNet-based (He et al., 2015) filtering model, which performs binary classification for detecting invalid objects. The second phase is a multi-class classification model for the valid objects, based on Graphical Neural Networks and Transformers (Carion et al., 2020). We obtain 82.7% and 85.9% F1 scores for the first and second phase, respectively.

In the rest of the paper, we first discuss the literature related to our work in Section 2, and we then describe the dataset and the annotation process in Section 3. The model architectures and experiment setup/results are presented in Section 4 and 5, respectively. Finally, we discuss the limitations and propose directions for future work in Section 6.

Related Work

Our work is related to several research areas, including mobile UI dataset quality, UI screen modeling and data-driven automatic cleaning tools.

Datasets are the foundation for modeling and data science research. We are not the first ones to identify limitations in mobile UI layout datasets. Previously, Liu et al. (Liu et al., 2018) attempted to infer UI component types of elements from Android class names using a set of heuristics. However, they realized that objects with generic types make it difficult to produce reliable layout classes. Li et al. (Li et al., 2020a) found that there are view hierarchies out of sync with the screenshots and asked human raters to remove these erroneous screens; Li et al. (Li et al., 2020b) filtered out and discarded screens with inaccurate view hierarchy for further data labeling. Similar to Android UI datasets, there are similar issues with iOS datasets. Zhang et al. (2021) noted that APIs for generating the screen layouts might have incomplete access to the UI metadata, and they asked human raters to manually label the layout of screens.

Recently, Fu et al. (Fu et al., 2021) recognized the need for cleaner layouts, and combined optical character recognition (OCR) with a graphic detector to recreate less cluttered layouts that are visually better synchronized with their screenshot. Another direction is to use object detection models to identify the layout without relying on the raw view hierarchy (Chen et al., 2020; Zhang et al., 2021). For instance, Zhang et al. (2021) proposed to use pixels to extract UI metadata such as UI element types and states. Similarly, others took a more classical computer vision approach (Nguyen and Csallner, 2015; Sun et al., 2020), which detects a layout by identifying connected components from the pixels. However, to develop these layout prediction methods, in the first place, it is crucial to have UI datasets with clean layouts.

Compared to previous work, we focus on addressing the two major problems with view hierarchies, by detecting invalid objects and assigning each object a well-defined type. To this end, we create a clean layout dataset based on a public UI corpus, and develop a series of deep learning models to enable data cleaning in an automatic fashion.

2. Mobile UI Screen Modeling

Although our goal is to develop tools for improving dataset quality for future modeling tasks, our approach itself processes UI screenshots and raw view hierarchies and leverages UI modeling techniques. Here we briefly survey existing UI modeling techniques. Zang et al. (2021) introduced a CenterNet-based model which, combined with text embeddings from the layout, predicts icons semantic types from screenshots. Bai et al. (2021) developed a pre-training model that facilitates multiple downstream tasks, including icon classification and app type prediction. Similarly, Fu et al. (2021) fed their own cleaned layouts to a Transformer model to perform a number of downstream tasks such as clickabilty prediction, relation prediction, or app classification. Finally, there is a rich body of work to connect natural language and mobile user interfaces. In Screen2Vec (Li et al., 2021b), Li et al. developed methods for embedding UI screens to enable tasks such as screen retrieval using nearest neighbors. Li et al. (2020a) developed models that ground language instructions to executable actions on mobile phones. Li et al. (2020b) developed models for generating captions for UI components on a mobile screen. Similarly, Wang et al. (2021) proposed an approach for mobile UI screen summary, which describes the functionalities of the screen. Burns et al. (2021) proposed a new task with a new dataset for automatic task completion based on mobile UI with iterative feedback.

There are several major deep architectures that have been used in these existing works. Computer vision models such as ResNet (He et al., 2015) are often used as the backbone for extracting features from images. Object detection models, such as Single-Shot multibox Detection (SSD) (Liu et al., 2016), Faster-RCNN (Ren et al., 2015), or CenterNet (Zhou et al., 2019) are often applied to detect UI objects on screenshots (Zhang et al., 2021; Chen et al., 2020; Zang et al., 2021). Increasingly, Transformers (Vaswani et al., 2017) have been used in a range of multimodal modeling tasks (Li et al., 2020a, b; Wang et al., 2021), which allows screenshot images and view hierarchy to be easily encoded via self-attention.

Based on previous works, we design our models based on ResNet to encode images, and Transformer to perform cross-modal encoding and final decoding. We also investigate Graph Neural Networks (Gilmer et al., 2017) in this work as it can directly capture the tree structure of the view hierarchy.

3. Data-Driven Automatic Cleaning Tools

The need for developing data cleaning tools is ubiquitous (Ridzuan and Wan Zainon, 2019). There have been a number of previous efforts on developing automatic tools for data cleaning. For example, Chang et al. (Chang et al., 2017) develop a tool for labeling datasets using crowd sourcing. Cleanix (Wang et al., 2014) is a tool that address abnormal value detection, incomplete data filling, deduplication, and conflict resolution in text-based data. SCAREd (Yakout et al., 2013) is an ML-based approach that attempts to learn correlations in correct text records, and predict adequate replacements in corrupted records. In the same vein, KATARA (Chu et al., 2015) aims at fixing inaccurate data by presenting a set of ML-issued corrections to crowd workers. In the domain of vision and images, Ng and Winkler (2014) propose a classifier to identify and remove outliers in a large scale face dataset. To the best of our knowledge, we are the first to propose automatic cleaning tools specifically targeted at mobile UI layout data.

Dataset and type taxonomy

To investigate our automatic approach for denoising layout data, we create a dataset of clean UI layouts based on an existing mobile UI corpus, dubbed as the CLAY dataset. In this section, we describe the dataset, the type taxonomy and the findings from the data collection.

We use the open sourced Rico dataset (Deka et al., 2017; Liu et al., 2018), which contains 72K screenshots and view hierarchies from more than 9.7K different Android applications in 27 different app categories.

Each data point consists of a screenshot and layout information in view hierarchy about the objects on the screen. A view hierarchy is a tree structure where each node in the tree should correspond to an object on the UI. Each node contains a set of properties, such as the position of the UI object, its Android class, an optional content description, the resource identifier, and various attributes that characterize the object, e.g., whether the object is clickable or focusable.

From the original Rico dataset, we removed layouts that contained no more than two objects as these layouts usually just contain one or two large container nodes and provide little information of the objects and structure on the screen. Before labeling, we preprocessed the view hierarchies to remove objects that are too narrow (width-height aspect ratio smaller than 0.01), too small (area smaller than 0.01% of the screen), too large (area larger than the entire screen) or invisible based on the visible-to-user and visibility attributes. For objects with duplicate bounding boxes, we keep the one with a more specific type inferred from its Android class name or the last box in pre-order traversal as it is rendered at last. Occluded boxes are cut off to include only the visible part. Blank boxes with uniform color and empty containers are removed. We release the source code in the aforementioned GitHub repository so the results can be reproducible. This process resulted in a dataset with 59,555 screens.

Moreover, we counted 9,331 unique Android classes in the dataset. Many of the top classes are too generic to be useful information for object types. The 100 most popular Android classes are displayed in Figure 2. There is a long tail distribution of app-specific object types, e.g., ColombiaNativeAdView, which convey little information of the object and are too sparse for model learning.

2. Taxonomy and Labeling

We define our type taxonomy based on the naming convention introduced previously in Liu et al. (2018), where semantic types (e.g., BUTTON) are assigned to the UI components of the Rico view hierarchies to describe their functionalities. The previous taxonomy was defined based on an analysis of 720 screens. Compared to the previous work (Liu et al., 2018), we have introduced the changes described below. These changes are the result of multiple iterations on the original taxonomy. The rationale behind them was to provide classes that describe the visual appearance of the elements, and therefore we chose to merge elements that were visually similar. For instance, we do not consider VIDEO and IMAGE as separate types as we assume them to be indistinguishable on static screenshots (this is true unless there is a visual cue such as a play button overlaid on the video). Similarly, we have removed WEB VIEW and MODAL, and merged MULTI-TAB with TOOLBAR. We have also split IMAGE into two categories: IMAGE, which encompasses any natural image, photo or drawing; and PICTOGRAM, which represents an image containing vector graphics and a limited number of colors as found in icons and logos.

On the other hand, we also added a few elements such as SPINNER and PROGRESS_BAR, as we considered them visually distinctive enough to justify new classes. Finally, we added a more structural and hierarchical label with CONTAINER. A summary of our chosen taxonomy is shown in Table 1. We intend to label each valid node in the view hierarchy with one of these types.

Based on the type taxonomy, we asked a group of 15 crowd human workers to label the filtered Rico dataset, which took at total of 1,577 hours. We developed a web interface for human workers to annotate each view hierarchy element by assigning it with an appropriate type label. The interface shows a screenshot of the mobile interface, together with the bounding boxes extracted from the raw view hierarchy. Workers can choose the best type label from the list of the taxonomy, or flag the element invalid if its rendered bounding box does not correspond to a valid object on the screenshot. To ensure the quality, we audited the results by randomly sampling 3.1% of the labeled examples during the labeling process and asked different labelers to verify them. It turned out 98.8% of the audited objects were correctly labeled. Furthermore, for the validation and test set (see Table 2), we labeled each object with 3 different labelers, and generated the final label by voting. After labeling the entire dataset that consists of 59,555 UI screens, 22,273 screens or 37.4% of the dataset contain at least one invalid element (see Figure 3). The ratio of invalid versus valid objects is approximately 1:8. Figure 4 shows the object type distribution of the labeled data. The dataset contains common UI objects including TEXT and CONTAINER, as well as rare types of objects, such as DATE_PICKER and KEYBOARD.

Task formulation and Model architecture

We design a two-phase approach for denoising UI layout data. For the first phase, we propose a visual-based model, which detects invalid objects based on the object pixels. For the second phase of object type recognition, we investigate two popular architectures: a GNN-based model (Gilmer et al., 2017) and a model based on the DeTR Transformer architecture (Carion et al., 2020; Zhu et al., 2021). Both of them are multi-modal, which rely on pixel information as well as raw view hierarchy to make predictions on the object type.

We first preprocess the layouts to filter out obviously invalid objects simply by looking at the layout tree and the rendering order of the objects. For example, objects fully occluded by other objects are removed, while the bounding boxes of objects partially occluded are trimmed.

We then further filter out invalid objects using a binary classification model. We augment the popular ResNet model (He et al., 2015) with an extra input mask channel, in addition to RGB, the three original image channels (see Figure 5). The model examines one object in the layout at a time. With this extra mask channel, the input of the model is a matrix of size $[H,W,4]$, with $H$ and $W$ as the height and width of the screenshot image. The first three channels correspond to the original pixels of the image, and the fourth channel indicates the bounding box of the object being inspected. The mask channel simply contains a binary mask with value $1$ at positions corresponding to the object bounding box, and otherwise. With the mask channel, the model is aware of the object location and focuses on the object pixels to make the prediction. In the meantime, the model has access to the context via the convolution operations in ResNet. The output of the model predicts how likely the object is invalid.

2. Object Type Recognition

In the second phase, we introduce two alternative deep learning approaches. We will discuss the pros and cons of each method in light of the experimental results. Both methods take the view hierarchy as input, and they use a similar approach for embedding each node (object) in the view hierarchy.

To represent each view hierarchy node as a dense vector, we embed its attributes separately and then combine these embeddings. For text-related information, we use the Android class name, content_desc and resource_id from each view hierarchy node. We use a vocabulary of size 28,536 to tokenize the text with the byte-pair encoding method, which is the same as BERT (Devlin et al., 2018). A maximum of first ten words of the three text fields are used for text embedding. These text embeddings are trained from scratch and max-pooled into a dense vector representing the information from the three text fields. At last, the text embedding of the node, denoted as $W$, is constructed by concatenating the three dense vectors.

To represent the object positional information, we use the four coordinates (i.e., the scalar values representing the left, right, top, and bottom location) of the object bounding box. Following Li et al. (2021c), each coordinate is mapped to a dense vector using fully connected layers and sinusoidal mapping. The four dense vectors are concatenated to form the positional encoding of the object, denoted as $P$. In this way the model can learn the representation of the coordinates via back-propagation for better performance.

2.2. Type Recognition with Graph Neural Networks

Our first proposed model for object type recognition is a multimodal GNN inspired by the message-passing neural network (MPNN) proposed by Gilmer et al. (2017), which is a supervised-learning architecture that takes a graph as input. The output of the MPNN is a prediction of type for every node in the input graph. In our case, the input is the raw view hierarchy and the output is the object type of each node in the view hierarchy. The motivation for introducing a GNN-based approach is that view hierarchies are a tree structure, which is a special case of graphs, and thus GNNs can naturally leverage the structure.

To incorporate the pixel information of an object into the input, we crop the object pixels based on its bounding box in the view hierarchy as input to a ResNet-50 model. The output of the ResNet is flattened and passed through a dense layer to generate a dense vector, $I$, as the pixel encoding of the object.

In our MPNN, each node or object $o$ is represented by a hidden state $h_{o}^{t}$, where $t=0,1,\ldots,T$ is the time step index. At $t=0$, the hidden state is initialized by concatenating the pixel, text and positional embeddings:

At every time step $t=1,2,\ldots,T$, a message kernel $M$ is applied to every pair of connected objects, according to the view hierarchy tree structure. For every node $o$, the messages from all its connections are gathered and aggregated via a pooling function, resulting in the vector $p_{o}^{t+1}$. The pooled vector is then fed to an $H$ kernel that updates the hidden state of a node at step $t+1$:

Finally, a readout kernel $Y$ is applied to compute a logit for every node:

The variables of the kernels $H$, $M$, and $Y$, as well as the weights of the image encoder are all trainable parameters. An overview of our GNN model is depicted in Figure 6.

2.3. Type Recognition with Transformer Models

GNN directly captures the structure of a view hierarchy that induces a strong bias, which can be vulnerable to noisy structures. Thus, we design a Transformer-based model (Vaswani et al., 2017) that can learn object relationship via self-attention. In particular, we design our model based on DeTR (Carion et al., 2020), a model architecture that combines ResNet and Transformer for object detection. Instead of feeding in object queries as DeTR does, we feed in the view hierarchy node embedding as input to the parallel decoder. We also replace the output head that originally uses expensive Hungrian matching for object detection with the classification head for type prediction. At a high level, our model uses Transformer encoder stacked on ResNet to encode the entire screenshot image, and a Transformer parallel decoder to predict the object type of each node while attending to the image encoding. An overview of the model is shown in Figure 7.

The screenshot is first encoded with a ResNet-50 model, and the encoding is split into patches as inputs to the Transformer encoder. The outputs of the Transformer encoder is a matrix $M$ of shape $[N_{m},H_{m}]$, where $N_{m}$ is the number of image patches and $H_{m}$ is the hidden state size of each patch. In the parallel decoder, each node or object $o$ is represented by a hidden state $h_{o}^{t}$, where $t=0,1,\ldots,T$ is the layer index. The inputs to the parallel decoder is constructed by adding up the object text embedding and the positional encoding:

The parallel decoder accesses the image encoding of the entire screenshot, $M$, by encoder-decoder attention. In each layer $t=1,2,\ldots,T$, the parallel decoder will generate the hidden states for each object via self-attention of hidden states from last layer and cross encoder-decoder attention to the image:

Finally, a dense layer $Y$ is applied to compute the logits for every node:

For all our models of binary or multi-class classification, we train these models using the cross-entropy loss. $L2$ regularization is used for all the trainable weights in the model to mitigate over-fitting.

Experiments

In this section, we describe our experiments for evaluating the models and the results. We first experiment with the binary classification model for detecting invalid objects, and then evaluate the two multi-class classification models for object type recognition, in comparison with a baseline method that uses heuristics to predict UI object types.

We split our dataset of 59,555 screens randomly into the training, validation and test set. The split was performed package-wise, i.e., screens from the same package are not shared among the three splits. This is to avoid information leakage because screens from the same package might have similar layouts. Table 2 shows the statistics of the three sets.

2. Model Configurations & Training

We here describe the configuration and training details of each model. Both the invalid object detection model and the GNN-based type recognition model are implemented in TensorFlowhttps://www.tensorflow.org. The Transformer-based model is implemented in JAXhttps://github.com/google/jax. We select the hyper-parameters to obtain the best performance on the validation set.

We train the model, based on ResNet-50, with a batch size of 1024 images for 15k steps to converge, with an initial learning rate 6e-4 and a reduced learning rate 6e-5 after 5.5k steps. To counter the skewed 8:1 distribution of valid and invalid objects, we re-sample the training data to have a ratio of 4:1 for valid and invalid objects, which has the best results on the validation set among the experiments using different ratios from 1:1 to 8:1. We do not apply resampling to the validation and test data.

2.2. GNN Models for Type Recognition

We use a ResNet-50 model to encode the pixel information, the image crop of each UI object is resized to a squares of size $64\times 64$, and the image embedding size is 32. The GNN nodes are connected by bidirectional edges that represent the parent-child relationships of the elements in the layout. We also connect nodes that are spatially next to each other on the screenshot. At each step, 5 rounds of messages of size 32 are passed between the nodes. The messages are then aggregated with an attention pooling function. The GNN model is trained to converge with 500K steps and a batch size of 32, with an initial learning rate of 2e-3, which is reduced to 1e-4 after 200K steps.

2.3. Transformer Models for Type Recognition

We use a ResNet-50 model as backbone, a 6-layer encoder for encoding the image, and a 6-layer parallel decoder to encode the view hierarchy objects and predict object types. The embedding dimension for view hierarchy objects is 256. For the Transformer encoder/decoder, we use 8 attention heads, MLP dimension 2048 and query/key/value dimension 256 (Vaswani et al., 2017). The model is trained for 15k steps to converge, with a batch size of 128 examples, an initial learning rate 6e-5 for the ResNet backbone and 1e-4 for the Transformer encoder/decoder, and a reduced learning rate by 10 times after 5k steps.

2.4. Heuristic Baseline

As a baseline to compare our multiclass type recognition models, we implemented a heuristic method for inferring the layout types. Similar to the approach presented previously Liu et al. (2018), the method deduces the object type from the Android class of the UI component in the view hierarchy. Since our type taxonomy is defined based on this previous work, we reused some of its mappings (Liu et al., 2018), and enhanced the inference rules based on the content description and resource id of the element. For instance, the NAVIGATION_BAR type can be identified from the resource ids android:id/navigationBarBackground or android:id/statusBarBackground. Similarly, the MAP type can be detected from elements whose resource id is com.google.android.apps.maps:id/map_frame. As we mentioned earlier, it is generally challenging to cover all the cases of mapping using a heuristic-based method. We release the code of the heuristic baseline for reproduction purposes in the aforementioned GitHub repository.

3. Results

We first report the model performance for the detection phase in Table 3. Our model performs well for detecting the invalid objects, obtaining 82.7% F-score with balanced precision and recall. This indicates that the visual-based model is effective for recognizing misaligned, invisible or grayed-out objects. The task is challenging because the ratio between invalid and valid objects is skewed. Balancing invalid and valid objects, i.e., positive and negative examples, in the training does significantly boost the model performance. We further analyze the quality of the model in Section 5.4.

Next, we report the model performances for object type recognition in Table 4. We average the scores across all the object types in two different ways: 1) weighted average where each type is weighted by the number of objects of that type, and 2) macro average where all the types have the same weight. Both GNN and Transformer-based model achieve significantly better performances than the heuristic baseline. The GNN model obtains better weighted average scores, which are dominated by the common object types. On the other hand, Transformer has better macro average scores. In Table 5, we can see that the GNN model has better performance for the more common types, while Transformer obtains more balanced scores across all types and performs better on rare types. Visual examples on various screenshots from the validation set are shown in Figure 11.

4. Error Analysis

We first analyze the errors of the invalid object detection model. We sample 100 (4.2%) from all the false positive cases (valid objects predicted as invalid) and 100 (3.1%) from all the false negative cases (invalid objects predicted as valid), and manually check the object on the screenshot to understand why the model makes the mistakes. Out of the 100 false positive errors, 47 cases have bounding boxes that are slightly shifted and partially cover the objects or do not cover the object tightly; 21 cases have bounding boxes overlapping with other objects; 14 cases are very small bounding boxes and the other 18 are due to reasons including blurred screen and confusion with background objects. The most common errors seem to be ambiguous cases which might be difficult even for a human labeler. For example, in the screenshot on the left of Figure 8 the object with the red bounding box is labeled as valid by human workers, but predicted as invalid. In our labeling guideline, we define the invalid objects as those whose bounding boxes do not well align with the rendered objects, which leaves some uncertainty about how much misalignment is allowed for an object to be valid.

Two error examples for the invalid object detection model. Both the false positive and the false negative example show that the model tends to make mistakes on ambiguous cases, which might be difficult even for human labelers.

Similarly, out of the 100 false negative errors, 54 cases have bounding boxes that are shifted but encompass the object partially or not tightly. An example is shown on the right side of Figure 8, where the object in red bounding box is labeled as invalid by human workers but predicted as valid by the model. 22 cases are grayed-out objects in the background but the model failed to detect it; 17 cases have bounding boxes overlapping with other objects; 7 cases are very small bounding boxes.

For object type recognition, as shown in the confusion matrix (see Figure 9), our model performs well for most cases. The confusions tend to occur between several object types which can be ambiguous or similar looking. Among the 5 most common types of confusion, we examined all 27 instances of the confusion between MAP and CONTAINER and sampled 50 instances for the other 4 types, consisting of 1.9% - 35.7% of all the confusion instances. Figure 10 illustrates the five types of confusions. Specifically, BUTTON is confused with TEXT for some objects that look like pure text but actually can be clickable for users to take actions (Figure 10(a)). IMAGE is confused with PICTOGRAM on some icon-like images (Figure 10(b)). LABEL is confused with TEXT for text-like LABEL objects (Figure 10(c)). The model predicts some instances of CARD_VIEW and MAP as CONTAINER, possibly due to that CARD_VIEW is a special type of container and might look similar (Figure 10(e)), and some large MAP objects contain other UI objects (Figure 10(d)). Data imbalance may be another reason as CARD_VIEW and MAP have much fewer instances than CONTAINER, which makes it more difficult for the model to learn.

Examples of the five most common confusion types. The confusions usually happen between object types that have similar appearance.

We further examined ADVERTISEMENT, for which both GNN and Transformer have lower scores. It is among the rare ones in the dataset, and usually has larger bounding boxes, for which the model confuses with CONTAINER or IMAGE. More training examples of such rare object types would potentially improve the model performance. We can merge some of the similar-looking object types to improve model performance when it is feasible for the downstream application.

Discussions

We created a large screen layout dataset based on 59,555 screens of Rico (Deka et al., 2017), with problematic objects flagged and more semantically meaningful types assigned to the valid objects. The cleaned layouts can be used for UI design research (e.g., similar layout retrieval (Wang et al., 2021)) and training new visual-based models (Sun et al., 2020) or data-cleaning models as described in this paper. The data-cleaning models can be used to preprocess the layout for downstream tasks, or clean large unlabeled UI dataset for training visual-based models, which can perform better with large-scale training data (Radford et al., 2021).

Our models achieved an F1 score of 82.7% for detecting invalid objects and an F1 score of 85.9% for object type recognition. They offer a practical solution for cleaning datasets at scale. The invalid object detection model is effective despite the skewed ratio between valid and invalid objects, and made incorrect predictions for only a small amount of the objects when evaluated on the test set. Our proposed GNN and Transformer models perform comparably for weighted and macro average scores. For future work, combining the strengths of the two models to achieve good weighted and macro average scores is an interesting direction. We can try ensemble of the two models by joining the prediction probabilities. Encoding the structural information explicitly in the Transformer model might help the model to perform better on common objects. For the GNN model, accessing the entire image instead of the cropped pixels is another promising direction.

One limitation of our work is that we train and evaluate our models only on Android screenshots. This limitation is due to the lack of public corpus of screen layouts for these mobile platforms. The invalid object detection model relies on screenshot images only and might generalize better to other mobile OS, compared to the object typing models which rely on the OS-specific screen layouts. We hope to include more diverse and recent screenshots in terms of packages and mobile OS in the future. For the denoising task, our model cleans up view hierarchy by labeling each node. However, more significant cleaning might be needed occasionally. For example, there might be objects that are rendered on the screen but missing in the view hierarchy. Therefore, adding new objects or adjusting objects’ positions and attributes might be needed. Our DeTR-based model can include an object detection decoder (Carion et al., 2020) for this purpose, which would deserve further investigation.

Conclusion

We present the CLAY pipeline, using a deep learning approach, for denoising mobile screen layouts, which are a critical data source for UI design research and UI semantic understanding. Our analysis reveals that automatically captured layouts in existing datasets are noisy and contain invalid objects and objects with noisy type information. To facilitate our investigation, we create the large CLAY dataset of clean UI layouts based on a public mobile UI corpus. We then propose a two-stage approach to first detect and remove invalid objects, and then classify the valid objects into the layout types defined in a taxonomy. Our experiments show that our models achieve good performance for both stages, which show a great potential to automatically denoise large layout datasets. These models will boost future efforts for large-scale UI modeling analysis.

References

Appendix A Prediction Examples

Figure 11 shows five example screens with the original layouts and the outputs from our models.