Relation-Aware Graph Attention Network for Visual Question Answering

Introduction

Recent advances in deep learning have driven tremendous progress in both Computer Vision and Natural Language Processing (NLP). Interdisciplinary area between language and vision, such as image captioning, text-to-image synthesis and visual question answering (VQA), has attracted rapidly growing attention from both vision and NLP communities. Take VQA as an example - the goal (and the main challenge) is to train a model that can achieve comprehensive and semantically-aligned understanding of multimodal input. Specifically, given an image and a natural language question grounded on the image, the task is to associate visual features in the image with the semantic meaning in the question, in order to correctly answer the question.

Most state-of-the-art approaches to VQA focus on learning a multimodal joint representation of images and questions. Specifically, a Convolutional Neural Network (CNN) or Region-based CNN (R-CNN) is commonly used as a visual feature extractor for image encoding, and a Recurrent Neural Network (RNN) is used for question encoding. After obtaining a sparse set of image regions from the visual feature extractor, multimodal fusion is applied to learn a joint representation that represents the alignment between each individual region and the question. This joint representation is then fed into an answer predictor to produce an answer.

This framework has proven to be useful for the VQA task, but there still exists a significant semantic gap between image and natural language. For example, given an image of a group of zebras (see Figure 1), the model may recognize the black and white pixels, but not which white and black pixels are from which zebra. Thus, it is difficult to answer questions such as “Is the zebra at the far right a baby zebra?” or “Are all the zebras eating grass?”. A VQA system needs to recognize not only the objects (“zebras“) and the surrounding environment (“grass”), but also the semantics about actions (“eating”) and locations (“at the far right”) in both images and questions.

In order to capture this type of action and location information, we need to go beyond mere object detection in image understanding, and learn a more holistic view of the visual scene in the image, by interpreting the dynamics and interactions between different objects in an image. One possible solution is to detect the relative geometrical positions of neighboring objects (e.g., ), to align with spacial descriptions in the question. Another direction is to learn semantic dependencies between objects (e.g., ) to capture the interactive dynamics in the visual scene.

Motivated by this, we propose a Relation-aware Graph Attention Network (ReGAT) for VQA, introducing a novel relation encoder that captures these inter-object relations beyond static object/region detection. These visual relation features can reveal more fine-grained visual concepts in the image, which in turn provides a holistic scene interpretation that can be used for answering semantically-complicated questions. In order to cover the high variance in image scenes and question types, both explicit (e.g., spatial/positional, semantic/actionable) relations and implicit relations are learned by the relation encoder, where images are represented as graphs and interactions between objects are captured via a graph attention mechanism.

Furthermore, the graph attention is learned based on the context of the question, permitting the injection of semantic information from the question into the relation encoding stage. In this way, the features learned by the relation encoder not only capture object-interactive visual contents in the image, but also absorb the semantic clues in the question, to dynamically focus on particular relation types and instances for each question on the fly.

Figure 1 shows an overview of the proposed model. First, a Faster R-CNN is used to generate a set of object region proposals, and a question encoder is used for question embedding. The convolutional and bounding-box features of each region are then injected into the relation encoder to learn the relation-aware, question-adaptive, region-level representations from the image. These relation-aware visual features and the question embeddings are then fed into a multimodal fusion module to produce a joint representation, which is used in the answer prediction module to generate an answer.

In principle, our work is different from (and compatible to) existing VQA systems. It is pivoted on a new dimension: using question-adaptive inter-object relations to enrich image representations in order to enhance VQA performance. The contributions of our work are three-fold:

We propose a novel graph-based relation encoder to learn both explicit and implicit relations between visual objects via graph attention networks.

The learned relations are question-adaptive, meaning that they can dynamically capture visual object relations that are most relevant to each question.

We show that our ReGAT model is a generic approach that can be used to improve state-of-the-art VQA models on the VQA 2.0 dataset. Our model also achieved state-of-the-art performance on the more challanging VQA-CP v2 dataset.

Related Work

The current dominant framework for VQA systems consists of an image encoder, a question encoder, multimodal fusion, and an answer predictor. In lieu of directly using visual features from CNN-based feature extractors, explored various image attention mechanisms to locate regions that are relevant to the question. To learn a better representation of the question, proposed to perform question-guided image attention and image-guided question attention collaboratively, to merge knowledge from both visual and textual modalities in the encoding stage. explored higher-order fusion methods to better combine textual information with visual information (e.g., using bilinear pooling instead of simpler first-order methods such as summation, concatenation and multiplication).

To make the model more interpretable, some literature also exploited high-level semantic information in the image, such as attributes, captions and visual relation facts. Most of these methods applied VQA-independent models to extract semantic knowledge from the image, while built a Relation-VQA dataset and directly mined VQA-specific relation facts to feed additional semantic information to the model. A few recent studies investigated how to incorporate memory to aid the reasoning step, especially for difficult questions.

However, the semantic knowledge brought in by either memory or high-level semantic information is usually converted into textual representation, instead of directly used as visual representation, which contains richer and more indicative information about the image. Our work is complementary to these prior studies in that we encode object relations directly into image representation, and the relation encoding step is generic and can be naturally fit into any state-of-the-art VQA model.

2 Visual Relationship

Visual relationship has been explored before deep learning became popular. Early work presented methods to re-score the detected objects by considering object relations (e.g., co-occurrence , position and size ) as post-processing steps for object detection. Some previous work also probed the idea that spatial relationships (e.g., “above”, “around”, “below” and “inside”) between objects can help improve image segmentation.

Visual relationship has proven to be crucial to many computer vision tasks. For example, it aided the cognitive task of mapping images to captions and improved image search and object localization . Recent work on visual relationship focused more on non-spatial relation, or known as “semantic relation” (i.e., actions of, or interactions between objects). A few neural network architectures have been designed for the visual relationship prediction task .

3 Relational Reasoning

We name the visual relationship aforementioned as explicit relation, which has been shown to be effective for image captioning . Specifically, exploited pre-defined semantic relations learned from the Visual Genome dataset and spatial relations between objects. A graph was then constructed based on these relations, and a Graph Convolutional Network (GCN) was used to learn representations for each object.

Another line of research focuses on implicit relations, where no explicit semantic or spatial relations are used to construct the graph. Instead, all the relations are implicitly captured by an attention module or via higher-order methods over the fully-connected graph of an input image , to model the interactions between detected objects. For example, reasons over all the possible pairs of objects in an image via the use of simple MLPs. In , a bilinear fusion method, called MuRel cell, was introduced to perform pairwise relationship modeling.

Some other work have been proposed for learning question-conditioned graph representations for images. Specifically, introduced a graph learner module that is conditioned on question representations to compute the image representations using pairwise attention and spatial graph convolutions. exploited structured question representations such as parse trees, and used GRU to model contextualized interactions between both objects and words. A more recent work introduced a sparser graph defined by inter/intra-class edges, in which relationships are implicitly learned via a language-guided graph attention mechanism. However, all these work still focused on implicit relations, which are less interpretable than explicit relations.

Our work is inspired by . However, different from them, ReGAT considers both explicit and implicit relations to enrich image representations. For explicit relations, our model uses Graph Attention Network (GAT) rather than a simple GCN as used in . As opposed to GCNs, the use of GAT allows for assigning different importances to nodes of the same neighborhood. For implicit relations, our model learns a graph that is adaptive to each question by filtering out question-irrelevant relations, instead of treating all the relations equally as in . In experiments, we conduct detailed ablation studies to demonstrate the effectiveness of each individual design.

Relation-aware Graph Attention Network

Here is the problem definition of the VQA task: given a question $q$ grounded in an image $I$, the goal is to predict an answer $\hat{a}\in\mathcal{A}$ that best matches the ground-truth answer $a^{\star}$. As common practice in the VQA literature, this can be defined as a classification problem:

By treating each object $v_{i}$ in the image as one vertex, we can construct a fully-connected undirected graph $\mathcal{G}_{imp}=(\mathcal{V},\mathcal{E})$, where $\mathcal{E}$ is the set of $K\times(K-1)$ edges. Each edge represents an implicit relation between two objects, which can be reflected by the learned weight assigned to each edge through graph attention. All the weights are learned implicitly without any prior knowledge. We name the relation encoder built on this graph $\mathcal{G}_{imp}$ the implicit relation encoder.

Pruned Graph with Prior Knowledge

On the other hand, if explicit relations between vertices are available, one can readily transform the fully-connected graph $\mathcal{G}_{imp}$ into an explicit relation graph, by pruning the edges where the corresponding explicit relation does not exist. For each pair of objects $i,j$, if <$i$-$p$-$j$>is a valid relation, an edge is created from $i$ to $j$, with an edge label $p$. In addition, we assign each object node $i$ with a self-loop edge and label this edge as identical. In this way, the graph becomes sparse, and each edge encodes prior knowledge about one inter-object relation in the image. We name the relation encoder built upon this graph the explicit relation encoder.

The explicit nature of these features requires pre-trained classifiers to extract the relations in the form of discrete class labels, which represent the dynamics and interactions between objects explicit to the human eye. Different types of explicit relations can be learned based on this pruned graph. In this paper, we explore two instances: spatial and semantic graphs, to capture positional and actionable relations between objects, which is imperative for visual question answering.

Spatial Graph Let $spa_{i,j}=$<$\texttt{object}_{i}$-predicate -$\texttt{object}_{j}$> denote the spatial relation that represents the relative geometric position of $\texttt{object}_{i}$ against $\texttt{object}_{i}$. In order to construct a spatial graph $\mathcal{G}_{spa}$, given two object region proposals $\texttt{object}_{i}$ and $\texttt{object}_{j}$, we classify $spa_{i,j}$ into $11$ different categories (e.g., $\texttt{object}_{i}$ is inside $\texttt{object}_{j}$ (class 1), $\texttt{object}_{j}$ is inside $\texttt{object}_{i}$ (class 2), as illustrated in Figure 3(a)), including a no-relation class retained for objects that are too far away from each other. Note that edges formed by spatial relations are symmetrical: if <$\texttt{object}_{i}$-$p_{i,j}$-$\texttt{object}_{j}$ > is a valid spatial relation, there must be a valid spatial relation $spa_{j,i}=$<$\texttt{object}_{j}$-$p_{j,i}$-$\texttt{object}_{i}$ >. However, the two predicates $p_{i,j}$ and $p_{j,i}$ are different.

Semantic Graph In order to construct semantic graph $\mathcal{G}_{sem}$, semantic relations between objects need to be extracted (e.g., ). This can be formulated as a classification task by training a semantic relation classifier on a visual relationship dataset (e.g., Visual Genome ). Given two object regions $i$ and $j$, the goal is to determine which predicate $p$ represents a semantic relation <$i$-$p$-$j$> between these two regions. Here, the relations between the subject $j$ and the object $i$ are not interchangeable, meaning that the edges formed by semantic relations are not symmetric. For a valid <$i$-$p_{i,j}$-$j$>, there may not exist a relation <$j$-$p_{j.i}$-$i$> within our definition. For example, is a valid relation, while there is no semantic relation from bat to man.

The classification model takes in three inputs: feature vector of the subject region ${\boldsymbol{v}}_{i}$, feature vector of the object region ${\boldsymbol{v}}_{j}$, and region-level feature vector ${\boldsymbol{v}}_{i,j}$ of the union bounding box containing both $i$ and $j$. These three types of feature are obtained from pre-trained object detection model, and then transformed via an embedding layer. The embedded features are then concatenated and fed into a classification layer to produce softmax probability over 14 semantic relations, with an additional no-relation class. The trained classifier is then used to predict relations between any pair of object regions in a given image. Examples of semantic relations are shown in Figure 3(b).

2 Relation Encoder

The proposed relation encoder is designed to encode relational dynamics between objects in an image. For the VQA task, there might be different types of relations that are useful for different question types. Thus, in designing the relation encoder, we use a question-adaptive attention mechanism to inject semantic information from the question into relation graphs, to dynamically assign higher weights to those relations that are mostly relevant to each question. This is achieved by first concatenating the question embedding ${\boldsymbol{q}}$ with each of the $K$ visual features ${\boldsymbol{v}}_{i}$, denoted as

Self-attention is then performed on the vertices, which generates hidden relation features $\{{\boldsymbol{v}}_{i}^{\star}\}_{i=1}^{K}$ that characterize the relations between a target object and its neighboring objects. Based on this, each relation graph goes through the following attention mechanism:

In the end, ${\boldsymbol{v}}_{i}^{\star}$ is added to the original visual feature ${\boldsymbol{v}}_{i}$ to serve as the final relation-aware feature.

Implicit Relation

Since the graph for learning implicit relation is fully-connected, $\mathcal{N}_{i}$ contains all the objects in the image, including object $i$ itself. Inspired by , we design the attention weight $\alpha_{ij}$ to not only depend on visual-feature weight $\alpha^{v}_{ij}$, but also bounding-box weight $\alpha^{b}_{ij}$. Specifically,

where $\alpha^{v}_{ij}$ represents the similarity between the visual features, computed by scaled dot-product :

Explicit Relation

We consider semantic relation encoder first. Since edges in the semantic graph $\mathcal{E}_{sem}$ now contain label information and are directional, we design the attention mechanism in (3) to be sensitive to both directionality ($v_{i}$-to-$v_{j}$, $v_{j}$-to-$v_{i}$ and $v_{i}$-to-$v_{i}$) and labels. Specifically,

where ${\bf W}_{\{\cdot\}},{\bf V}_{\{\cdot\}}$ are matrices, and ${\boldsymbol{b}}_{\{\cdot\}},{\boldsymbol{c}}_{\{\cdot\}}$ are bias terms. $dir(i,j)$ selects the transformation matrix wrt the directionality of each edge, and $lab(i,j)$ represents the label of each edge. Consequently, after encoding all the regions $\{{\boldsymbol{v}}^{\prime}_{i}\}_{i=1}^{K}$ via the above graph attention mechanism, the refined region-level features $\{{\boldsymbol{v}}^{\star}_{i}\}_{i=1}^{K}$ are endowed with the prior semantic relations between objects.

As opposed to graph convolutional networks, this graph attention mechanism effectively assigns different weights of importance to nodes of the same neighborhood. Combining with the question-adaptive mechanism, the learned attention weights can reflect which relations are relevant to a specific question. The relation encoder can work in the same manner on the spatial graph $\mathcal{E}_{spa}$, with a different set of parameters to be learned, thus details are omitted for simplicity.

3 Multimodal Fusion and Answer Prediction

After obtaining relation-aware visual features, we want to fuse question information ${\boldsymbol{q}}$ with each visual representation ${\boldsymbol{v}}_{i}^{\star}$ through a multi-model fusion strategy. Since our relation encoder preserves the dimensionality of visual features, it can be incorporated with any existing multi-modal fusion method to learn a joint representation $\mathbf{J}$:

where $f$ is a multi-modal fusion method and $\Theta$ are trainable parameters of the fusion module.

For the Answer Predictor, we adopt a two-layer multi-layer perceptron (MLP) as the classifier, with the joint representation $\mathbf{J}$ as the input. Binary cross entropy is used as the loss function, similar to .

In the training stage, different relations encoders are trained independently. In the inference stage, we combine the three graph attention networks with a weighted sum of the predicted answer distributions. Specifically, the final answer distribution is calculated by:

where $\alpha$ and $\beta$ are trade-off hyper-parameters ($0\leq\alpha+\beta\leq 1,0\leq\alpha,\beta\leq 1)$. $Pr_{sem}(a=a_{i})$, $Pr_{spa}(a=a_{i})$ and $Pr_{imp}(a=a_{i})$ denote the predicted probability for answer $a_{i}$, from the model trained with semantic, spatial and implicit relations, respectively.

Experiments

We evaluate our proposed model on VQA 2.0 and VQA-CP v2 datasets . In addition, Visual Genome is used to pre-train the semantic relation classifier. It is also used to augment the VQA dataset when testing on the test-dev and test-std splits. We use accuracy as the evaluation metric:

VQA 2.0 dataset is composed of real images from MSCOCO with the same train/validation/test splits. For each image, an average of 3 questions are generated. These questions are divided into 3 categories: Y/N, Number and Other. 10 answers are collected for each image-question pair from human annotators, and the most frequent answer is selected as the correct answer. Both open-ended and multiple-choice question types are included in this dataset. In this work, we focus on the open-ended task, and take the answers that appeared more than 9 times in the training set as candidate answers, which produces $3,129$ answer candidates. The model is trained on the training set, but when testing on the test set, both training and validation set are used for training, and the max-probable answer is selected as the predicted answer.

VQA-CP v2 dataset is a derivation of the VQA 2.0 dataset, which was introduced to evaluate and reduce the question-oriented bias in VQA models. In particular, the distribution of answers with respect to question types differs between training and test splits.

Visual Genome contains 108K images with densely annotated objects, attributes and relationships, which we used to pre-train the semantic relation classifier in our model. We filtered out those images that also appeared in the VQA validation set, and split the relation data into 88K for training, 8K for validation, and 8K for testing. Furthermore, we selected the top-14 most frequent predicates in the training data, after normalizing the predicates with relationship-alias provided in Visual Genome. The final semantic relation classifier is trained over 14 relation classes plus a no-relation class.

2 Implementation Details

Each question is tokenized and each word is embedded using 600-dimensional word embeddings (including 300-dimensional GloVe word embeddings ). The sequence of embedded words is then fed into GRU for each time step up to the 14th token (similar to ). Questions shorter than 14 words are padded at the end with zero vectors. The dimension of the hidden layer in GRU is set as 1024. We employ multi-head attention with 16 heads for all three graph attention networks. The dimension of relation features is set to 1024. For implicit relation, we set the embedded relative geometry feature dimension $d_{h}$ to be 64.

For the semantic relation classifier, we extract pre-trained object detection features with known bounding boxes from Faster R-CNN model in conjunction with ResNet-101 . More specifically, the features are the output of the Pool5 layer after RoI pooling from Res4b22 feature map . The Faster R-CNN model is trained over 1,600 selected object classes and 400 attribute classes, similar to the bottom-up attention .

Our model is implemented based on PyTorch . In experiments, we use Adamax optimizer for training, with the mini-batch size as 256. For choice of learning rate, we employ the warm-up strategy . Specifically, we begin with a learning rate of 0.0005, linearly increasing it at each epoch till it reaches 0.002 at epoch 4. After 15 epochs, the learning rate is decreased by 1/2 for every 2 epochs up to 20 epochs. Every linear mapping is regularized by weight normalization and dropout ($p=0.2$ except for the classifier with $0.5$).

3 Experimental Results

To better understand how these two components help answer questions, we further visualize and compare the attention maps learned by the ablated instances in Section 4.5.

To better illustrate the effectiveness of adding graph attention and question-adaptive mechanism, we compare the attention maps learned by the complete ReGAT model in a single-relation setting with those learned by two ablated models. As shown in Figure 4, the second, third and last rows correspond to line 1, 3 and 4 in Table LABEL:tab:ablation, respectively. Comparing row 2 with row 3 leads to the observation that graph attention helps to capture the interactions between objects, which contributes to a better alignment between image regions and questions. Row 3 and row 4 show that adding the question-adaptive attention mechanism produces sharper attention maps and focuses on more relevant regions. These visualization results are consistent with the quantitative results reported in Table LABEL:tab:ablation.

Figure 5 provides visualization examples on how different types of relations help improve the performance. In each example, we show the top-3 attended regions and the learned relations between these regions. As shown in these examples, each relation type contributes to a better alignment between image regions and questions. For example, in Figure 5(a), semantic relations “Holding” and “Riding” resonate with the same words that appeared in the corresponding questions. Figure 5(b) shows how spatial relations capture the relative geometric positions between regions.

To visualize implicit relations, Figure 5(c) shows the attention weights to the top-1 region from every other region. Surprisingly, the learned implicit relations are able to capture both spatial and semantic interactions. For example, the top image in Figure 5(c) illustrates spatial interaction “on” between the table and the vase, and the bottom image illustrates the semantic interaction “walking” between the traffic light and the person.

We have presented Relation-aware Graph Attention Network (ReGAT), a novel framework for visual question answering, to model multi-type object relations with question-adaptive attention mechanism. ReGAT exploits two types of visual object relations: Explicit Relations and Implicit Relations, to learn a relation-aware region representation through graph attention. Our method achieves state-of-the-art results on both VQA 2.0 and VQA-CP v2 datasets. The proposed ReGAT model is compatible with generic VQA models. Comprehensive experiments on two VQA datasets show that our model can be infused into state-of-the-art VQA architectures in a plug-and-play fashion. For future work, we will investigate how to fuse the three relations more effectively and how to utilize each relation to solve specific question types.