An Empirical Study on Leveraging Scene Graphs for Visual Question Answering

Introduction

Scene understanding and reasoning has long been a core task that the computer vision community strives to advance. In recent years, we have witnessed significant improvement in many representative sub-tasks such as object recognition and detection, in which the machines’ performance is on par or even surpasses humans’ [He et al.(2015)He, Zhang, Ren, and Sun, He et al.(2016)He, Zhang, Ren, and Sun], motivating the community to move toward higher-level sub-tasks such as visual captioning [Anderson et al.(2018)Anderson, He, Buehler, Teney, Johnson, Gould, and Zhang, Mao et al.(2015)Mao, Xu, Yang, Wang, Huang, and Yuille, Xu et al.(2015)Xu, Ba, Kiros, Cho, Courville, Salakhudinov, Zemel, and Bengio] and visual question answering (Visual QA) [Anderson et al.(2018)Anderson, He, Buehler, Teney, Johnson, Gould, and Zhang, Antol et al.(2015)Antol, Agrawal, Lu, Mitchell, Batra, Lawrence Zitnick, and Parikh, Yang et al.(2016)Yang, He, Gao, Deng, and Smola, Zhu et al.(2016)Zhu, Groth, Bernstein, and Fei-Fei].

One key factor to the recent successes is the use of neural networks [Goodfellow et al.(2016)Goodfellow, Bengio, and Courville], especially the convolutional neural network (CNN) [LeCun et al.(1995)LeCun, Bengio, et al.] which captures the characteristics of human vision systems and the presence of objects in a scene. An object usually occupies nearby pixels and its location in the image does not change its appearance much, making CNNs, together with region proposals [Uijlings et al.(2013)Uijlings, Van De Sande, Gevers, and Smeulders], suitable models for object recognition and detection. Indeed, when CNN [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] and R-CNN [Girshick et al.(2014)Girshick, Donahue, Darrell, and Malik, Ren et al.(2015)Ren, He, Girshick, and Sun] were introduced, we saw a leap in benchmarked performance.

Such a leap, however, has yet to be seen in high-level tasks. Taking Visual QA [Antol et al.(2015)Antol, Agrawal, Lu, Mitchell, Batra, Lawrence Zitnick, and Parikh, Zhu et al.(2016)Zhu, Groth, Bernstein, and Fei-Fei, Goyal et al.(2017)Goyal, Khot, Summers-Stay, Batra, and Parikh, Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.] as an example, which takes an image and a question as inputs and outputs the answer. In the past four years there are over a hundred of publications, but the improvement of the most advanced model over a simple multi-layer perceptron (MLP) baseline is merely $5\sim 7\%$ [Anderson et al.(2018)Anderson, He, Buehler, Teney, Johnson, Gould, and Zhang, Hu et al.(2018)Hu, Chao, and Sha, Yu et al.(2018)Yu, Yu, Xiang, Fan, and Tao]. Besides, the improvement is mostly built upon deep image and language features [He et al.(2016)He, Zhang, Ren, and Sun, Hochreiter and Schmidhuber(1997), Mikolov et al.(2013)Mikolov, Sutskever, Chen, Corrado, and Dean], multi-modal fusion [Ben-Younes et al.(2017)Ben-Younes, Cadene, Cord, and Thome, Fukui et al.(2016)Fukui, Park, Yang, Rohrbach, Darrell, and Rohrbach], and attention [Anderson et al.(2018)Anderson, He, Buehler, Teney, Johnson, Gould, and Zhang, Yang et al.(2016)Yang, He, Gao, Deng, and Smola], not a fundamental breakthrough like CNNs in object recognition.

Interestingly, if we take the visual input away from Visual QA, there has been a long history of development in question answering (QA), especially on leveraging the knowledge graphs (or bases) [Bordes et al.(2014)Bordes, Chopra, and Weston, Berant et al.(2013)Berant, Chou, Frostig, and Liang, Lukovnikov et al.(2017)Lukovnikov, Fischer, Lehmann, and Auer, Yao and Van Durme(2014), Yih et al.(2015)Yih, Chang, He, and Gao]. The basic idea is to represent the knowledge via entities and their relationships and then query the structured knowledge during testing time. In the vision community, we have also seen attempts to construct the so-called scene graph [Johnson et al.(2015)Johnson, Krishna, Stark, Li, Shamma, Bernstein, and Fei-Fei, Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.] that can describe a visual scene in a similar way to a knowledge graph [Li et al.(2017)Li, Ouyang, Zhou, Wang, and Wang, Li et al.(2018)Li, Ouyang, Zhou, Shi, Zhang, and Wang, Wang et al.(2018b)Wang, Liu, Zeng, and Yuille, Xu et al.(2017)Xu, Zhu, Choy, and Fei-Fei, Yang et al.(2018a)Yang, Lu, Lee, Batra, and Parikh, Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi]. Nevertheless, we have not seen a notable improvement or comprehensive analysis in exploiting the structured scene graph for Visual QA, despite the fact that several recent works have started to incorporate it [Ben-Younes et al.(2019)Ben-Younes, Cadene, Thome, and Cord, Cadene et al.(2019)Cadene, Ben-younes, Cord, and Thome, Li et al.(2019)Li, Gan, Cheng, and Liu, Liang et al.(2019)Liang, Bai, Zhang, Qian, Zhu, and Mei, Shi et al.(2019)Shi, Zhang, and Li, Yang et al.(2018b)Yang, Yu, Yang, Qin, and Hu].

There are multiple possible reasons. Firstly, we may not yet have algorithms to construct high-quality scene graphs. Secondly, we may not yet have algorithms to effectively leverage scene graphs. Thirdly, perhaps we do not explicitly need scene graphs for Visual QA: either they do not offer useful information or existing algorithms have implicitly exploited them.

In this paper we aim to investigate these possible reasons. Specifically, we take advantage of the recently published graph network (GN) [Battaglia et al.(2018)Battaglia, Hamrick, Bapst, Sanchez-Gonzalez, Zambaldi, Malinowski, Tacchetti, Raposo, Santoro, Faulkner, et al.], which offers a flexible architecture to encode nodes (e.g, object entities and attributes), edges (e.g, object relationships), and global graph properties as well as perform (iterative) structured computations among them. By treating the question (and image) features as the input global graph properties and the answer as the output global properties, GNs can be directly applied to incorporate scene graphs and be learned to optimize the performance of Visual QA.

We conduct comprehensive empirical studies of GNs on the Visual Genome dataset [Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al., Chao et al.(2018)Chao, Hu, and Sha], which provides human annotated scene graphs. Recent algorithms on automatic scene graph generation from images [Li et al.(2017)Li, Ouyang, Zhou, Wang, and Wang, Li et al.(2018)Li, Ouyang, Zhou, Shi, Zhang, and Wang, Wang et al.(2018b)Wang, Liu, Zeng, and Yuille, Xu et al.(2017)Xu, Zhu, Choy, and Fei-Fei, Yang et al.(2018a)Yang, Lu, Lee, Batra, and Parikh, Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi] have also focused on Visual Genome, allowing us to evaluate these algorithms using Visual QA as a down-stream task (see Fig. 1). Our experiments demonstrate that human annotated or even automatically generated scene graphs have already captured essential information for Visual QA. Moreover, applying GNs without complicated attention and fusion mechanisms shows promising results but with a much cleaner architecture. By analyzing the GN features along the scene graph we can further interpret the reasoning process, making graph networks suitable models to leverage scene graphs for Visual QA tasks.

Related Work

Visual QA requires comprehending and reasoning with visual and textual information. Existing algorithms mostly adopt the pipeline that first extracts image and question features [Jabri et al.(2016)Jabri, Joulin, and Van Der Maaten], followed by multi-modal fusion [Ben-Younes et al.(2017)Ben-Younes, Cadene, Cord, and Thome, Fukui et al.(2016)Fukui, Park, Yang, Rohrbach, Darrell, and Rohrbach] and attention [Anderson et al.(2018)Anderson, He, Buehler, Teney, Johnson, Gould, and Zhang, Lu et al.(2016)Lu, Yang, Batra, and Parikh, Yang et al.(2016)Yang, He, Gao, Deng, and Smola] to obtain multi-modal features to infer the answer. While achieving promising results, it remains unclear if the models have been equipped with reasoning abilities or solely rely on exploiting dataset biases. Indeed, the performance gain between a simple MLP baseline and a complicated attention model is merely within $5\sim 7\%$ [Hu et al.(2018)Hu, Chao, and Sha]. The improvement brought by newly proposed models usually lies within $0.5\%$ [Ben-Younes et al.(2017)Ben-Younes, Cadene, Cord, and Thome, Mun et al.(2018)Mun, Lee, Shin, and Han, Anderson et al.(2018)Anderson, He, Buehler, Teney, Johnson, Gould, and Zhang].

Some new paradigms thus aim to better understand the intrinsic behavior of models [Kafle and Kanan(2017)]. One direction is to leverage object relationships for better scene understanding [Johnson et al.(2017a)Johnson, Hariharan, van der Maaten, Fei-Fei, Lawrence Zitnick, and Girshick, Santoro et al.(2017)Santoro, Raposo, Barrett, Malinowski, Pascanu, Battaglia, and Lillicrap, Shi et al.(2019)Shi, Zhang, and Li, Liang et al.(2019)Liang, Bai, Zhang, Qian, Zhu, and Mei, Li et al.(2019)Li, Gan, Cheng, and Liu]. The other aims for interpretable Visual QA via neuro-symbolic learning [Mao et al.(2019)Mao, Gan, Kohli, Tenenbaum, and Wu, Yi et al.(2018)Yi, Wu, Gan, Torralba, Kohli, and Tenenbaum, Vedantam et al.(2019)Vedantam, Desai, Lee, Rohrbach, Batra, and Parikh], developing symbolic functional programs to test machines’ reasoning capability in Visual QA. Nevertheless, many of these works mainly experiment on synthetic data.

QA with knowledge bases.

In conventional QA systems without visual inputs, exploiting knowledge bases (KBs) [Fader et al.(2014)Fader, Zettlemoyer, and Etzioni, Bollacker et al.(2008)Bollacker, Evans, Paritosh, Sturge, and Taylor, Hoffart et al.(2011)Hoffart, Suchanek, Berberich, Lewis-Kelham, De Melo, and Weikum] to store complex structured and unstructured information so as to support combinatorial reasoning has been widely investigated [Fader et al.(2014)Fader, Zettlemoyer, and Etzioni, Bordes et al.(2014)Bordes, Chopra, and Weston, Berant et al.(2013)Berant, Chou, Frostig, and Liang, Lukovnikov et al.(2017)Lukovnikov, Fischer, Lehmann, and Auer, Yao and Van Durme(2014), Yih et al.(2015)Yih, Chang, He, and Gao]. There are generally two kinds of KBs [Fader et al.(2014)Fader, Zettlemoyer, and Etzioni]: curated and extracted. Curated KBs, such as Freebase [Bollacker et al.(2008)Bollacker, Evans, Paritosh, Sturge, and Taylor] and YAGO2 [Hoffart et al.(2011)Hoffart, Suchanek, Berberich, Lewis-Kelham, De Melo, and Weikum], extract $\mathsf{<entity-relationship-entity>}$ triples from knowledge sources like Wikipedia and WordNet [Miller(1995)]. Extracted KBs [Banko et al.(2007)Banko, Cafarella, Soderland, Broadhead, and Etzioni, Carlson et al.(2010)Carlson, Betteridge, Kisiel, Settles, Hruschka, and Mitchell], on the other hand, extract knowledge in the form of natural language from millions of web pages.

The scene graph [Johnson et al.(2015)Johnson, Krishna, Stark, Li, Shamma, Bernstein, and Fei-Fei, Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.] examined in our work could be seen as a form of curated KBs that extracts triplets from images. Specifically, an image will be abstractly represented by a set of object entities and their relationships. Some Visual QA works indeed extract scene graphs followed by applying algorithms designed for reasoning on curated KBs [Krishnamurthy and Kollar(2013), Wang et al.(2018a)Wang, Wu, Shen, Dick, and van den Hengel]. In our work we also exploit scene graphs, but apply the recently published graph networks (GNs) [Battaglia et al.(2018)Battaglia, Hamrick, Bapst, Sanchez-Gonzalez, Zambaldi, Malinowski, Tacchetti, Raposo, Santoro, Faulkner, et al.] that can easily incorporate advanced deep features for entity and relationship representations and allow end-to-end training to optimize overall performance.

Scene graphs.

Lately, generating scene graphs from images has achieved notable progress [Xu et al.(2017)Xu, Zhu, Choy, and Fei-Fei, Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi, Li et al.(2017)Li, Ouyang, Zhou, Wang, and Wang, Newell and Deng(2017), Yang et al.(2018a)Yang, Lu, Lee, Batra, and Parikh, Krishna et al.(2018)Krishna, Chami, Bernstein, and Fei-Fei]. As an abstraction of images, scene graphs have been shown to improve image retrieval [Johnson et al.(2015)Johnson, Krishna, Stark, Li, Shamma, Bernstein, and Fei-Fei], generation [Johnson et al.(2018)Johnson, Gupta, and Fei-Fei], captioning [Gupta and Davis(2008), Anderson et al.(2016)Anderson, Fernando, Johnson, and Gould, Yao et al.(2018)Yao, Pan, Li, and Mei], video understanding [Ma et al.(2018)Ma, Kadav, Melvin, Kira, AlRegib, and Peter Graf, Vicol et al.(2018)Vicol, Tapaswi, Castrejon, and Fidler], and human object interaction [Kato et al.(2018)Kato, Li, and Gupta].

Recent works on Visual QA [Ben-Younes et al.(2019)Ben-Younes, Cadene, Thome, and Cord, Cadene et al.(2019)Cadene, Ben-younes, Cord, and Thome, Li et al.(2019)Li, Gan, Cheng, and Liu, Liang et al.(2019)Liang, Bai, Zhang, Qian, Zhu, and Mei, Shi et al.(2019)Shi, Zhang, and Li, Yang et al.(2018b)Yang, Yu, Yang, Qin, and Hu] and visual reasoning [Shi et al.(2019)Shi, Zhang, and Li, Johnson et al.(2017b)Johnson, Hariharan, van der Maaten, Hoffman, Fei-Fei, Lawrence Zitnick, and Girshick] have also begun to exploit scene graphs. However, they usually integrate multiple techniques (e.g, scene graph generation, attention, multi-modal fusion) into one hybrid model to obtain state-of-the-art performance. It is thus hard to conclude if scene graphs truly contribute to the improvement. Indeed, in an early study [Wu et al.(2017)Wu, Teney, Wang, Shen, Dick, and van den Hengel] of the Visual Genome dataset, only $40\%$ of the questions could be answered exactly with the human annotated scene graphs, assuming that the reasoning on the graphs is perfect In the study [Wu et al.(2017)Wu, Teney, Wang, Shen, Dick, and van den Hengel], a question is considered answerable by the scene graph if its answer exactly matches any node or relationship names or their combinations from the scene graph.. This ratio is surprisingly low considering the detailed knowledge encoded by scene graphs and the existing Visual QA models’ performance (e.g, $>50\%$ [Hu et al.(2018)Hu, Chao, and Sha]).

We therefore perform a systematic study on leveraging scene graphs without applying specifically designed attention and fusion mechanisms. The results can then faithfully indicate the performance gain brought by performing structured reasoning on scene graphs.

Leveraging Scene Graphs for Visual QA

In this section we describe the graph network (GN) architecture [Battaglia et al.(2018)Battaglia, Hamrick, Bapst, Sanchez-Gonzalez, Zambaldi, Malinowski, Tacchetti, Raposo, Santoro, Faulkner, et al.] and how we apply it to reason from a scene graph according to the input image and question for Visual QA. Fig. 2 gives an illustration of the GN-based Visual QA framework. In what follows, we first define the Visual QA task and the scene graph, and then introduce a general Visual QA framework.

A Visual QA model takes an image $i$ and a related question $q$ as inputs, and needs to output the correct answer $a$. In this work, we consider the multiple-choice Visual QA setting [Antol et al.(2015)Antol, Agrawal, Lu, Mitchell, Batra, Lawrence Zitnick, and Parikh, Zhu et al.(2016)Zhu, Groth, Bernstein, and Fei-Fei, Chao et al.(2018)Chao, Hu, and Sha], in which the model needs to pick the correct answer $a$ from a set of $K$ candidate answers $\mathcal{A}=\{a,d_{1},\cdots,d_{K-1}\}$. $d_{k}$ is called a negative answer or decoy. Nevertheless, our model can be easily extended into the open-ended setting with the answer embedding strategy [Hu et al.(2018)Hu, Chao, and Sha].

We explicitly consider encoding the image $i$ via a directed scene graph $g=(V,E)$ [Johnson et al.(2015)Johnson, Krishna, Stark, Li, Shamma, Bernstein, and Fei-Fei]. The node set $V=\{v_{n}\}_{n=1:N}$ contains $N$ objects of the image, where each $v_{n}$ records an object’s properties such as its name (e.g, a car or a person), attributes (e.g, colors, materials), location, and size. The edge set $E=\{(e_{m},s_{m},o_{m})\}_{m=1:M}$ contains $M$ pairwise relationships between nodes, where $e_{m}$ encodes the relationship name (e.g, on the top of); $s_{m},o_{m}$ are the indices of the subject and object nodes, respectively. For now let us assume that $g$ is given for every $i$. We note that $N$ and $M$ may vary among different images.

A general Visual QA model thus can be formulated as follows [Jabri et al.(2016)Jabri, Joulin, and Van Der Maaten, Chao et al.(2018)Chao, Hu, and Sha],

where $c\in\mathcal{A}$ is a candidate answer and $h_{\theta}$ is a learnable scoring function on how likely $c$ is the correct answer of $(i,q,g)$.

In Visual QA without the scene graph $g$, $h_{\theta}(c,i,q)$ can be modeled by an MLP with the concatenated features of $(c,i,q)$ as the input. For example, [Jabri et al.(2016)Jabri, Joulin, and Van Der Maaten, Chao et al.(2018)Chao, Hu, and Sha] represents $c$ and $q$ via the average word vectors and $i$ via CNN features. Alternatively, one can factorize $h_{\theta}(c,i,q)$ by $\gamma_{\theta}(\alpha_{\theta}(i,q),\beta_{\theta}(c))$, where $\alpha_{\theta}(i,q)$ is the joint image-question embedding and $\beta_{\theta}(c)$ is the answer embedding [Hu et al.(2018)Hu, Chao, and Sha]. $\gamma_{\theta}$ measures the compatibility of the two embeddings, e.g, by inner products. We denote the former as unfactorized models and the later as factorized models.

In Visual QA with scene graphs, we must define $h_{\theta}(c,i,q,g)$ so that it can take the nodes $V$ and edges $E$ into account as well as maintain the permutation invariant characteristic of a graph That is, even we permute the indices of the nodes or edges, the output of $h_{\theta}$ should be the same.. Moreover, $h_{\theta}(c,i,q,g)$ must be able to reason from the graph $g$ according to $(c,i,q)$. In the next subsection we introduce the graph network (GN), which provides a flexible computational architecture to fulfill the requirements above.

2 Graph networks (GNs)

The graph network (GN) proposed in [Battaglia et al.(2018)Battaglia, Hamrick, Bapst, Sanchez-Gonzalez, Zambaldi, Malinowski, Tacchetti, Raposo, Santoro, Faulkner, et al.] defines a computational block (module) that performs graph-to-graph mapping. That is, it takes a graph $g=(u,V,E)$ as the input and outputs an updated graph $g^{\prime}=(u^{\prime},V^{\prime},E^{\prime})$, which has the same graphical structure but different encoded information (i.e, $v_{n}$ and $e_{m}$ will be updated). The components $u$ and $u^{\prime}$ encode certain global properties (features) of the graph. For example, in Visual QA, $u$ can be used to encode the image $i$ and question $q$, while $u^{\prime}$ can be used to predict the final answer $a$.

There are various ways to define the updating procedure, as long as it maintains permutation invariant with respect to the graph. Here we describe a procedure that updates edges, nodes, and global features in order. We will then discuss how this updating procedure is particularly suitable for Visual QA with scene graphs.

The edge updating function $f^{e}_{\theta}$ is performed per edge $m$ according to its features (encoded information) $e_{m}$, the features of the subject and object nodes $v_{s_{m}}$ and ${v}_{o_{m}}$, and the global features $u$,

Node updates.

The subsequent node updates first aggregate the information from incoming edges of each node $n$,

where $\phi^{e\rightarrow v}$ is an aggregation function and $E^{\prime}_{n}=\{(e^{\prime}_{m},s_{m},o_{m})\}_{o_{m}=n}$ is the set of incoming edges. We then apply the node updating function $f^{v}_{\theta}$ to each node $n$,

Global updates.

Finally, the global updating function $f^{u}_{\theta}$ is applied to update the global features. It begins with aggregating the edge and node features,

where $E^{\prime}=\{(e^{\prime}_{m},s_{m},o_{m})\}_{m=1:M}$ and $V^{\prime}=\{v^{\prime}_{n}\}_{n=1:N}$. The updated $u^{\prime}$ is then computed by

In our studies, we assume that $u,v_{n},e_{m}$ (and $u^{\prime},v^{\prime}_{n},e^{\prime}_{m}$) each can be represented by a real vector. We make the following choices of aggregation and updating functions. The aggregation functions $\phi^{e\rightarrow v},\phi^{e\rightarrow u},\phi^{v\rightarrow u}$ should be able to take various number of inputs and must be permutation invariant with respect to the indices of their inputs. Representative options are element-wise average, sum, and max operations and we use element-wise average in the paper. The resulting $\overline{e^{\prime}_{n}}$, $\overline{e^{\prime}}$, and $\overline{v^{\prime}}$ are thus real vectors.

The updating functions $f^{e}_{\theta}$, $f^{v}_{\theta}$, and $f^{u}_{\theta}$ can be any learnable modules such as MLPs and CNNs. In this paper, we apply MLPs with the concatenated features as the input. We note that, all the updating functions are optimized simultaneously; all the edges and nodes share the same updating functions $f^{e}_{\theta}$ and $f^{v}_{\theta}$. More importantly, all the updating functions can be shared across different graphs even if they have different structures.

The resulting graph $g^{\prime}=(u^{\prime},V^{\prime},E^{\prime})$ then can be used to perform inference for tasks like Visual QA, or serve as the input to a subsequent GN block.

3 GN-based Visual QA with scene graphs

We apply GNs to Visual QA with scene graphs for multiple reasons. First, GNs explicitly consider the graph structures in its computations and can share the learned functions across different graphs Different images may have different scene graphs with varied numbers of nodes and edges.. Second, by encoding the question features (as well as the image and candidate answer features) into the global graph features $u$, GNs directly supports reasoning on graphs with appropriate choices of updating functions and procedures. Finally, GNs can be easily incorporated into Eq. (1) and learned to optimize the overall Visual QA performance. In the following we give more details.

We encode $q$ and $c$ (a candidate answer) with averaged word vectors [Mikolov et al.(2013)Mikolov, Sutskever, Chen, Corrado, and Dean] and $i$ with CNN features [He et al.(2016)He, Zhang, Ren, and Sun], following [Jabri et al.(2016)Jabri, Joulin, and Van Der Maaten, Chao et al.(2018)Chao, Hu, and Sha, Hu et al.(2018)Hu, Chao, and Sha]. We obtain the scene graph of each image either via human annotation [Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.] or via automatic scene graph generation [Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi]. Ideally, every node in a graph will be provided with a node name (e.g, car) and a set of attributes (e.g, red, hatchback) and we represent each of them by the averaged word vectors. We then concatenate them to be the node features $v_{n}$. For edges that are provided with the relationship name (e.g, on the top of), we again encode each $e_{m}$ by the average word vectors. We note that more advanced visual and natural language features [Devlin et al.(2019)Devlin, Chang, Lee, and Toutanova] can be applied to further improve the performance.

Learning unfactorized Visual QA models.

We model the scoring function $f_{\theta}(c,i,q,g)$ in Eq. (1) as below,

where $[\cdot]$ is a concatenation operation. That is, the output of the GN global updating function $f_{\theta}^{u}$ is a scalar indicating whether $c$ is the correct answer of the image-question pair. We learn the parameters of the GN block to optimize the binary classification accuracy, following [Jabri et al.(2016)Jabri, Joulin, and Van Der Maaten]. We note that, even if only $u^{\prime}$ is being used to predict the answer, all the three updating functions will be learned jointly according to the updating procedures in Sect. 3.2.

Learning factorized Visual QA models.

We can further factorize $f_{\theta}(c,i,q,g)$ in Eq. (1) by $\gamma_{\theta}(\alpha_{\theta}(i,q,g),\beta_{\theta}(c))$ [Hu et al.(2018)Hu, Chao, and Sha]. Specifically, we model $\alpha_{\theta}(i,q,g)$ by a GN block, $\beta_{\theta}(c)$ by an MLP, and $\gamma_{\theta}$ by another MLP following [Conneau et al.(2017)Conneau, Kiela, Schwenk, Barrault, and Bordes],

where $*$ indicates element-wise product. The factorized model offers faster training and inference and can be used for the open-ended setting directly: the GN block will be computed just once for an image-question pair no matter how many candidate answers are considered.

Experiments

We conduct experiments on the Visual Genome (VG) [Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.] dataset. VG contains 101,174 images annotated with in total 1,445,322 $(i,q,a)$ triplets. Chao et al [Chao et al.(2018)Chao, Hu, and Sha] further augment each triplet with 6 auto-generated decoys (incorrect answers) and split the data into 727K/283K/433K triplets for training/validation/testing, named qaVG. Following [Chao et al.(2018)Chao, Hu, and Sha, Hu et al.(2018)Hu, Chao, and Sha], we evaluate the accuracy of picking the correct answer from 7 candidates.

Scene graphs.

We evaluate both human annotated and machine generated scene graphs. VG directly provides human-annotated graphs [Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.], and we obtain machine generated ones by the start-of-the-art Neural Motifs (NM) [Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi]. Since the released NM model is trained from a different data split of qaVG, we re-train the model from scratch using the training images of qaVG. One key difference between the ground-truth and MotifsNet graphs is that the first ones provide both object names and attributes for nodes, while the later only provide names.

Baseline methods.

We compare with the unfactorized MLP models [Jabri et al.(2016)Jabri, Joulin, and Van Der Maaten, Chao et al.(2018)Chao, Hu, and Sha] and the factorized fPMC(MLP) and fPMC(SAN) [Hu et al.(2018)Hu, Chao, and Sha] models. SAN stands for stacked attention networks [Yang et al.(2016)Yang, He, Gao, Deng, and Smola]. We note that, while the MLP model is extremely simple, it is only outperformed by fPMC(SAN) with layers of attentions by $5\%$.

Variants of our models.

We denote the unfactorized GN model in Eq. (8) as u-GN and the factorized GN model in Eq. (9) as f-GN. We compare different combinations of input global features $u$ (i.e, $[c,i,q],[c,q]$ for u-GN and $[i,q],[q]$ for f-GN) with or without image features. We also compare different node features, with or without attributes. Finally, we consider removing edges (i.e, all nodes are isolated) or even removing the whole graph (i.e, Visual QA using the global features only).

Implementation details.

For the GN block, we implement each updating function by a one-hidden-layer MLP whose architecture is a fully-connected (FC) layer followed by batch normalization [Ioffe and Szegedy(2015)], ReLU, Dropout (0.5) [Srivastava et al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov], and another FC layer. The hidden layer is 8,192-dimensional. For $f^{v}_{\theta}$ and $f^{e}_{\theta}$, we keep the output size the same as the input. For $f^{u}_{\theta}$, u-GN has a 1-dimensional output while f-GN has a 300-dimensional output. For f-GN, $\beta_{\theta}$ and $\gamma_{\theta}$ are implemented by the same MLP architecture as above with a 8,192-dimensional hidden layer. $\beta_{\theta}$ has a 300-dimensional output while $\gamma_{\theta}$ has a 1-dimensional output.

We learn both u-GN and f-GN using the binary classification objective (i.e, if $c$ is the correct answer or not). We use the Adam optimizer [Kingma and Ba(2015)] with an initial learning rate $10^{-3}$ and a batch size 100. We divide the learning rate by $10$ whenever the validation accuracy decreases. We train for at most $30$ epochs and pick the best model via the validation accuracy.

2 Main results

We summarize the main results in Table 1, in which we compare our u-GN and f-GN models to existing models on multiple-choice Visual QA. Both u-GN (NM graphs) and f-GN (VG graphs) outperform existing unfactorized methods, while f-GN (VG graphs) is outperformed by fPMC(SAN$\star$) by $1\%$. We note that fPMC(SAN) uses LSTM [Hochreiter and Schmidhuber(1997)] to encode questions and fPMC(SAN$\star$) additionally uses it to encode answers. We thus expect that our GN models to have improved performance with better language features.

We then analyze different variants of our models. Comparing u-GN (No graphs) to u-GN (VG graphs) and u-GN (NM graphs), we clearly see the benefit brought by leveraging scene graphs. Specifically, even without the image features (i.e, input = $q$, $c$), u-GN (NM graphs) and u-GN (VG graphs) can already be on par or even surpass u-GN (No graphs) with input image features, suggesting that scene graphs, even automatically generated by machines, indeed encode essential visual information for Visual QA.

By further analyzing u-GN (VG graphs) and u-GN (NM graphs), we found that including node attributes always improve the performance, especially when no image features are used. We suggest that future scene graph generation should also predict node attributes.

We observe similar trends when applying the f-GN model.

We study the case where edges are removed and nodes are isolated: no edge updates are performed; node and global updates do not consider edge features. The result with f-GN (VG graphs) is $61.3\%$, compared to $62.5\%$ in Table 1, justifying the need to take the relationships between nodes into account for Visual QA.

Stacked GNs.

We investigate stacking multiple GN blocks for performance improvement. We stack two GN blocks: the updated edges and nodes of the first GN block are served as the inputs to the second one. The intuition is that the messages will pass through the scene graph with one more step to support complicated reasoning. When paired with stacked GNs, f-GN (VG graphs) achieves a gain of $+0.4\%$ in overall accuracy compared to $62.5\%$.

3 Analysis

We provide detailed results on qaVG with different question types in Table 2. We found that without input image features, scene graphs with node names significantly improve all question types but “when”, which needs holistic visual features. Even with image features, scene graphs with node names can still largely improve the “what”, “who”, and “number” types; the former two take advantage of node names while the later takes advantage of number of nodes. Adding the node attributes specifically benefits the “color” type. Overall, the VG graphs are of higher quality than NM graphs except for the “number” type. We surmise that well-trained object detectors adopted in [Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi] can capture smaller objects easily while human annotators might focus on salient ones.

Conclusion

In this paper we investigate if scene graphs can facilitate Visual QA. We apply the graph network (GN) that can naturally encode information on graphs and perform structured reasoning. Our experimental results demonstrate that scene graphs, even automatically generated by machines, can definitively benefit Visual QA if paired with appropriate models like GNs. Specifically, leveraging scene graphs largely increases the Visual QA accuracy on questions related to counting, object presence and attributes, and multi-object relationships. We expect that the GN-based model can be further improved by incorporating image features on nodes as well as advanced multi-modal fusion and attention mechanisms.

The computational resources are supported by the Ohio Supercomputer Center (PAS1510) [Center(1987)].

References

Supplementary Material

In this Supplementary Material, we provide details omitted in the main text.

Section A: Implementation details (Sect. 4.1 of the main text).

Section B: Additional experimental results (Sect. 4.2 and Sect. 4.3 of the main text).

Appendix A Implementation Details

In this section, we provide more details about the configuration of scene graph generation, scene graph encoding, the stacked GN model, and the corresponding training procedures.

We leverage the ground truth scene labels of the Visual Genome (VG) dataset [Krishna et al.(2017)Krishna, Zhu, Groth, Johnson, Hata, Kravitz, Chen, Kalantidis, Li, Shamma, et al.] as our human annotated scene graphs The VG scene graphs are obtained from https://visualgenome.org/without any modifications..

Machine generated scene graph.

The commonly used data split of scene graph generation research is different from $qaVG$ [Chao et al.(2018)Chao, Hu, and Sha] The $qaVG$ follows the same data split of Visual Genome dataset for Visual QA task.. Thus we retrain the start-of-the-art Neural Motifs (NM) [Zellers et al.(2018)Zellers, Yatskar, Thomson, and Choi] model only using the training images from $qaVG$. Specifically, we retrain both object detector and relationship classifier of the NM to ensure the model never uses testing images for training. We run the well-trained NM model on $qaVG$ to obtain the machine generated scene graph by removing entities (¡0.2) and relationships (¡0.1) with small probabilities. We show the scene graph detection performance in Table 3.

A.2 Scene graph encoding

A.3 Stacked GNs

We provide more details about stacked GNs. As mentioned in Sect. 4.2 of the main paper, the overall accuracy could be improved with respect to the number of GN blocks. Here, we propose one design choice of the stacked GNs as below,

where the first GN block only performs edge and node updating, and the updated properties will be served as the inputs of the second one. Finally, the resulting global feature $u^{\prime}$ can be used to perform inference in Visual QA. Similarly, multiple GN blocks can be stacked in such manner. We expect varied designs of multi-layer GN could be proposed for performance improvement, such as jointly learning global feature within the latent GN blocks and making GN blocks recurrent.

A.4 Optimization

For all above models, we train for at most 30 epochs using stochastic gradient optimization with Adam [Kingma and Ba(2015)]. The initial learning rate is $10^{-3}$, which is divided by 10 after $M$ epochs. We turn the $M$ on the validation set and choose the best model via validation performance.

Within each mini-batch, we sample 100 $(i,q,a)$ triplets. In order to prevent unbalanced training In $qaVG$ [Chao et al.(2018)Chao, Hu, and Sha], each $(i,q)$ pair contains 3 QoU-decoys (incorrect answers), 3 IoU-decoys (incorrect answers), and 1 target (correct answer). The machine tends to predict the dominant label if the training is performed among all samples., we follow the sampling strategy suggested by Chao et al [Chao et al.(2018)Chao, Hu, and Sha]. We randomly choose to use QoU-decoys or IoU-decoys for each triplet as negative samples when training. Then the binary classifier is trained on top of the target and 3 decoys for each triplet. That is, 100 triplets in the each mini-batch related to 400 samples with binary labels. In the testing stage, we evaluate the performance of picking the correct answer from all 7 candidates.

Appendix B Additional Results