Visual Coreference Resolution in Visual Dialog using Neural Module Networks

Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, Marcus Rohrbach

Introduction

The task of Visual Dialog involves building agents that ‘see’ (i.e. understand an image) and ‘talk’ (i.e. communicate this understanding in a dialog). Specifically, it requires an agent to answer a sequence of questions about an image, requiring it to reason about both the image and the past dialog history. For instance, in Fig. 1, to answer ‘What color is it?’, the agent needs to reason about the history to know what ‘it’ refers to and the image to find out the color. This generalization of visual question answering (VQA) to dialog takes a step closer to real-world applications (aiding visually impaired users, intelligent home assistants, natural language interfaces for robots) but simultaneously introduces new modeling challenges at the intersection of vision and language. The particular challenge we focus on in this paper is that of visual coreference resolution in visual dialog. Specifically, we introduce a new model that performs explicit visual coreference resolution and interpretable entity tracking in visual dialog.

It has long been understood that humans use coreferences, different phrases and short-hands such as pronouns, to refer to the same entity or referent in a single text. In the context of visually grounded dialog, we are interested in referents which are in the image, e.g. an object or person. All phrases in the dialog which refer to the same entity or referent in the image are called visual coreferences. Such coreferences can be noun phrases such as ‘a dragon head’, ‘the head’, or pronouns such as ‘it’ (Fig. 1). Especially when trying to answer a question that contains an anaphora, for instance the pronoun ‘it’, which refers to its full form (the antecedent) ‘a dragon head’, it is necessary to resolve the coreference on the language side and ground it to the underlying visual referent. More specifically, to answer the question ‘What color is it?’ in Fig. 1, the model must correctly identify which object ‘it’ refers to, in the given context. Notice that a word or phrase can refer to different entities in different contexts, as is the case with ‘it’ in this example.

Our approach to explicitly resolve visual coreferences is inspired from the functionality of variables or memory in a computer program. In the same spirit as how one can refer back to the contents of variables at a later time in a program without explicitly re-computing them, we propose a model which can refer back to entities from previous rounds of dialog and reuse the associated information; and in this way resolve coreferences.

Prior work on VQA has (understandably) largely ignored the problem of visual coreference resolution since individual questions asked in isolation rarely contain coreferences. In fact, recent empirical studies suggest that today’s vision and language models seem to be exploiting dataset-level statistics and perform poorly at grounding entities into the correct pixels. In contrast, our work aims to explicitly reason over past dialog interactions by referring back to previous references. This allows for increased interpretability of the model. As the dialog progresses (Fig. 1), we can inspect the pool of entities known to the model, and also visualize which entity a particular phrase in the question has been resolved to. Moreover, our explicit entity tracking model has benefits even in cases that may not strictly speaking require coreference resolution. For instance, by explicitly referring ‘dragon’ in Q3 (Fig. 1) back to a known entity, the model is consistent with itself and (correctly) grounds the phrase in the image. We believe such consistency in model outputs is a strongly desirable property as we move towards human-machine interaction in dialog systems.

Our main technical contribution is a neural module network architecture for visual dialog. Specifically, we propose two novel modules—Refer and Exclude—that perform explicit, grounded, coreference resolution in visual dialog. In addition, we propose a novel way to handle captions using neural module networks at a word-level granularity finer than a traditional sentence-level encoding. We show quantitative benefits of these modules on a reasoning-wise complicated but visually simple MNIST dialog dataset , where achieve near perfect accuracy. On the visually challenging VisDial dataset , our model not only outperforms other approaches but also is more interpretable by construction and enables word-level coreference resolution. Furthermore, we qualitatively show that our model is (a) more interpretable (a user can inspect which entities were detected and tracked as the dialog progresses, and which ones were referred to for answering a specific question), (b) more grounded (where the model looked to answer a question in the dialog), (c) more consistent (same entities are considered across rounds of dialog).

Related Work

We discuss: (a) existing approaches to visual dialog, (b) related tasks such as visual grounding and coreference resolution, and (c) neural module networks.

Visual Dialog. Though the origins of visual dialog can be traced back to , it was largely formalized by who collected human annotated datasets for the same. Specifically, paired annotators to collect free-form natural-language questions and answers, where the questioner was instructed to ask questions to help them imagine the hidden scene (image) better. On the other hand, dialogs from are more goal driven and contain yes/no questions directed towards identifying a secret object in the image. The respective follow up works used reinforcement learning techniques to solve this problem . Other approaches to visual dialog include transferring knowledge from a discriminatively trained model to a generative dialog model , using attention networks to solve visual coreferences , and more recently, a probabilistic treatment of dialogs using conditional variational autoencoders . Amongst these, is the closest to this work, while are complementary. To solve visual coreferences, relies on global visual attentions used to answer previous questions. They store these attention maps in a memory against keys based on textual representations of the entire question and answer, along with the history. In contrast, operating at a finer word-level granularity within each question, our model can resolve different phrases of a question, and ground them to different parts of the image, a core component in correctly understanding and grounding coreferences. E.g., ‘A man and woman in a car. Q: Is he or she driving?’, which requires resolving ‘he’ and ‘she’ individually to answer the question.

Grounding language in images and video. Most works in this area focus on the specific task of localizing a textual referential expression in the image or video . Similar to these works, one component of our model aims to localize words and phrases in the image. However, the key difference is that if the phrase being grounded is an anaphora (e.g., ‘it’, ‘he’, ‘she’, etc.), our model first resolves it explicitly to a known entity, and then grounds it by borrowing the referent’s visual grounding.

Coreference resolution. The linguistic community defines coreference resolution as the task of clustering phrases, such as noun phrases and pronouns, which refer to the same entity in the world (see, for example, ). The task of visual coreference resolution links the coreferences to an entity in the visual data. For example, links character mentions in TV show descriptions with their occurrence in the video, while links text phrases to objects in a 3D scene. Different from these works, we predict a program for a given natural language question about an image, which then tries to resolve any existing coreferences, to then answer the question. An orthogonal direction is to generate language while jointly grounding and resolving coreferences – e.g., explore this for movie descriptions. While out of scope for this work, it is an interesting direction for future work in visual dialog, especially when generating questions.

Neural Module Networks are an elegant class of models where an instance-specific architecture is composed from neural ‘modules’ (or building blocks) that are shared across instances. The high-level idea is inspired by ‘options’ or sub-tasks in hierarchical RL. They have been shown to be successful for visual question answering in real images and linguistic databases and for more complex reasoning tasks in synthetic datasets . For this, learn program prediction and module parameters jointly, end-to-end. Within this context, our work generalizes the formulation in from VQA to visual dialog by introducing a novel module to perform explicit visual coreference resolution.

Approach

Recall that visual dialog involves answering a question $Q_{t}$ at the current round $t$ , given an image $I$ , and the dialog history (including the image caption) $H=(\underbrace{C}_{H_{0}},\underbrace{(Q_{1},A_{1})}_{H_{1}},\cdots,\underbrace{(Q_{t-1},A_{t-1})}_{H_{t-1}})$ , by ranking a list of $100$ candidate answers $\mathcal{A}_{t}=\{A_{t}^{(1)},\cdots,A_{t}^{(100)}\}$ . As a key component for building better visual dialog agents, our model explicitly resolves visual coreferences in the current question, if any.

Towards this end, our model first identifies relevant words or phrases in the current question that refer to entities in the image (typically objects and attributes). The model also predicts whether each of these has been mentioned in the dialog so far. Next, if these are novel entities (unseen in the dialog history), they are localized in the image before proceeding, and for seen entities, the model predicts the (first) relevant coreference in the conversation history, and retrieves its corresponding visual grounding. Therefore, as rounds of dialog progress, the model collects unique entities and their corresponding visual groundings, and uses this reference pool to resolve any coreferences in subsequent questions.

Our model has three broad components: (a) Program Generation (Sec. 3.3), where a reasoning pathway, as dictated by a program, is predicted for the current question $Q_{t}$ , (b) Program Execution (Sec. 3.4), where the predicted program is executed by dynamically connecting neural modules to produce a context vector summarizing the semantic information required to answer $Q_{t}$ from the context ( $I,H$ ), and lastly, (c) Answer Decoding (Sec. 3.4), where the context vector $c_{t}$ is used to obtain the final answer $\hat{A}_{t}$ . We begin with a general characterization of neural modules used for VQA in Sec. 3.1 and then discuss our novel modules for coreference resolution (Sec. 3.2) with details of the reference pool. After describing the inner working of the modules, we explain each of the above three components of our model.

The main technical foundation of our model is the neural module network (NMN) . In this section, we briefly recap NMNs and more specifically, the attentional modules from . In the next section, we discuss novel modules we propose to handle additional challenges in visual dialog.

For a module $m$ , let $x_{vis}$ and $x_{txt}$ be the input image and text embeddings, respectively. In particular, the image embeddings $x_{vis}$ are spatial activation maps of the image $I$ from a convolutional neural network. The text embedding $x_{txt}$ is computed as a weighted sum of embeddings of words in the question $Q_{t}$ using the soft attention weights $\alpha$ predicted by a program generator for module $m$ (more details in Sec. 3.3). Further, let $\{a_{i}\}$ be the set of $n_{m}$ single-channel spatial maps corresponding to the spatial image embeddings, where $n_{m}$ is the number of attention inputs to $m$ . Denoting the module parameters with $\theta_{m}$ , a neural module $m$ is essentially a parametric function $y=f_{m}(x_{vis},x_{txt},\{a_{i}\}_{i=1}^{n_{m}};\theta_{m})$ . The output from the module $y$ can either be a spatial image attention map (denoted by $a$ ) or a context vector (denoted by $c$ ), depending on the module. The output spatial attention map $a$ feeds into next level modules while a context vector $c$ is used to obtain the final answer $A_{t}$ . The upper part of Tab. 1 lists modules we adopt from prior work, with their functional forms. We shortly summarize their behavior. A Find module localizes objects or attributes by producing an attention over the image. The Relocate module takes in an input image attention and performs necessary spatial relocations to handle relationships like ‘next to’, ‘in front of’, ‘beside’, etc. Intersection or union of attention maps can be obtained using And and Or, respectively. Finally, Describe, Exist, and Count input an attention map to produce the context vector by describing an attribute, checking for existence, or counting, respectively, in the given input attention map. As noted in , these modules are designed and named for a potential ‘atomic’ functionality. However, we do not enforce this explicitly and let the modules discover their expected behavior by training in an end-to-end manner.

2 Neural Modules for Coreference Resolution

We now introduce novel components and modules to handle visual dialog.

Reference Pool ( $P_{ref}$ ). The role of the reference pool is to keep track of entities seen so far in the dialog. Thus, we design $P_{ref}$ to be a dictionary of key-value pairs $(x_{txt},a)$ for all the Find modules instantiated while answering previous questions $(Q_{i})_{i=1}^{t-1}$ . Recall that Find localizes objects/attributes specified by $x_{txt}$ , and thus by storing each output attention map $y$ , we now have access to all the entities mentioned so far in the dialog with their corresponding visual groundings. Interestingly, even though $x_{txt}$ and $y$ are intermediate outputs from our model, both are easily interpretable, making our reference pool a semantic dictionary. To the best of our knowledge, our model is the first to attempt explicit, interpretable coreference resolution in visual dialog. While maintains a dictionary similar to $P_{ref}$ , they do not consider word/entity level coreferences nor do their keys lend similar interpretability as ours. With $P_{ref}=\{(x_{p}^{(i)},a_{p}^{(i)})\}_{i}$ as input to Refer, we can now resolve references in $Q_{t}$ .

Refer Module. This novel module is responsible for resolving references in the question $Q_{t}$ and ground them in the conversation history $H$ . To enable grounding in dialog history, we generalize the above formulation to give the module access to a pool of references $P_{ref}$ of previously identified entities. Specifically, Refer only takes the text embedding $x_{txt}$ and the reference pool $P_{ref}$ as inputs, and resolves the entity represented by $x_{txt}$ in the form of a soft attention $\alpha$ over $Q_{t}$ . in this section after introducing $P_{ref}$ . For the example shown in Fig. 2, $\alpha$ for Refer attends to ‘it’, indicating the phrase it is trying to resolve.

Not Module. Designed to focus on regions of the image not attended by the input attention map $a$ , it outputs $y=\text{norm}_{L_{1}}(1-a)$ , where norm ${}_{L_{1}}(.)$ normalizes the entries to sum to one. This module is used in Exclude, described next.

Exclude Module. To handle questions like ‘What other red things are present?’, which seek other objects/attributes in the image than those specified by an input attention map $a$ , we introduce yet another novel module – Exclude. It is constructed using Find, Not, and And modules as $y=$ And[Find[ $x_{txt},x_{vis}$ ], Not[ $a$ ]], where $x_{txt}$ is the text feature input to the Exclude module, for example, ‘red things’. More explicitly, Find first localizes all objects instances/attributes in the image. Next, we focus on regions of the image other than those specified by $a$ using Not[ $a$ ]. Finally, the above two outputs are combined via And to obtain the output $y$ of the Exclude module.

3 Program Generation

A program specifies the network layout for the neural modules for a given question $Q_{t}$ . Following , it is serialized through the reverse polish notation (RPN) . This serialization helps us convert a hard, structured prediction problem into a more tractable sequence prediction problem. In other words, we need a program predictor to output a series of module tokens in order, such that a valid layout can be retrieved from it. There are two primary design considerations for our predictor. First, in addition to the program, our predictor must also output a soft attention $\alpha_{ti}$ , over the question $Q_{t}$ , for every module $m_{i}$ in the program. This attention is responsible for the correct module instantiation in the current context. For example, to answer the question ‘What color is the cat sitting next to the dog?’, a Find module instance attending to ‘cat’ qualitatively serves a different purpose than one attending to ‘dog’. This is implemented by using the attention over $Q_{t}$ to compute the text embedding $x_{txt}$ that is directly fed as an input to the module during execution. Second, to decide whether an entity in $Q_{t}$ has been seen before in the conversation, it must be able to ‘peek’ into the history $H$ . Note that this is unique to our current problem and does not exist in . To this effect, we propose a novel augmentation of attentional recurrent neural networks with memory to address both the requirements (Fig. 2).

The program generation proceeds as follows. First, each of the words in $Q_{t}$ are embedded to give $\{w_{ti}\}_{i=1}^{T}$ , where $T$ denotes the number of tokens in $Q_{t}$ . We then use a question encoder, a multi-layer LSTM, to process $w_{ti}$ ’s, resulting in a sequence of hidden states $\{\hat{w}_{ti}\}_{i=1}^{T}$ (4). Notice that the last hidden state $h_{T}$ is the question encoding, which we denote with $q_{t}$ . Next, each piece of history $(H_{i})_{i=0}^{t-1}$ is processed in a similar way by a history encoder, which is a multi-layer LSTM akin to the question encoder. This produces encodings $(h_{i})_{i=0}^{t-1}$ (5) that serve as memory units to help the program predictor ‘peek’ into the conversation history. Using the question encoding $q_{t}$ , we attend over the history encodings $(h_{i})_{i=0}^{t-1}$ , and obtain the history vector $\hat{h}_{t}$ (6). The history-agnostic question encoding $q_{t}$ is then fused with the history vector $\hat{h}_{t}$ via a fully connected layer to give a history-aware question encoding $\hat{q}_{t}$ (7), which is fed into the program decoder.

The decoder is another multi-layer LSTM network (with hidden states $\{d_{ti}\}$ ) which, at every time step $i$ , produces a soft attention map $\alpha_{ti}$ over the input sequence ( $Q_{t}$ ) . This soft attention map for each module is used to compute the corresponding text embedding, $x_{txt}=\sum_{j}\alpha_{ti}^{(j)}w_{tj}$ . Finally, to predict a module token $m_{i}$ at time step $i$ , a weighted sum of encoder hidden states $e_{ti}$ (8) and the history-aware question vector $\hat{q}_{t}$ are combined via another fully-connected layer (9), followed by a softmax to give a distribution $P(m_{i}|\{m_{k}\}_{k=1}^{i-1},Q_{t},H)$ over the module tokens (10). During training, we minimize the cross-entropy loss $\mathcal{L}_{Q}^{prog}$ between this predicted distribution and the ground truth program tokens. Fig. 2 outlines the schematics of our program generator.

Modules on captions. As the image caption $C$ is also a part of the dialog (history $H_{0}$ at round ), it is desirable to track entities from $C$ via the coreference pool $P_{ref}$ . To this effect, we propose a novel extension of neural module networks to captions by using an auxiliary task that checks the alignment of a (caption, image) pair. First, we learn to predict a program from $C$ , different from those generated from $Q_{t}$ , by minimizing the negative log-likelihood $\mathcal{L}_{C}^{prog}$ , akin to $\mathcal{L}_{Q}^{prog}$ , of the ground truth caption program. Next, we execute the caption program on two images $I^{+}=I$ and $I^{-}$ (a random image from the dataset), to produce caption context vectors $c_{C}^{+}$ and $c_{C}^{-}$ , respectively. Note that $c_{C}^{+}$ and $c_{C}^{-}$ are different from the context vector $c_{t}$ produced from execution of the question program. Finally, we learn a binary classifier on top to output classes $+1/-1$ for $c_{C}^{+}$ and $c_{C}^{-}$ , respectively, by minimizing the binary cross entropy loss $\mathcal{L}_{C}^{aux}$ . The intuition behind the auxiliary task is: to rightly classify aligned $(C,I^{+})$ from misaligned $(C,I^{-})$ , the modules will need to localize and focus on salient entities in the caption. These entities (specifically, outputs from Find in the caption program) are then collected in $P_{ref}$ for explicit coreference resolution on $Q_{t}$ .

Entities in answers. Using an analogous argument as above, answers from the previous rounds $\{A_{i}\}_{i=1}^{t-1}$ could have entities necessary to resolve coreferences in $Q_{t}$ . For example, ‘Q: What is the boy holding? A: A ball. Q: What color is it?’ requires resolving ‘it’ with the ‘ball’ mentioned in the earlier answer. To achieve this, at the end of round $t-1$ , we encode $H_{t-1}=(Q_{t-1},A_{t-1})$ as $h^{ref}_{t}$ using a multi-layer LSTM, obtain the last image attention map $a$ fed to the last module in the program that produced the context vector $c_{t}$ , and add $(h^{ref},a)$ as an additional candidate to the reference pool $P_{ref}$ . Notice that $h^{ref}$ contains the information about the answer $A_{t-1}$ in the context of the question $Q_{t-1}$ , while $a$ denotes the image attention which was the last crucial step in arriving at $A_{t-1}$ in the earlier round. In resolving coreferences in $Q_{t}$ , if any, all the answers from previous rounds now become potential candidates by virtue of being in $P_{ref}$ .

4 Other Model Components

Program Execution. This component takes the generated program and associated text features $x_{txt}$ for each participating module, and executes it. To do so, we first deserialize the given program from its RPN to a hierarchical module layout. Next, we arrange the modules dynamically according to the layout, giving us the network to answer $Q_{t}$ . At this point, the network is a simple feed-forward neural network, where we start the computation from the leaf modules and feed outputs activations from modules at one layer as inputs into modules at the next layer (see Fig. 2). Finally, we feed a context vector $c_{t}$ produced from the last module into the next answer decoding component.

Answer Decoding. This is the last component of our model that uses the context vector $c_{t}$ to score answers from a pool of candidates $\mathcal{A}_{t}$ , based on their correctness. The answer decoder: (a) encodes each candidate $A_{t}^{(i)}\in\mathcal{A}_{t}$ with a multi-layer LSTM to obtain $o_{t}^{(i)}$ , (b) computes a score via a dot product with the context vector, i.e., $c_{t}^{T}o_{t}^{(i)}$ , and (c) applies a softmax activation to get a distribution over the candidates. During training, we minimize the negative log-likelihood $\mathcal{L}_{A}^{dec}$ of the ground truth answer $A^{gt}_{t}$ . At test time, the candidate with the maximum score is picked as $\mathcal{A}_{t}$ . Using nomenclature from , this is a discriminative decoder. Note that our approach is not limited to a discriminative decoder, but can also be used with a generative decoder (see supplement).

Training Details. Our model components have fully differentiable operations within them. Thus, to train our model, we combine the supervised loss terms from both program generation $\{\mathcal{L}_{Q}^{prog},\mathcal{L}_{C}^{prog},\mathcal{L}_{C}^{aux}\}$ and answer decoding $\{\mathcal{L}^{dec}_{A}\}$ , and minimize the sum total loss $\mathcal{L}^{total}$ .

Experiments

We first show results on the synthetic MNIST Dialog dataset , designed to contain complex coreferences across rounds while being relatively easy textually and visually. It is important to resolve these coreferences accurately in order to do well on this dataset, thus stress testing our model. We then experiment with a large visual dialog dataset on real images, VisDial , which offers both linguistic and perceptual challenge in resolving visual coreferences and grounding them in the image. Implementation details are in the supplement.

Dataset. The dialogs in the MNIST dialog dataset are grounded in images composed from a $4\times 4$ grid of MNIST digits . Digits in the grid have four attributes—digit class ( $0-9$ ), color, stroke, and background color. Each dialog has $10$ question-answer pairs, where the questions are generated through language templates, and the answers are single words. Further, the questions are designed to query attributes of target digit(s), count digits with similar attributes, etc., all of which need tracking of the target digits(s) by resolving references across dialog rounds. Thus, coreference resolution plays a crucial part in the reasoning required to answer the question, making the MNIST dataset both interesting and challenging (Fig. 3). The dataset contains $30k$ training, $10k$ validation, and $10k$ test images, with three $10$ -round dialogs for each image.

Models and baselines. Taking advantage of single-word answers in this dataset, we simplify our answer decoder to be a $N$ -way classifier, where $N$ is the number of possible answers. Specifically, the context vector $c_{t}$ now passes through a fully connected layer of size $N$ , followed by softmax activations to give us a distribution over possible answer classes. At training time, we minimize the cross-entropy $\mathcal{L}_{A}^{dec}$ of the predicted answer distribution with the ground truth answer, at every round. Note that single-word answers also simplify evaluation as answer accuracy can now be used to compare different models. We further simplify our model by removing the memory augmentation to the program generator, i.e., $\hat{q}_{t}=q_{t}$ (7), and denote it as CorefNMN. In addition to the full model, we also evaluate an ablation, CorefNMN $\backslash$ Seq, without $\Delta_{i}t$ that additionally captured sequential nature of dialog (see Refer description). We compete against the explicit reasoning model (NMN) and a comprehensive set of baselines AMEM, image-only (I), and question-only (Q), all from .

Supervision. In addition to the ground truth answer, we also need program supervision for questions to learn the program generation. For each of the $5$ ‘types’ of questions, we manually create one program which we apply as supervision for all questions of the corresponding type. The type of question is provided with the question. Note that our model needs program supervision only while training, and uses predictions from program generator at test time.

Results. Tab. 4.1 shows the results on MNIST dataset. The following are the key observations: (a) The text-only Q ( $36.6\%$ ) and image-only I ( $20.2\%$ ) do not perform well, perhaps as expected as MNIST Dialog needs resolving strong coreferences to arrive at the correct answer. For the same reason, NMN has a low accuracy of $23.8\%$ . Interestingly, Q outperforms NMN by around $13\%$ (both use question and image, but not history), possibly due to the explicit reasoning nature of NMN prohibiting it from capturing the statistic dataset priors. (b) Our CorefNMN outperforms all other models with near perfect accuracy of $99.3\%$ . Examining the failure cases reveals that most of the mistakes made by CorefNMN was due to misclassifying qualitatively hard examples from the original MNIST dataset. (c) Factoring the sequential nature of the dialog additionally in the model is beneficial, as indicated by the $10.6\%$ improvement in CorefNMN, and $7.2\%$ in AMEM. Intuitively, phrases with multiple potential referents, more often than not, refer to the most recent referent, as seen in Fig. 1, where ‘it’ has to be resolved to the closest referent in history. Fig. 3 shows a qualitative example.

Models and baselines. In addition to the CorefNMN model described in Sec. 3, we also consider ablations without the memory network augmented program generator (CorefNMN $\backslash$ Mem) or the auxiliary loss $\mathcal{L}_{C}^{aux}$ to train modules on captions (CorefNMN $\backslash\mathcal{L}_{C}^{aux}$ ), and without both (CorefNMN $\backslash$ Mem $\backslash\mathcal{L}_{C}^{aux}$ ). As strong baselines, we consider: (a) neural module network without history with answer generation, (b) the best discriminative model based on memory networks MN-QIH-D from , (c) history-conditioned image attentive encoder (HCIAE-D-MLE) , and (d) Attention-based visual coreference model (AMEM+SEQ-QI) . We use ImageNet pretrained VGG-16 to extract $x_{vis}$ , and also ResNet-152 for CorefNMN. Further comparisons are in supplement.

Evaluation. Evaluation in visual dialog is via retrieval of the ground truth answer $A^{gt}_{t}$ from a pool of $100$ candidate answers $\mathcal{A}_{t}=\{A_{t}^{(1)},\cdots A_{t}^{(100)}\}$ . These candidates are ranked based the discriminative decoder scores. We report Recall $@k$ for $k=\{1,5,10\}$ , mean rank, and mean reciprocal rank (MRR), as suggested by , on the set of $40k$ validation images (there is not test available for v0.9).

Supervision. In addition to the ground truth answer $A^{gt}_{t}$ at each round, our model gets program supervision for $Q_{t}$ , to train the program generator. We automatically obtain (weak) program supervision from a language parser on questions (and captions) and supervision to predict for Refer from an off-the-shelf text coreference resolution toolhttps://github.com/huggingface/neuralcoref, based on . For questions that are a part of coreference chain, we replace Find with Refer in the parser supervised program. Our model predicts everything from the questions at test time.

Results. We summarize our observations from Tab. 4.1 below: (a) Our CorefNMN outperforms all other approaches across all the metrics, highlighting the importance of explicitly resolving coreferences for visual dialog. Specifically, our R $@k$ ( $k=1,2,5$ ) is at least 1 point higher than the best prior work (AMEM+SEQ-QI), and almost 2 points higher than NMN. (b) Removing memory augmentation (CorefNMN $\backslash$ Mem) hurts performance uniformly over all metrics, as the model is unable to peek into history to decide when to resolve coreferences via the Refer module. Modules on captions seems to have varied effect on the full model, with decrease in R $@1$ , but marginal increase or no effect in other metrics. (c) Fig. 4 illustrates the interpretable and grounded nature of our model.

Dataset. Das et al. recently released VisDial v1.0 dataset. Specifically, VisDial v1.0 comprises of: (a) A re-organization of train and validation splits from v0.9 to form the new train v1.0. Thus, train v1.0 now contains $120k$ images with $10-$ round dialogs for each images, resulting in a total of $1.2$ million question-answer pairs. (b) An additional 10k COCO-like images from Flickr, on which crowd-sourced dialogs between pairs of humans were collected similar to v0.9. The 10k images are further split into 2k validation (val v1.0) and 8k test sets (test-std v1.0). Dense candidate option annotations, which indicate the correctness of each candidate in the pool, were also collected for these 10k images. Each image in the val v1.0 split is associated with a $10-$ round dialog, while an image in test-std v1.0 has a variable-round dialog.

Additional Metrics and Models. Just as in the previous version (v0.9), the performance on the VisDial v1.0 dataset is benchmarked using standard retrieval metrics like Recall $@k$ ( $k=\{1,5,10\}$ ), mean reciprocal rank (MRR) and mean rank. Further, Das et al. also propose to use normalized discounted cumulative gain (NDCG) to score the sorted pool of candidate answers, to evaluate on VisDial v1.0https://visualdialog.org/challenge/2018. Intuitively, NDCG penalizes accurate answers that appear lower in the sorted pool based on a logarithmic weighting scheme, normalizing for the number of accurate answers across instances. We train our CorefNMN model on train v1.0 and report numbers on test-std v1.0 split. We compare against LF-QIH-D, HRE-QIH-D, and MN-QIH-D from , and out-of-the-box neural module network (NMN) . The LF-QIH-D, HRE-QIH-D, and MN-QIH-D use VGG image features, while the neural module based models (CorefNMN and NMN) use ResNet-152 features.

Performance on VisDial v1.0 is given in Tab. 4.3. Our CorefNMN outperforms all other approaches on all metrics, except the neural module baseline (NMN) on the NDCG metric. We note that recent state-of-the-art, as reported on the leaderboardhttps://evalai.cloudcv.org/web/challenges/challenge-page/103/leaderboard/298, has reached up to 0.578 (NDCG) from team DL-61, but the approach and other details (e.g. features, use of an ensemble) are not fully known and unpublished at this point in time.

We introduced a novel model for visual dialog based on neural module networks that provides an introspective reasoning about visual coreferences. It explicitly links coreferences and grounds them in the image at a word-level, rather than implicitly or at a sentence-level, as in prior visual dialog work. Our CorefNMN outperforms prior work on both the MNIST dialog dataset (close to perfect accuracy), and on VisDial dataset, while being more interpretable, grounded, and consistent by construction.

Acknowledgements. This work was supported in part by NSF, AFRL, DARPA, Siemens, Google, Amazon, ONR YIPs and ONR Grants N00014-16-1-{2713,2793}, N000141210903. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.

Sec. 0.A shows the results of our model using a discriminative decoder with image features extracted using ImageNet pretrained ResNet-152 , showing superior performance of our explicit coreference model CorefNMN,

Sec. 0.B details our experiments with a generative answer decoder,

Implementation details for our experiments are given in Sec. 0.C, and

Schematics of our novel Refer module are in Fig. 5, Fig. 6 visualizes the auxiliary task used to run modules on captions, a novel way to handle captions at a fine word-level granularity, and Fig. 7 shows another qualitative example from VisDial.

Appendix 0.A Discriminative Decoder Experiments

Comparisons with ResNet-152 features. As mentioned in Sec. 4.2 of the main paper, the models trained on VisDial v0.9 used an ImageNet pretrained VGG to extract the image features $x_{vis}$ . In this section, we present results where a pretrained ResNet-152 was used to obtain the image features for our CorefNMN model, in Tab. 0.B. For a fair comparison, we obtained performance metrics from the authors for few of these baselines (MNQIH-G, LF-QH-G), and retrain NMN with ResNet-152 features.

Appendix 0.B Generative Decoder Experiments

The main paper describes a discriminative answer decoder (Sec.3.4) and presents results for the same (Sec. 4.2) on VisDial v0.9 dataset. As a reminder, a discriminative decoder takes the context vector $c_{t}$ as an input, and scores candidate answers according to their correctness. In other words, a discriminative decoder needs to be presented with a list of candidate answers and cannot ‘generate’ novel answers. We now introduce a generative answer decoder and present results on VisDial v0.9 dataset.

Generative Answer Decoder is a language model composed of a multi-layer LSTM with $c_{t}$ as its initial state. During training, we minimize the negative log-likelihood $\mathcal{L}^{dec}_{A}$ of the ground truth answer $A^{gt}_{t}$ with respect to the model. At test time, we use the decoder to score all candidates in the answer pool $\mathcal{A}_{t}$ by model log-likelihood, and rank them accordingly. Note that this ranking of candidate answers is done to comply with the evaluation protocol of VisDial v0.9, and is not a limitation of generative answer decoder, unlike the discriminative one. Thus, the generative answer decoder can potentially be used to generate novel answers to a given question in the visual dialog via language generation.

Models and baselines. We denote our model, as described in Sec. 3, with CorefNMN to indicate that the model uses history $H$ . We also consider ablations which do not have the memory network augmented program generator (CorefNMN $\backslash$ Mem), or, the auxiliary loss $\mathcal{L}_{C}^{aux}$ to train modules on captions (CorefNMN $\backslash\mathcal{L}_{C}^{aux}$ ), and a combination of both (CorefNMN $\backslash$ Mem $\backslash\mathcal{L}_{C}^{aux}$ ). As strong baselines, we consider: (a) neural module network without history with answer generation, (b) the best generative model based on memory networks MN-QIH-G from , in addition to their LF-QIG-G and HRE-QIH-G models, and (c) history-conditioned image attentive encoder (HCIAE-G-MLE) . We do not consider the HCIAE-G-DIS model with perceptual loss as its contribution is complementary to our model. Our model uses ImageNet pretrained ResNet-152 to extract features for images $x_{vis}$ , while some of these baseline models originally use VGG-16 features. For a fair comparison, we obtained performance metrics from the authors for few of these baselines (MN-QIH-G, LF-QH-G), and report the others (HCIAE-G-MLE) as is.

All our models are implemented with Tensorflow v1.0 . To optimize, we use Adam with a learning rate of $0.0001$ . Gradients at each iteration are clamped to $[-2.0,2.0]$ to avoid gradient explosion. To preprocess text, we follow , i.e., we lowercase all questions and answers, and tokenize using the Python NLTK framework . We then construct a dictionary of all words that appear at least five times in the training set. Specific model hyperparameters for each experiment are given below.

Due to the synthetic nature, the text in the dialog is of low variability and is made up of a small vocabulary. Thus, we only use a single-layered LSTM with a hidden size of $64$ to encode both questions and history. For each word in our vocabulary of size $73$ , we learn embeddings with $32$ dimensions. We learn a similar dimensional embedding for each of our modules. To extract image features, we design a convolutional neural network (CNN) with the same architecture as . Specifically, our CNN has four $3\times 3$ convolutional layers, each followed by a batch norm, ReLU non-linearity, and a $2\times 2$ max pool layer. While the first two convolutional layers have $32$ feature channels, the last two have $64$ channels each. To pick the best model, we use early stoppage on the provided validation set of $10k$ images.

C.2 VisDial Dataset

Each LSTM used in our VisDial experiments has two layers and with a hidden size of $1000$ . To represent images, we use convolutional features before the final mean pooling from a ImageNet pre-trained ResNet-152 model. Further, we also add two additional dimensions indicating the $X$ (columns) and $Y$ (rows) locations respectively, to facilitate the model in handling spatial reasoning. With a large vocabulary of around $8k$ words, our word and module embeddings are $300$ dimensional vectors. We also initialize our word embeddings with GloVe . We pick the best model via early stopping using mean reciprocal rank metric on a subset of $3k$ images, set aside from the $83k$ training images of VisDial v0.9.

C.3 Document Changelog

To help the readers track changes to this document, a brief changelog describing the revisions is provided below:

v0: ECCV 2018 camera-ready version (not on arXiv).

v1: Initial arXiv version: Added experiments on VisDial v1.0 dataset.