Meta Module Network for Compositional Visual Reasoning

Introduction

Visual reasoning requires a model to learn strong compositionality and generalization abilities, i.e., understanding and answering compositional questions without having seen similar semantic compositions before. Such compositional visual reasoning is a hallmark for human intelligence that endows people with strong problem-solving skills given limited prior knowledge. Neural module networks (NMNs) have been proposed to perform such complex reasoning tasks. NMN requires a set of pre-defined functions and explicitly encodes each function into unique shallow neural networks called modules, which are composed dynamically to build an instance-specific network for each input question. This approach has high compositionality and interpretability, as each module is designed to accomplish a specific sub-task, and multiple modules can be combined to perform an unseen combination of functions during inference.

However, NMN suffers from two major limitations. 1) Scalability: When the complexity of the task increases, the set of functional semantics scales up, so does the number of neural modules. For example, in the recent GQA dataset , a larger set of functions (48 vs 25, see Appendix for details) with varied arity is involved, compared to previous CLEVR dataset . To solve this task with standard NMN framework , an increased amount of modules are required to implement for these functions, leading to higher model complexity. 2) Generalizability: Since the model is tied to a pre-defined set of functionalities when a new question with unseen functional semantics appears, no existing module can be readily applied to the new semantics, limiting the model’s ability to generalize.

In order to enhance NMN for more practical use, we propose Meta Module Network (MMN). As depicted in Figure 1, MMN is based on a meta module (a general-purpose neural network), which can take a function recipe (key-value pairs) as input to embed it into continuous vector space and feed it as a side input to instantiate different instance modules. Depending on the specification provided in function recipe, different instance modules are created to accomplish different sub-tasks. As different instance modules inherit the same parameters from the meta module, model complexity remains the same as the function set enlarges. For example, if the recipe has $K$ slots, and each slot takes $N$ values, a compact vector of the recipe can represent up to $N^{K}$ different functions. This effectively solves the scalability issue of NMN. When creating instance modules for specified sub-tasks, the input recipes are encoded into the embedding space for function instantiation. Thus, when an unseen recipe appears, it can be encoded into the embedding space to instantiate a novel instance module based on embedding similarity with previously observed recipes. This metamorphous design effectively overcomes NMN’s limitation on generalizability.

MMN draws inspiration from Meta Learning as a learning-to-learn approach - instead of learning independent functions to solve different sub-tasks, MMN learns a meta-function that can generate a function to solve specific sub-task.borrowing the concept of Meta as in: a book in which a character is writing a book, or a movie in which a character is making a movie, can be described as Meta. The learning algorithm of MMN is based on a teacher-student framework to provide module supervision: an accurate “symbolic teacher” first traverses a given scene graph to generate the intermediate outputs for the given functions from specific recipe; the intermediate outputs are then used as guidelines to teach each “student” instance module to accomplish its designated sub-task in the function recipe. The module supervision together with the original question answering supervision are used jointly to train the model.

The model architecture of MMN is illustrated in Figure 2: ($1$) the coarse-to-fine semantic parser converts an input question into its corresponding program (i.e., a sequence of functions); ($2$) the meta module is instantiated into different instance modules based on the function recipes of the predicted program, which is composed into an execution graph; ($3$) the visual encoder encodes the image features that are fed to the instance modules; ($4$) during training, we provide intermediate module supervision and end-step answer supervision to jointly train all the components.

Our main contributions are summarized as follows. ($i$) We propose Meta Module Network that effectively extends the scalability and generalizability of NMN for more practical use, allowing it to handle tasks with unseen compositional function from new domain. With a metamorphous meta module learned through teacher-student supervision, MMN provides great flexibility on model design and model training that cleverly overcomes the rigid hand-crafting of NMN. ($ii$) Experiments conducted on CLEVR and GQA benchmarks demonstrate the scalability of MMN to accommodate larger set of functions. ($iii$) Qualitative visualization on the inferential chain of MMN also demonstrates its superb interpretability and strong transferability.

Related Work

Neural Module Networks By parsing a question into a program and executing the program through dynamically composed neural modules, NMN excels in interpretability and compositionality by design . For example, IEP and N2NMN aims to make the whole model end-to-end trainable via the use of reinforcement learning. Stack-NMN proposes to make soft layout selection so that the whole model is fully differentiable, and Neural-Symbolic VQA proposes to perform completely symbolic reasoning by encoding images into scene graphs. However, its success is mostly restricted to simple datasets with a limited set of functions, whose performance can be surpassed by simpler methods such as relational network and FiLM . Our MMN is a module network in concept, thus possessing high interpretability and compositionality. However, different from traditional NMN, to enhance its scalability and generalizability, MMN uses only a general-purpose meta module for program execution recurrently, which makes MMN inherently a monolithic network, ensuring its strong empirical performance without sacrificing model interpretability.

Monolithic Network Another line of research on visual reasoning is focused on designing monolithic network architecture, such as MFB , BAN , DCN , and MCAN . These black-box models have achieved strong performance on challenging datasets, such as VQA and GQA , surpassing the NMN approach. More recently, multimodal pre-training algorithms have been proposed that further lift state of the art on diverse tasks such as VQA , NLVR2 , and VCR . They use a unified neural network to learn general-purpose reasoning skills , which is more flexible and scalable than NMN. Most monolithic networks for visual reasoning resort to attention mechanism for multimodal fusion . To realize multi-hop reasoning on complex questions, SAN , MAC and MuRel models have been proposed. As the monolithic network is not tied to any pre-defined functionality, it has better generalizability to unseen questions. However, since the reasoning procedure is conducted in the feature space, such models usually lack interpretability, or the ability to capture the compositionality in language.

GQA Models GQA was introduced in for real-world visual reasoning. Simple monolithic networks , MAC netowrk , and language-conditioned graph neural networks have been developed for this task. LXMERT , a large-scale pre-trained encoder, has also been tested on this dataset. Recently, Neural State Machine (NSM) proposed to first predict a probabilistic scene graph, then perform multi-hop reasoning over the graph for answer prediction. The scene graph serves as a strong prior to the model. Our model is designed to leverage dense visual features extracted from object detection models, thus orthogonal to NSM and can be enhanced with their scene graph generator once it is publicly available. Different from the aforementioned approaches, MMN also performs explicit multi-hop reasoning based on predicted programs to demonstrate inferred reasoning chain.

Proposed Approach

Figure 2 provides an overview of Meta Module Network (MMN), which consists of three components: ($i$) Program Generator (Sec. 3.1), which generates a functional program from the input question; ($ii$) Visual Encoder (Sec. 3.2), which consists of self-attention and cross-attention layers on top of an object detection model, transforming an input image into object-level feature vectors; ($iii$) Meta Module (Sec. 3.3), which can be instantiated to different instance modules to execute the program for answer prediction.

Similar to other programming languages, we define a set of syntax rules for building valid programs and a set of semantics to determine the functionality of each program. Specifically, we define a set of functions $\mathcal{F}$ with their fixed arity $n_{f}\in\{1,2,3,4\}$ based on the “semantic string” provided in GQA dataset . The definitions for all the functions are provided in the Appendix. The defined functions can be divided into 10 different function types (e.g., “relate”, “verify”, “filter”, “choose”), and each abstract function type is further implemented with different realizations based on fine-grained functionality (e.g., “verify”: “verify_attribute”, “verify_geometric”, “verify_relation”, “verify_color”), which take different arguments as inputs.

In total, there are 48 different functions defined in GQA environment, which poses great challenges to the scalability in visual reasoning. The returned values of these functions are List of Objects, Boolean, or String (Object refers to the detected bounding box, and String refers to object name, attributes, relations, etc.) A program $P$ is viewed as a sequence of function calls $f_{1},\cdots,f_{L}$. For example, in Figure 3, $f_{2}$ is Relate(, beside, boy), the functionality of which is to find a boy who is beside the objects returned by $f_{1}:$ Select(ball). Formally, we call Relate the “function name”, the “dependency” (previous execution results), and beside, boy the “arguments”. By exploiting the dependency relationship between functions, we build an execution graph, where each node represents a function and each edge denotes an input-output dependency relationship between connected nodes.

In order to generate syntactically plausible programs, we follow and adopt a coarse-to-fine two-stage generation paradigm, as illustrated in Figure 3. We first encode the question as a context vector, and then decode a sketch step by step (the sketch only contains the function name without arguments). Once the sketch is decoded, the arity and types of the decoded functions are determined. For example, after generating “Relate”, there are three arguments following this function with the first argument as the dependency. The sketch is thus expanded as “Relate (#1, #2, #3)”, where “#$i$” denotes the $i$-th unfilled slot. We then apply a fine-grained generator to fill in the slots of dependencies and arguments for the sketch as a concrete program $P$. During the slot-filling phase, we mask the infeasible tokens at each time step to greatly reduce the search space.

Such a two-stage generation process helps guarantee the plausibility and grammaticality of synthesized programs. For example, if function Filter is sketched, we know there are two tokens required to complete the function. The first token should be selected from the dependency set (, , …), while the second token should be selected from the attribute set (e.g., color, size). With these syntactic constraints to shrink the search space, our program synthesizer can achieve a 98.8% execution accuracy (i.e., returning the same result as the ground truth after execution) compared to execution accuracy of 93% of a standard sequence generation model.

2 Visual Encoder

3 Meta Module

As opposed to having a full inventory of task-specific parameterized modules for different functions as in NMN , we design an abstract meta module that can be instantiated into instance modules based on an input function recipe, which is a set of pre-defined key-value pairs specifying the properties of the function. As exemplified in Figure 4, when taking recipe Function:relate; Geometric:to the left as the input, the Recipe Embedder produces a recipe vector to instantiate the abstract meta module into a “geometric relation” module, which specifically searches for target objects that the current object is to the left of. When taking recipe Function:filter; Type:color; Attribute:pink as input, the Embedder will instantiate the meta module into a “filter pink” module, which specifically looks for the objects with pink color in the input objects.

4 Learning

In order to train the meta module to learn the instantiation process from given function recipes (i.e., how to generate functions), we propose a Teacher-Student framework depicted in Figure 5. First, we define a Symbolic Executor as the “Teacher”, which can take the input function $f$ and traverse the provided training scene graph to obtain intermediate results (i.e., distribution over the objects on the ground-truth scene graph). The “Teacher” exhibits it as a guideline $\bm{\gamma}$ for the “Student” instance module $g_{f}$ to follow.

Symbolic Teacher We first execute the program $P=f_{1},\cdots,f_{L}$ on the ground-truth scene graph $G$ provided in the training data to obtain all the intermediate execution results. According to the function definition (see Appendix for details), the intermediate results are either of type List of Objects or Boolean. The strategy of representing the results follows: $(i)$ Non-empty List of Objects: use the first element’s vertexes $[x_{1},y_{1},x_{2},y_{2}]$; $(ii)$ Empty List of Objects: use dummy vertexes $$;$(iii)$“True” from Boolean: use the vertexes from last step;$(iv)$“False” from Boolean: use dummy vertexes as in$(ii)$. Therefore, the intermediate results can be unified in the form of quadruples denoted as$b_{i}$. To align these quadruple$b_{i}$regions with the regions$\mathbf{R}$proposed by the object detector from the visual encoder, we compute its overlap against all the regions$r_{j}\in R$as$a_{i,j}=\frac{Intersect(b_{i},r_{j})}{Union(b_{i},r_{j})}$. Based on whether there exist any overlaps, we handle the following two cases differently:

We call distributions $\gamma_{i,j}$ the guideline from symbolic teacher. Please refer to the rightmost part of Figure 5 to better understand the computation.

Joint Optimization Formally, given the quadruple of $(P,Q,\mathbf{R},a)$ and the pre-computed guideline distribution $\gamma$, we propose to add KL divergence to the standard loss function with a balancing factor $\eta$:

The objective jointly provides the module supervision and end-task supervision, the parameters $\phi,\psi$ of visual encoder and the module network are optimized w.r.t to it.

Experiments

In this section, we conduct the following experiments. $(i)$ We first evaluate the proposed Meta Module Network on CLEVR datast to preliminarily validate its effectiveness on the synthetic environment. $(ii)$ We then evaluate on the GQA v1.1 dataset and compare it with state-of-the-art methods. As GQA is a more realistic testbed to demonstrate the scalability and generalizability of our model, we will focus on it throughout our experiments. $(iii)$ We provide visualization of the inferential chains and perform fine-grained error analysis based on that. $(iv)$ We design synthesized experiments to quantitatively measure our model’s generalization ability towards unseen functional semantics.

The CLEVR dataset consists of rendered images featuring 3D-objects of various shapes, materials, colors, and sizes, coupled with machine-generated compositional multi-step questions that measure performance on an array of challenging reasoning skills. Each question is also associated with a tree-structured functional program that was used to generate it, specifying the reasoning operations that should be performed to compute the answer. We use the standard training set containing 700K questions-program pairs to train our model and parser along with the provided scene graphs. We define a set of function $\mathcal{F}$ with arity of $n_{f}\in\{1,2\}$ provided in the dataset. The definitions of all functions are provided in Appendix, and there are 25 functions in total with similar return type as GQA. We follow NS-VQA to train detectors based on MaskRCNN and detect top 32 bounding boxes ranked by their confidence scores. We use a hidden dimension $D=256$ for both the visual encoder and the meta module.

We report our experimental results on the standard test set in Table 1. We observe that MMN can outperform the standard NMN with more compact representation with meta module and scene-graph-based intermediate supervision. Except for numerical operations like Count and Compare Number, MMN can achieve similar accuracy as the state-of-the-art NS-VQA model. As CLEVR dataset is not the focus of this paper due to its synthetic nature and limited set of semantic functions, we use it only as the preliminary study. Note that we have also attempted to re-implement the NS-VQA approach on GQA, and observe that the accuracy is very low ($\approx$ 30% on test set). This is due to that NS-VQA performs pure symbolic reasoning, thus requiring the scene graph to be accurately generated; while generating scene graphs is challenging in GQA with open-domain real images.

2 GQA Experimental Setup

GQA Dataset contains 22M questions over 140K images. This full “all-split” dataset has unbalanced answer distributions, thus, is further re-sampled into a “balanced-split” with a more balanced answer distribution. The new split consists of 1M questions. Compared with the VQA v2.0 dataset , the questions in GQA are designed to require multi-hop reasoning to test the reasoning skills of developed models. Compared with the CLEVR dataset , GQA greatly increases the complexity of the semantic structure of questions, leading to a more diverse function set. The real-world images in GQA also bring in a bigger challenge in visual understanding. In GQA, around 94% of questions need multi-hop reasoning, and 51% questions are about the relationships between objects. Following , the evaluation metric used in our experiments is accuracy (including binary and open-ended).

The dimensionality of input image features $D_{v}$ is 2048, extracted from the bottom-up-attention model https://github.com/peteanderson80/bottom-up-attention. For each image, we keep the top 48 bounding boxes ranked by confidence score with the positional information of each bounding box in the form of [top-left-x, top-left-y, bottom-right-x, bottom-right-y], normalized by the image width and height. Both the meta module and the visual encoder have a hidden dimension $D$ of 512 with 8 heads. GloVe embeddings are used to encode both questions and function keywords with 300 dimensions. The total vocabulary size is 3761, including all the functions, objects, and attributes. For training, we first use the 22M unbalanced “all-split” to bootstrap our model with a batch size of 2048 for 5 epochs, then fine-tune on the “balanced-split” with a batch size of 256. The testdev-balanced split is used for model selection.

3 GQA Experimental Results

We report our experimental results on the test2019 split (from the public GQA leaderboard) in Table 2. First, we observe significant performance gain from MMN over NMN , which demonstrates the effectiveness of the proposed meta module mechanism. Further, we observe that our model outperforms the VQA state-of-the-art monolithic model MCAN by a large margin, which demonstrates the strong compositionality of our module-based approach. Overall, our single model achieves competitive performance (top 2) among published approaches. Notably, we achieve higher performance than LXMERT , which is pre-trained on large-scale out-of-domain datasets. The performance gap with NSM is debatable since our model is standalone without relying on well-tuned external scene graph generation model .

To verify the contribution of each component in MMN, we perform several ablation studies. (1) w/o Module Supervision vs. w/ Module Supervision. We investigate the influence of module supervision by changing the hyper-parameter $\eta$ from to $2.0$ to see how much influence the module supervision has on the model performance. (2) w/o Bootstrap vs. w/ Bootstrap. We investigate the effectiveness of bootstrapping in training to validate whether we could use the large-scale unbalanced split to benefit on the model’s performance.

We further report the ablation results for the validation split in Table 3. From Ablation (1), we observe that without module supervision, our MMN already achieves decent improvement over 6-layered MCAN . Since all the modules have shared parameters, our model has similar parameter size as 1-layered MCAN. The result demonstrates the efficiency of the parameterization in our MMN. By increasing $\eta$ from 0.1 to 0.5, accuracy steadily improves, which reflects the effectiveness of module supervision. Further increasing the value of $\eta$ did not improve the performance empirically. From Ablation (2), we observe that bootstrapping is a critical step for MMN, as it explores more data to better regularize functionalities of reasoning modules. Bootstrap for 4 epochs can yield better performance in our experiments.

4 Generalization Experimental Results

Similar to Meta Learning , we also evaluate whether our meta module has learned the ability to adapt to unseen sub-tasks. To evaluate such generalization ability, we perform additional experiments, where we held out all the training instances containing verify_shape, relate_name, filter_location, choose_name to quantitatively measure model’s performance on these unseen functions. Standard NMN fails to handle these unseen functions, as it requires training instances for the randomly initialized shallow module network for these unseen functions. In contrast, MMN can generalize the unseen functions from recipe space and exploits the structural similarity with its related functions to infer its semantic functionality. For example, if the training set contains verify_size (function: verify, type: size, attr: ?) and filter_shape (function: filter, type: shape, attr: ?) functions in the recipes, and instantiated module is capable of inferring the functionality of an unseen but similar function verify_shape (function: verify, type:shape, attr: ?) from the recipe embedding space. Table 4 shows that the zero-shot accuracy of the proposed meta module is significantly higher than NMN (equivalent to random guess), which demonstrates the generalizability of proposed MMN architecture. Instead of handcrafting a new module every time when new function appears like NMN , our MMN is more flexible and extensible for handling growing function sets. Such observation further validates the value of the proposed method to adapt a more challenging environment where we need to handle unknown functions.

5 Interpretability and Error Analysis

To demonstrate the interpretability of MMN, Figure 6 provides some visualization results to show the inferential chain during reasoning. As shown, the model correctly executes the intermediate results and yields the correct final answer. More visualization examples are provided in the Appendix. To better interpret the model’s behavior, we also perform quantitative analysis to diagnose the errors in the inferential chain. Here, we held out a small validation set to analyze the execution accuracy of different functions. Our model obtains Recall@1 of 59% and Recall@2 of 73%, which indicates that the object selected by the symbolic teacher has 59% chance of being top-1, and 73% chance as the top-2 by the student model, significantly higher than random-guess Recall@1 of 2%, demonstrating the effectiveness of module supervision.

Furthermore, we conduct a detailed analysis of function-wise execution accuracy to understand the limitation of MMN. We found that most erroneous functions are relate and query, having 44% and 60% execution accuracy respectively. These errors are mainly related to scene understanding, which suggests that the scene graph model is critical to surpassing NSM on performance. However, this is out of the scope of this paper and we plan to leave it for future study.

Conclusion

In this paper, we propose Meta Module Network that resolves the known challenges of NMN. Our model is built upon a meta module, which can be instantiated into an instance module to perform designated functionalities dynamically. Our approach can significantly outperform baseline methods and achieve comparable performance to state of the art while maintaining strong explainability.

References

Appendix A Appendix for “Meta Module Network for Compositional Visual Reasoning”

The multi-head attention network is illustrated in Figure 7.

A.2 Recipe Embedding

The recipe embedder is illustrated in Figure 8.

A.3 Implementation

The implementation of the proposed model is demonstrated in Figure 9. Our model can be efficiently implemented by adding masks on top of the Transformer model, guided by an additional supervision signal.

A.4 Function Statistics

The function statistics is listed in Table 5.

A.5 Inferential Chains

More inferential chains are visualized in Figure 10 and 11.

A.6 Detailed Error Analysis

Furthermore, we conduct a detailed analysis of function-wise execution accuracy to understand the limitation of MMN. Results are shown in Table 6. Below are the observed main bottlenecks: $(i)$ relation-type functions such as relate, relate_inv; and $(ii)$ object/attribute recognition functions such as query_name, query_color. We hypothesize that this might be attributed to the quality of visual features from standard object detection models , which does not capture the relations between objects well. Besides, the object and attribute classification network are not fine-tuned on GQA. This suggests that scene graph modeling for visual scene understanding is critical to surpassing NSM on performance.

A.7 Function Description

The detailed function descriptions for CLEVR and GQA are provided in Figure 12 and Figure 13, respectively.