Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks

Introduction

Increasing interest rises in Vision-Language Navigation (VLN) tasks, where an agent navigates in 3D indoor environments following a natural language instruction, such as Walk between the columns and make a sharp turn right. Walk down the steps and stop on the landing. The agent begins at a random point and goes toward a goal by means of active exploration. A vision image is given at each step and a global step-by-step instruction is provided at the beginning of the trajectory.

Recent research in feature extraction , attention and multi-modal grounding have helped the agent to understand the environment. Previous works in Vision-Language Navigation have focused on improving the ability of perceiving the vision and language inputs and cross-modal matching . With these approaches, an agent is able to perceive the vision-language inputs and encode historical information for navigation.

However, the VLN task remains challenging since rich semantic information contained in the environments is neglected: 1) Past actions affect the actions to be taken in the future. To make a correct action requires the agent to have a thorough understanding of its activity in the past. 2) The agent is not able to explicitly align the trajectory with the instruction. Thus, it is uncertain whether the vision-language encoding can fully represent the current status of the agent. 3) The agent is not able to accurately assess the progress it has made. Even though Ma et al. proposed a progress monitor to estimate the normalized distance toward the goal, labels in this method are biased and noisy. 4) The action space of the agent is implicitly limited since only neighbour nodes in the navigation graph are reachable. Therefore, if the agent gains knowledge of the navigation map and understands the consequence of its next action, the navigation process will be more accurate and efficient.

We introduce auxiliary reasoning tasks to solve these problems. There are three key advantages to this solution. First of all, auxiliary tasks produce additional training signals, which improves the data efficiency in training and makes the model more robust. Secondly, using reasoning tasks to determine the actions makes the actions easier to explain. It is easier to interpret the policy of an agent if we understand why the agent takes a particular action. An explainable mechanism benefits human understanding of how the agent works. Thirdly, the auxiliary tasks have been proven to help reduce the domain gap between seen and unseen environments. It has been demonstrated that self-supervised auxiliary tasks facilitate domain adaptation. Besides, it has been proven that finetuning the agent in an unseen environment effectively reducing the domain gap . We use auxiliary tasks to align the representations in the unseen domain alongside those in the seen domain during finetuning.

In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework facilitates navigation learning. AuxRN consists of four auxiliary reasoning tasks: 1) A trajectory retelling task, which makes the agent explain its previous actions via natural language generation; 2) A progress estimation task, to evaluate the percentage of the trajectory that the model has completed; 3) An angle prediction task, to predict the angle by which the agent will turn next. 4) A cross-modal matching task which allows the agent to align the vision and language encoding. Unlike “proxy tasks” which only consider the cross-modal alignment at one time, our tasks handle the temporal context from history in addition to the input of a single step. The knowledge learning of these four tasks are presumably reciprocal. As shown in Fig. 1, the agent learns to reason about the previous actions and predict future information with the help of auxiliary reasoning tasks.

Our experiment demonstrates that AuxRN dramatically improves the navigation performance on both seen and unseen environments. Each of the auxiliary tasks exploits useful reasoning knowledge respectively to indicate how an agent understands an environment. We adopt Success weighted by Path Length (SPL) as the primary metric for evaluating our model. AuxRN pretrained in seen environments with our auxiliary reasoning tasks outperforms our baseline by 3.45% on validation set. Our final model, finetuned on unseen environments with auxiliary reasoning tasks obtains 65%, 4% higher than the previous state-of-the-art result, thereby becoming the first-ranked result in the VLN Challenge in terms of SPL.

Related Work

Vision-Language Reasoning Bridging vision and language is attracting attention from both the computer vision and the natural language processing communities. Various associated tasks have been proposed, including Visual Question Answering (VQA) , Visual Dialog Answering , Vision-Language Navigation (VLN) and Visual Commonsense Reasoning (VCR) . Vision-Language Reasoning plays an important role in solving these problems. Anderson et al. apply an attention mechanism on detection results to reason visual entities. More recent works, such as LXMERT , ViLBERT , and B2T2 obtain high-level semantics by pretraining a model on a large-scale dataset with vision-language reasoning tasks.

Learning with Auxiliary Tasks Self-supervised auxiliary tasks have been widely applied in the field of machine learning. Moreover, the concept of learning from auxiliary tasks to improve data efficiency and robustness has been extensively investigated in reinforcement learning. Mirowski et al. propose a robot which obtains additional training signals by recovering a depth image with colored image input and predicting whether or not it reaches a new point. Furthermore, self-supervised auxiliary tasks have been widely applied in the fields of computer vision , natural language processing and meta learning . Gidaris et al. unsupervisedly learn image features with a 2D rotate auxiliary loss, while Sun et al. indicate that self-supervised auxiliary tasks are effective in reducing domain shift.

Vision Language Navigation A number of simulated 3D environments have been proposed to study navigation, such as Doom , AI2-THOR and House3D . However, the lack of photorealism and natural language instruction limits the application of these environments. Anderson et al. propose Room-to-Room (R2R) dataset, the first Vision-Language Navigation (VLN) benchmark based on real imagery .

The Vision-Language Navigation task has attracted widespread attention since it is both widely applicable and challenging. Earlier work combined model-free and model-based reinforcement learning to solve VLN. Fried et al. propose a speaker-follower framework for data augmentation and reasoning in supervised learning. In addition, a concept named “panoramic action space” is proposed to facilitate optimization. Later work has found it is beneficial to combine imitation learning and reinforcement learning . The self-monitoring method is proposed to estimate progress made towards the goal. Researchers have identified the existence of the domain gap between training and testing data. Unsupervised pre-exploration and Environmental dropout are proposed to improve the ability of generalization.

Method

The Vision-and-Language Navigation (VLN) task gives a global natural sentence $I=\{w_{0},...,w_{l}\}$ as an instruction, where each $w_{i}$ is a token while the $l$ is the length of the sentence. The instruction consists of step-by-step guidance toward the goal. At step $t$, the agent observes a panoramic view $O_{t}=\{o_{t,i}\}_{i=1}^{36}$ as the vision input. The panoramic view is divided into 36 RGB image views, while each of these views consists of image feature $v_{i}$ and an orientation description ($sin\ \theta_{t,i}$, $cos\ \theta_{t,i}$, $sin\ \phi_{t,i}$, $cos\ \phi_{t,i}$). For each step, the agent chooses a direction to navigate over all candidates in the panoramic action space . Candidates in the panoramic action space consist of $k$ neighbours of the current node in the navigation graph and a stop action. Candidates for the current step are defined as $\{c_{t,1},...,c_{t,k+1}\}$, where $c_{t,k+1}$ stands for the stop action. Note that for each step, the number of neighbours $k$ is not fixed.

2 Vision-Language Forward

We first define the attention module, which is widely applied in our pipeline. Then we illustrate vision embedding and vision-language embedding mechanisms. At last, we demonstrate the approach of action prediction.

Attention Module At first we define the attention module, an important part of our pipeline. Suppose we have a sequence of feature vectors noted as $\{f_{0},...,f_{n}\}$ to fuse and a query vector $q$. We implement an attention layer $\widehat{f}=\textnormal{Attn}(\{f_{0},...,f_{n}\},q)$ as:

$W_{Attn}$ represents the fully connected layer of the attention mechanism. $\alpha_{i}$ is the weight for the $i$th feature for fusing.

Vision Embedding As mentioned above, the panoramic observation $O_{t}$ denotes the 36 features consisting of vision and orientation information. We then fuse $\{o_{t,1},...,o_{t,36}\}$ with cross-modal context of the last step $\widehat{f}_{t-1}$ and introduce an LSTM to maintain a vision history context $\widetilde{f}^{o}_{t}$ for each step:

where $\widetilde{f}_{t}^{o}=h_{t}$ is the output of the $\textnormal{LSTM}_{v}$. Note that unlike the other two LSTM layers in our pipeline (as shown in Fig. 2) which are computed within a step. $\textnormal{LSTM}_{v}$ is computed over a whole trajectory.

Vision-Language Embedding Similar to , we embed each word token $w_{i}$ to word feature $f^{w}_{i}$, where $i$ stands for the index. Then we encode the feature sequence by a Bi-LSTM layer to produce language features and a global language context $\overline{f}^{w}$:

The global language context participates $\overline{f}^{w}$ the auxiliary task learning descripted in Sec. 3.4. Finally, we fuse the language features $\{\widetilde{f}^{w}_{0},...,\widetilde{f}^{w}_{l}\}$ with the vision history context $\widetilde{f}^{o}_{t}$ to produce the cross-modal context $\widehat{f}_{t}$:

Action Prediction In the VLN setting, the adjacent navigable node is visible. Thus, we can obtain the reachable candidates $C=\{c_{t,1},...,c_{t,k+1}\}$ from the navigation graph. Similar to observation $O$, candidates in $C$ are concatenated features of vision features and orientation descriptions. We obtain the probability function $p_{t}(a_{t})$ for action $a_{t}$ by:

Three ways for action prediction are applied to different scenarios: 1) imitation learning: following the labeled teacher action $a_{t}^{*}$ regardless of $p_{t}$; 2) reinforcement learning: sample action following the probability distribution $a_{t}\sim p_{t}(a_{t})$; 3) testing: choose the candidate which has the greatest probability $a_{t}=\textnormal{argmax}(p_{t}(a_{t}))$.

3 Objectives for Navigation

In this section, we introduce two learning objectives for the navigation task: imitation learning (IL) and reinforcement learning (RL). The navigation task is jointly optimized by these two objectives.

Imitation Learning forces the agent to mimic the behavior of its teacher. IL has been proven to achieve good performance in VLN tasks. Our agent learns from the teacher action $a_{t}^{*}$ for each step:

where $a_{t}^{*}$ is a one-hot vector indicating the teacher choice.

Reinforcement Learning is introduced for generalization since adopting IL alone could result in overfitting. We implement the A2C algorithm, the parallel version of A3C , and our loss function is calculated as:

$A_{t}$ is a scalar representing the advantage defined in A3C.

Joint Optimization Firstly, the model samples trajectory by teacher forcing approach and calculates gradients with imitation learning. Secondly, the model samples trajectory under the same instruction by student forcing approach and calculates gradients with reinforcement learning. Finally, we add the gradients together and use the added gradients to update the model.

4 Auxiliary Reasoning Learning

The vision-language navigation task remains challenging, since the rich semantics contained in the environments are neglected. In this section, we introduce auxiliary reasoning learning to exploit additional training signals from environments.

In Sec. 3.2, we obtain the vision context $\widetilde{f}^{o}_{t}$ from Eq. 2, the global language context $\overline{f}^{w}$ from Eq. 3 and the cross-modal context $\widehat{f}_{t}$ from Eq. 4. In addition to action prediction, we give the contexts to the reasoning modules in Fig. 2 to perform auxiliary tasks. We discuss four auxiliary objectives use the contexts for reasoning below.

Trajectory Retelling Task Trajectory reasoning is critical for an agent to decide what to do next. Previous works train a speaker to translate a trajectory to a language instruction. The methods are not end-to-end optimized, which limit the performances.

As shown in Fig. 2, we adopt a teacher forcing method to train an end-to-end speaker. The teacher is defined as $\{f^{w}_{0},...,f^{w}_{l}\}$, the same word embeddings as in Eq. 4. We use $\textnormal{LSTM}_{s}$ to encode these word embeddings. We then introduce a cycle reconstruction objective named trajectory retelling task:

Our trajectory retelling objective is jointly optimized with the main task. It helps the agent to obtain better feature representations since the agent comes to know the semantic meanings of the actions. Moreover, trajectory retelling makes the activity of the agent explainable. Since the model could deviate a lot in student forcing, we does not train the trajectory retelling task in RL scenarios.

Progress Estimation Task We propose a progress estimation task to learn the navigation progress. Earlier research uses normalized distances as labels and optimizes the prediction module with Mean Square Error (MSE) loss. However, we use the percentage of steps $r_{t}$, noted as a soft label $\{\frac{t}{T},1-\frac{t}{T}\}$ to represent the progress:

Here $W_{r}$ is the weight of the fully connected layer and $\sigma$ is the sigmoid activation layer. Our ablation study reveals that the method that learning from percentage of steps $r_{t}$ with BCE loss achieves higher performance than previous method. Normalized distance labels introduce noise, which limits performance. Moreover, we also find that Binary Cross Entropy (BCE) loss performs better than MSE loss with our step-percentage label since logits learned from BCE loss are unbiased. The progress estimation task requires the agent to align the current view with corresponding words in the instruction. Thus, it is beneficial to vision language grounding.

Cross-modal Matching Task We propose a binary classification task, motivated by LXMERT , to predict whether or not the trajectory matches the instruction. We shuffle $\overline{f}^{w}$ from Eq. 3 with feature vector in the same batch with the probability of $0.5$. The shuffled operation is marked as “S” in the white circle in Fig. 2 and the shuffled feature is noted as $\overline{f{}^{\prime}}^{w}$. We concatenate the shuffled feature with the attended vision-language feature $\widehat{f}_{t}$. We then supervise the prediction result with $m_{t}$, a binary label indicating whether the feature has been shuffled or remains unchanged.

where $W_{m}$ stands for the fully connected layer. This task requires the agent to align historical vision-language features in order to distinguish if the overall trajectory matches the instruction. Therefore, it facilitates the agent to encode historical vision and language features.

Angle Prediction Task The agent make the choice among the candidates to decide which step it will take next. Compared with the noisy vision feature, the orientation is much cleaner. Thus we consider learning from orientation information in addition to learning from candidate classification. We thus propose a simple regression task to predict the orientation that the agent will turn to:

where $a_{t}$ is the angle of the teacher action in the imitation learning, while $W_{a}$ stands for the fully connected layer. Since this objective requires a teacher angle for supervision, we do not forward this objective in RL.

Above all, we jointly train all the four auxiliary reasoning tasks in an end-to-end manner:

Experiment

Dataset and Environments We evaluate the proposed AuxRN method on the Room-to-Room (R2R) dataset based on Matterport3D simulator . The dataset, comprising 90 different housing environments, is split into a training set, a seen validation set, an unseen validation set and a test set. The training set consists of 61 environments and 14,025 instructions, while the seen validation set has 1,020 instructions using the save environments with the training set. The unseen validation set consists of another 11 environments with 2,349 instructions, while the test set consists of the remaining 18 environments with 4,173 instructions.

Evaluation Metrics A large number of metrics are used to evaluate models in VLN, such as Trajectory Length (TL), the trajectory length in meters, Navigation Error (NE), the navigation error in meters, Oracle Success Rate (OR), the rate if the agent successfully stops at the closest point, Success Rate (SR), the success rate of reaching the goal, and Success rate weighted by (normalized inverse) Path Length (SPL) . In our experiment, we take all of these into consideration and regard SPL as the primary metric.

Implementation Details We introduce self-supervised data to augment our dataset. We sample the augmented data from training and testing environments and use the speaker trained in Sec. 3.2 to generate self-supervised instructions.

Our training process consists of three steps: 1) we pretrain our model on the training set; 2) we pick the best model (the model with the highest SPL) at step 1 and finetune the model on the augmented data sampled from training set ; 3) we finetune the best model at step 2 on the augmented data sampled from testing environments for pre-exploration, which is similar to . We pick the last model at step 3 to test. The training iterations for each steps are 80K. We train each model with auxiliary tasks and set all auxiliary loss weight to 1. At steps 2 and 3, since augmented data contains more noise than labeled training data, we reduce the loss weights for all auxiliary tasks by half.

2 Test Set Results

In this section, we compare our model with previous state-of-the-art methods. We compare the proposed AuxRN with two baselines and five other methods. A brief description of previous models as followed. 1) Random: randomly take actions for 5 steps. 2) Seq-to-Seq: A sequence to sequence model reported in . 3) Look Before You Leap: a method combining model-free and model-based reinforcement learning. 4) Speaker-Follower: a method introduces a data augmentation approach and panoramic action space. 5) Self-Monitoring: a method regularized by a self-monitoring agent. 6) The Regretful Agent: a method based on learnable heuristic search 7) FAST: a search based method enables backtracking 8) Reinforced Cross-Modal: a method with cross-modal attention and combining imitation learning with reinforcement learning. 9) ALTR: a method focus on adapting vision and language representations 10) Environmental Dropout: a method augment data with environmental dropout. Additionally, we evaluate our models on three different training settings: 1) Single Run: without seeing the unseen environments and 2) Pre-explore: finetuning a model in the unseen environments with self-supervised approach. 3) Beam Search: predicting the trajectories with the highest rate to success.

As shown in Tab. 1, AuxRN outperforms previous models in a large margin on all three settings. In Single Run, we achieve 3% improvement on oracle success, 4% improvement on success rate and 4% improvement on SPL. In Pre-explore setting, our model greatly reduces the error to 3.69, which shows that AuxRN navigates further toward the goal. AuxRN significantly boost oracle success by 5%, success rate 4% and SPL to 4%. AuxRN achieves similiar improvements on other two domains, which indicates that the auxiliary reasoning tasks is immune from domain gap.

We also achieve the state-of-the-art in Beam Search setup. Our final model with Beam Search algorithm achieves 71% success rate, which is 2% higher than Environmental Dropout, the previous state-of-the-art.

3 Ablation Experiment

Auxiliary Reasoning Tasks Comparison In this section, we compare performances between different auxiliary reasoning tasks. We use the previous state-of-the-art as our baseline. We train the models with each single task based on our baseline. We evaluate our models on both the seen and unseen validation set and the results are shown in Tab. 2. It turns out that each task promotes the performance based on our baseline independently. And training all tasks together is able to further boost the performance, achieving improvements by 3.02% on the seen validation set and by 2.78% on the unseen validation set. It indicates that the auxiliary reasoning tasks are presumably reciprocal.

Moreover, our experiments show that our auxiliary losses and back-translation method has a mutual promotion effect. On the seen validation set, baseline with back-tranlation gets 5.50% improvement while combining back-translation promotes SPL by 11.30%, greater than the sum of the performance improvement of baseline with auxiliary losses and with back-translation independently. Similar results have been observed on the unseen validation set. Baseline with back-translation gets 3.95% promotion while combining back-translation boosts SPL by 7.40%.

The result of ablation study for Trajectory Retelling Task is shown as Tab. 3. Firstly, teacher forcing outperforms Matching Critic by 1.8% and 4.22% respectively. Teacher forcing performs 7.07% and 6.76% more than Matching Critic in terms of accuracy. Secondly, teacher forcing outperforms student forcing by 1.46% and 2.04% in terms of SPL in two validation sets. The results also indicate that teacher forcing is better in sentence prediction compared with student forcing. Thirdly, in terms of SPL, standard teacher forcing outperforms the teacher forcing with shared context on the unseen validation set by 0.77%. Besides, we notice that the teacher forcing with shared context outperforms standard teacher forcing about 12% in word prediction accuracy (Acc). We infer that the teacher forcing with shared context overfits on the trajectory retelling task.

Progress Estimation Task To valid the progress estimation task, we investigation two variants in addition to our standard progress estimator. 1) Progress Monitor: We implement Progress Monitor based on our baseline method. 2) we train our model use Mean Square Error (MSE) rather than BCE Loss with the same step-wise label $\frac{t}{T}$. We compare these models with four metris: OR, SR, Error and SPL. The Error is calculated by the mean absolute error between the progress estimation prediction and the label.

The result is shown as Tab. 4. Our standard model outperforms other two variants and the baseline on most of the metrics. Our Step-wise MSE model performs 2.62% higher on the seen validation set 2.53% higher on the unseen validation set than Progress Monitor , indicating that label measured by normalized distances is noisier than label measured by steps. In addition, we find that the Progress Monitor we implement performs even worse than baseline. When the agent begins to deviate from the labeled path, the progress label become even noisier.

We compare different loss functions with step-wise labels. Our model with BCE loss is 6.34% higher on the seen validation set and 1.58% higher on the unseen validation set. Furthermore, the prediction error of the model trained by MSE loss is higher than which trained by BCE loss. The Error of the Step-wise+MSE model is 0.14 higher on the seen validation set and 0.16 higher on the unseen validation set than Step-wise+BCE model.

4 Visualization

Regularized Language Attention We visualize the attention map for $\textnormal{Attn}_{w}$ after $\textnormal{Bi-LSTM}_{w}$. The dark region in the map stands for where the language features receive high attention. We observe from Fig. 4 that the attention regions on both two maps go left while the navigation step is increasing (marked as 1). It means that both models learns to pay an increasing attention to the latter words. At the last few steps, our model learns to focus on the first feature and the last feature (marked as 2 and 3), since the Bi-LSTM encodes sentence information at the first and the last feature. We infer from our experiments that auxiliary reasoning losses help regularize the language attention map, which turns out to be beneficial.

Navigation Visualization We visualize two sample trajectories to show the process of navigation. To further demonstrate how AuxRN understand the environment, we show the result of the progress estimator and matching function. The estimated progress continues growing during navigation while the matching result is increasing exponentially. When AuxRN reaches the goal, the progress and matching results jump to almost 1. It turns out that our agent precisely estimating the current progress and the instruction trajectory consistency.

Conclusion

In this paper, we presented a novel framework, auxiliary Reasoning Navigation (AuxRN), that facilitates navigation learning with four auxiliary reasoning tasks. Our experiments confirm that AuxRN improves the performance of the VLN task quantitatively and qualitatively. We plan to build a general framework for auxiliary reasoning tasks to exploit the common sense information in the future.

Acknowledgement

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant No.U19A2073 and in part by the National Natural Science Foundation of China (NSFC) under Grant No.61976233, and by the Air Force Research Laboratory and DARPA under agreement number FA8750-19-2-0501, Australian Research Council Discovery Early Career Researcher Award (DE190100626).

References