Audio-Visual Floorplan Reconstruction

Introduction

Floorplans of complex 3D environments—such as homes, offices, shops, churches—are a compact ground-plane representation of their overall layout, showing the different rooms and their connectivity. Floorplans are useful for visualizing a large space, navigating an unfamiliar building, planning safety routes, and communicating architectural designs. In robotics, an agent entering a new building needs to quickly sense the overall layout, but without visiting every part of it.

Traditionally a floorplan is created by distilling a fully observed 3D environment into its footprint—whether manually or with the aid of 3D sensors . Recent research aims to infer room layouts using imagery and/or scans, with impressive results . However, existing methods are limited to mapping the regions they directly observe. They either require a dense walk-through for the camera to capture most of the space—wasteful if not impossible for a robotic agent trying to immediately perform tasks in a new environment—or else they simply fail to map rooms beyond those where the camera was placed.

Our idea is to “see" beyond the visible regions by listening. Audio provides strong spatial and semantic signals that complement the mapping capabilities of visual sensing. In particular, the value of audio for floorplan estimation is threefold. First, observed sound is inherently driven by geometry; audio reflections bounce off major surfaces and reveal the shape of a room, beyond the camera’s field of view. Second, sounds heard from afar—even multiple rooms away—can suggest the existence of distant freespace where the sounding object could exist (e.g., a dog barking in another room). Third, hearing semantically meaningful sounds from different directions naturally reveals the plausible room layouts based on the activities or objects those sounds represent. For example, a shower running suggests the direction of the bathroom, even before we see it; microwave beeps suggest a kitchen; climbing footsteps suggest a staircase. See Figure 1.

To this end, we propose a new research direction: audio-visual floorplan reconstruction. Given a short RGB video complete with multi-channel audio, the goal is to produce a 2D floorplan that shows the freespace and occupied regions and divides them into a discrete set of semantic room labels (family room, kitchen, etc.). Importantly, the floorplan output extends significantly beyond the area directly observable in the video frames. This efficiency is critical for navigating robots that need to act without exhaustively touring a space, as well as offline scenarios where a user wants to extract a broad map from an existing short video.

Our AV-Map approach works as follows. We devise a deep convolutional neural network architecture that leverages sequences of audio and visual data to reason about the structure and semantics of the floorplan. Our model independently extracts floorplan-aligned features for audio and RGB data, encodes sequences of features of each modality using self-attention mechanisms, and finally fuses information from audio and RGB using a decoder architecture.

We consider two settings: device-generated sounds (active) and environment-generated sounds (passive). In the active setting, the camera emits a known sound while it moves. This corresponds to a use case where a person or robot does a swift walk-through of an environment while their phone/camera emits some sound. In the passive setting, we observe only naturally occurring sounds made by objects and people in the building. This corresponds to a use case where we are simply given a passively recorded video, likely captured for some other purpose.

To our knowledge, ours is the first attempt to infer floorplans from audio-visual data. Our results on 85 large real-world, multi-room environments show that AV-Map not only consistently outperforms traditional vision-based mapping, but also improves the state-of-the-art approach for extrapolating occupancy maps beyond visible regions (with a relative gain of 8% in floorplan accuracy). Though observing only a small fraction of the full homes, our model yields good interior maps covering much of their area. We also show audio and vision are synergistic signals to classify room types, allowing high-level perception of the semantics of the space even before directly visiting each room.

Related Work

The vision and graphics communities have explored various ways to use visual data, depth sensors, and laser scanners to build floorplans. Geometric approaches use 3D point cloud inputs to construct building-wide floor plans . Given RGB-D scans, FloorNet and Floor-SP estimate 2D floorplans and rooms’ semantic labels using a mix of deep learning and optimization. Given monocular RGB images or $360^{\circ}$ panoramas , other methods estimate a 3D indoor Manhattan room layout. Using only a small portion of a $360^{\circ}$ panorama, models can be trained to infer missing viewpoints and/or semantic labels . Unlike any of the above, our approach leverages both audio and visual sensing to infer a 2D floorplan map and its semantic room labels. As our results show, audio offers the advantage of sensing further beyond the field of view of visual sensors.

Mapping for navigation

With adequate overlapping views, structure-from-motion methods can recover the 3D structure of an environment (e.g., ). Laser-based 2D SLAM is often used in mobile robotics to obtain the ground plane map . Recent work leverages scans of indoor environments and fast simulation tools to facilitate work on embodied visual navigation . While often the map is implicitly learned, some methods explicitly estimate a 2D occupancy map, projecting the observed point cloud to the ground plane and growing the map over time . To navigate to a specified room, the method of predicts a 2D semantic map with room labels, learning the layout patterns in houses. In contrast to navigation, where an intelligent agent controls the camera and builds its map in service of reaching a target, our goal is to transform a passive video sequence (with audio) into a map. We show the advantages of our audio-visual approach over OccAnt , the state-of-the-art navigation method that extrapolates beyond visible scene points using vision alone.

Audio for spatial sensing

Prior work explores ways to exploit audio alone to sense the shape of a room or object. Given multiple microphone recordings of a known sound, the method of computes the shape of a single convex polyhedral room, while sound reflections are used to sense the 3D shape of an object hidden around a corner . In robotics, echolocation can detect distances to surfaces based on the reflections . Multi-channel audio is also used to track dynamic objects . Unlike any of these methods, our approach takes a video (both the audio and visual streams) as input and produces a floorplan as output. Furthermore, our model is not restricted to known microphone layouts or known emitted sounds; rather, it can learn from natural sounds sensed passively in the environment (e.g., running water, door shutting). While environment semantics are explored in acoustic scene analysis , our problem is quite different: the target output is a geometric map, not a label for the acoustic event that occurred.

Audio-visual spatial sensing

Audio and vision together offer powerful cues for spatial perception. At the object level, they reveal shape and material properties , e.g., via the sound of one object striking another. At the environment level, audio can help sense 3D surfaces in cases where depth sensing would fail, e.g., transparent, shiny, or textureless surfaces , or provide self-supervisory cues for imagery . Recent work leverages audio-visual sensing to address navigation tasks, learning to move efficiently to a sounding target . In , a curiosity-driven framework is proposed that leverages audio-visual cues to explore environments. None of the above methods produce audio-visual floorplans. Furthermore, an insight unique to our work is the use of naturally occurring semantic sounds to understand a multi-room layout.

Approach

Our goal is to estimate the 2D layout of an environment depicted in a short video. The 2D layout has two components: the structure of the interior area, and the semantic labels (room types) associated with each region of the interior. First, we formally describe the problem (Sec 3.1). We then describe our proposed model AV-Map (Sec 3.2) and introduce our training and inference procedure (Sec 3.3).

We consider videos generated by a camera and an ambisonic microphone following short trajectories through various home environments. An ambisonic mic captures omni-directional multi-channel audio . We represent a video by $V=\{(v_{1},a_{1}),(v_{2},a_{2})...\}$ where $v_{t}$ is the RGB frame and $a_{t}$ is the audio clip sampled at time step $t$. Additionally, let us denote by $P_{V}=\{\textbf{0},{r_{2}},{r_{3}},...\}$ the position of the camera and microphone relative to the first time step in the coordinate system of the floorplan, where $r_{i}=(x_{i},y_{i},\theta_{i})$ represents the movement along the x- and y-axis on the 2D ground plane and $\theta_{i}$ represents the rotation about the gravity axis. Relative pose changes in a video can be estimated using computer vision ; for simplicity we assume correct relative camera poses are available for all methods. However, we find our proposed model is robust to noise in pose estimation within the range of odometry noise models considered in the literature , owing to the resolution of the output floorplan.

Each floorplan is parameterized as two variables: $M_{\text{int}}$ and $M_{\text{room}}$, which represent the structure and semantics, respectively. The interior map $M_{\text{int}}$ is a 2D binary grid that is a top-down view of the environment and represents the existence of floor, objects, furniture by label $1$, and walls and areas outside the environment by label . The room map $M_{\text{room}}$ is a 2D grid taking $N_{r}+1$ possible values with labels $\{1,\ldots,N_{r}\}$ representing the $N_{r}$ room types (kitchen, bathroom, etc.) and representing walls and areas outside the environment. Each cell in the floorplan (an entry in the matrix $M$) represents a 25cm2 area.

The goal of our work is to learn a mapping $\mathcal{F}$ that estimates the floorplan (both $M_{\text{int}}$ and $M_{\text{room}}$) of an environment using the video $V$ and the relative pose changes $P_{V}$. The visual information in $v_{t}$ captures the geometric properties and room types of visible regions. The audio information captured in $a_{t}$ is either actively emitted by the camera, or else passively generated by objects and people in the environment (details below). Since the placement of objects is highly correlated with room types (for example, showers are in bathrooms and dishwashers are in kitchens), the audio signal captures a strong semantic signal indicating the room types. Furthermore, the echoes propagating through the environment capture geometric properties of the walls and other major surfaces. Our key insight is that the audio observations will illuminate the map for regions beyond what is visible in the frames of a short video.

2 AV-Map Floorplan Estimation Model

We now present our AV-Map floorplan estimation model $\mathcal{F}$. Fig. 2 overviews our proposed model, consisting of three components: Top-Down Feature Extraction, Feature Alignment, and a Sequence Encoder-Decoder architecture. At each time step, AV-Map estimates the interior map ($M_{\text{int}}$) and semantic room labels ($M_{\text{room}}$) in a neighborhood centered around the camera, integrating estimates over time.

The first stage of our proposed model involves extracting features for a given video $V$. The purpose of the feature extraction step is to project egocentric visual frames $v_{t}$ and ambisonic audio clips $a_{t}$ to a 2D feature grid that is spatially aligned with the top-down floorplan that we wish to estimate at each time step.

Feature Alignment

Sequence Encoder-Decoder

We now wish to encode features for each time step that account for information present in the entire sequence. For example, the appearance of a wall in the second frame should inform the features in the first step and vice-versa. Self-attention operations have shown to be useful to encode such bi-directional relationships. Inspired by this, we design a sequence of two self-attention and convolution operations (as shown in Fig. 2) which we refer to as $\mathtt{ConvSelfAttention}$. The self-attention operations are responsible for communication across time steps at each pixel location. We use convolutions with stride 2 to also simultaneously downsample the features. We denote the resulting features for the RGB frames as:

So far, we have processed the visual and audio information independently. In order to take full advantage of the presence of both modalities, we allow cross-modal information transfer. We accommodate this in the final decoding stage of by concatenating the corresponding intermediate visual and audio features. For the decoder, we follow an architecture similar to the encoder by replacing the convolutions with transposed convolutions to upsample the features. We refer to these layers as $\mathtt{UpConvSelfAttention}$. More concretely, the decoder consists of three $\mathtt{UpConvSelfAttention}$ layers which are used to compute the output as:

In practice, for training, we fix the length of sequences $t_{V}=4$, which balances memory constraints with learning to integrate over time. For illustration, Fig. 2 depicts an instance of the model with $t_{V}=3$.

In summary, the proposed AV-Map floorplan estimation model processes audio-visual sequences at various levels. The feature extraction independently processes each time step. The top-down alignment brings the features to a common coordinate frame. The encoders process sequences of each modality independently while integrating information across time, and finally the decoder fuses information from both visual and audio modalities.

3 Training and Inference

The output of AV-Map is a 2D map with $N_{r}+1$ channels. The model is trained to predict two floorplan maps: the interior structure and the pixel-wise room labels.

Prediction of interior maps is a pixel-wise binary classification problem where s represent the walls or exterior points and $1$s represent the points inside the environment (floors, furniture, objects, etc.). From $s_{t}$, the pixel-wise binary classification probability is computed using the sigmoid function: p_{t_{\text{int}}}[i,j]=\frac{1}{1+\exp{~{}\-s_{t}\big{[}0,i,j~{}\big{]}}} for each pixel location $i,j$ in the 2D grid.

Predicting Room Floorplans

Prediction of room floorplans is similar to prediction of interior maps, but requires multi-class classification of each pixel into one of $N_{r}$ semantic room types. Therefore, the class-wise probabilities at each pixel are computed using the softmax function. Concretely, the classification probability for class $n\in\{1,2,3,...,N_{r}\}$ at pixel location $(i,j)$ is computed as: p_{t_{\text{room}}}[n-1,i,j]=\frac{\exp{~{}s_{t}\big{[}n,i,j\big{]}~{}}}{\sum_{k=1}^{N_{r}}\exp{~{}s_{t}\big{[}k,i,j\big{]}~{}}}.

Training Objectives

For each time step $t$, let the ground truth interior and room maps of the $H\times W$ area around the camera be represented by $y_{t_{\text{int}}}\in\{0,1\}^{H\times W}$ and $y_{t_{\text{room}}}\in\{0,1,\ldots N_{r}\}^{H\times W}$. Since our model’s predictions are aligned with time step $t=1$, we similarly align the ground truth maps to obtain $\hat{y}_{t_{\text{int}}}\{0,1\}^{H^{\prime}\times W^{\prime}}$ and $\hat{y}_{t_{\text{room}}}\in\{0,1,\ldots N_{r}\}^{H^{\prime}\times W^{\prime}}$ by padding with zeros, translating and rotating by $r_{t}$ (where $H^{\prime},W^{\prime}$ are the increased dimensions due to padding). The interior and room map classification objectives for each time step $t$ for pixel location $(i,j)$ are then defined as:

During inference, we estimate the interior and room maps for the whole sequence. As explained above, this is done by max-pooling the predictions $s_{t}$ to produce a sequence-level prediction $S$. Importantly, the self-attention layers in our proposed model ensure that entire sequences are used to reason about each time step. Furthermore, since self-attention layers can process sequences of arbitrary length, we can apply the trained model on videos of varying length.

In order to predict the binary interior map, we simply threshold at $p=0.5$ the final pixel-wise interior probabilities. To obtain the room map prediction, we assign the most likely room label to each location and use the thresholded interior map prediction as a binary mask to get its shape.

4 Video Sequence Generation

In order to generate videos in a variety of 3D environments for which we know ground truth floorplans, we use the Matterport3D dataset The Matterport3D license is available at http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf. and the SoundSpaces audio simulations. SoundSpaces provides highly realistic audio for 85 fully scanned real environments split 59/11/15 for train/val/test, respectively. Most environments are large multi-room homes and contain a variety of furnishings. SoundScapes provides precomputed impulse responses (IR) for all source-receiver locations on a dense grid sampled at 1m spatial resolution. The simulations use SoTA multi-band ray tracing, computing the IRs from arbitrary geometries and frequency-dependent acoustic material properties, and modeling both transmission (including through walls) and scattering. The IRs can be convolved with any audio clip to generate realistic audio for any chosen source-receiver location, including multiple simultaneous sources. See for details of the simulations and Supp. videos for examples.

Generating Floorplans We use the Habitat-API to generate top-down interior floorplans for each environment by projecting the point cloud to the 2D ground plane. Room floorplans are constructed using the Matterport3D room annotations by assigning a room label to each pixel of the interior floorplan. We use the $N_{r}$=13 most frequent room labels from Matterport3D (laundry, kitchen, bathroom, etc.).

Camera Trajectories We generate videos by recording egocentric frames and ambisonic audio along short camera trajectories. Due to the grid constraint of the SoundSpaces data, we restrict camera positions to the same 1m grid. At each location, the camera is parallel to the ground plane and can have a rotation around the gravity-axis in the set $\{0^{\circ},30^{\circ},60^{\circ},...,330^{\circ}\}$. At each step, while the RGB frame is constant, we record audio for 3 seconds.

During training, the trajectories are randomly sampled. We train the models with fixed trajectory lengths of $t_{V}=4$ steps due to GPU memory constraints. In Supp. we provide an ablation with $t_{V}=1$ to demonstrate the power of learning across the sequence. For evaluation on the validation and test environments, we sample trajectories of variable length $t_{V}$, for $t_{V}\in\{1,2,4,8,16\}$.

Audio We consider two settings of audio: device-generated (active) and environment-generated (passive). For the device-generated (Dev. Gen.) setting, the video recording device (e.g., cell phone, AR headset, or robot) also emits a fixed recurring sound at each time step. We use a 3 sec frequency sweep chirp signal in the audible range (20Hz-20KHz). Though any emitted sound could provide useful echoes, the wide range of frequencies activated in the sweep is expected to provide a particularly rich learning signal .

In the environment-generated setting, rather than emit a sound, the system listens for naturally occurring sounds in homes. To achieve this, we first collect a set of 56/32/32 train/val/test audio clipsDownloaded from freesound.org of duration 3 sec that capture sounds made by objects in different room types (for example, sound of a flush, dishwasher, etc.). This allows us to place source sounds in the Matterport3D environments in the appropriate rooms. For each trajectory, the location of sound source(s) is randomly chosen, and the waveform played is dependent on the room type of that location.

We consider three passive settings. In the first setting (referred to as Env. Telephone), the source is near (within 40m2 area) one of the steps in the trajectory and plays the “telephone ring” sound. In the second (Env. Nearby) there is again a single sound source near the trajectory, but the audio clip varies according to the room type of the sampled source location. In the third (Env. All Room), a source is randomly placed in each room and all sources simultaneously play a sound associated with their room type.

Results

Through extensive qualitative and quantitative results, we demonstrate that our proposed model can effectively leverage both audio and visual signals to reason about the extent of the interior of environments (Sec 4.1) and classify regions of the interior into the associated rooms (Sec 4.2).

In order to conduct a thorough analysis of our proposed model, we consider several baselines.

Interior-only: simple baseline that predicts interior pixels (1s) everywhere in the considered neighborhood.

Projected Depth: standard occupancy map computed by projecting depth maps to the ground plane . Note that our model does not leverage depth, only RGB and audio.

OccAnt : The SoTA Occupancy Anticipation model infers a interior map (at each time step) for the 9m2 area in front of the camera from RGB-D by learning to extrapolate beyond the visible ground-plane projections. It is a key baseline to test our claim that audio can better “see" beyond the visual observations. We use the authors’ code.

Acoustic Echoes : This method assumes that all room shapes are convex polyhedra and estimates room shape by listening to audio echoes. However, this approach requires knowing the ground-truth impulse responses at each microphone location, which our method does not have access to. While this method’s setup is artificial, we use it as an upper bound for what an existing audio-only method could provide.

Ours audio-only and RGB-only: As ablations of our model, we train variants with either modality removed.

Note that existing models like FloorNet are not applicable, since they require fully scanned point clouds as input. In our setting, the input is simply a short sequence of egocentric RGB views and audio.

1 Floorplan Interior Reconstruction

First we present interior floorplan results. We set $H\times W$ such that it covers 40m2 area at each time step (see Supp. for similar results with 164m2). Since we aggregate the predictions from all time steps, the final accumulated area varies with the number of steps and direction of movement, adding at most 6.25m2 at each step, for final output areas ranging from 40m2 (1 step) to 134m2 (16 steps).

Evaluation Metrics We use three metrics: Average Precision (AP), Accuracy (Acc.), and Edge Average Precision (Edge AP). AP and Acc compare $S[0,:,:]$ and the binary ground truth map. Edge AP compares the edges of the predicted and ground truth maps in order to emphasize differences in boundary shapes. Pixels are reweighted in all metrics to balance the contribution of labels and $1$.

Comparison to Baselines Table 1 presents our central result, a quantitative evaluation of all the baseline models on test trajectories of length $4$ steps in unseen environments. Our proposed AV-Map model using the device-generated audio outperforms all the baselines on all three metrics. Furthermore, our full AV model outperforms the audio-only and RGB-only variants by good margins. This shows our model successfully performs joint inference by leveraging important cues from both modalities. Our model with RGB-only is itself stronger than the baselines from existing literature , showing the strength of our proposed framework even without the advantage of audio. Fig. 3 shows example map predictions compared to , the best existing method. They highlight how audio allows “seeing" both behind the camera as well as inferring freespace behind walls in large multi-room homes. Fig. 4 compares examples from AV-Map and its RGB-only variant.

Comparison of Audio Settings: Table 2 compares our AV-Map model in the three audio settings described in Sec 3.4. Our model performs slightly better when operated in the device-generated audio setting. This is expected since the frequency sweep audio allows us to capture all the frequencies in the audible range, unlike the naturally occurring sounds in the environment-generated settings. Furthermore, the relative location of the source is known in the device-generated setting (since it is always at the camera). In every setting, AV-Map outperforms the RGB-only and Audio-only ablations. Despite the challenges in Env. All Room (multiple simultaneous sounds from different source locations), we observe minimal decline in our interior map performance.

Effect of Trajectory Length: Figure 5 shows our full-house accuracy gains as a function of trajectory length.Absolute numbers are lower than in Tab. 1 because the scored area here is the entire house’s area, vs. the maximum output map area in Tab. 1. Our model outperforms the RGB-only ablation consistently across varying trajectory lengths. Importantly, our gains are largest when the video is shorter, when less area is visible. This again confirms the power of audio to “see" beyond the images. As the length increases, the visible fraction of the environment becomes larger, diminishing the impact of audio signals (dotted blue line).

2 Floorplan Room Classification

Finally, we evaluate the semantic room label maps. We evaluate with environment-generated audio, which provides natural object-room cues. Figure 6 compares the mean of the pixel-wise room classification average precision for our proposed model in the Env. Gen. All Room setting and its ablations.Note that the baselines from Table 1 are not applicable here because they produce only geometric interior maps. Results are averaged over all trajectory lengths. Audio can identify rooms in the neighborhood of the trajectory, and we see best results when both modalities are used together. The room map examples show AV-Map provides better room classification compared to our RGB-only variant, and does best with many natural semantic sounds—encouraging for deployment in a busy household. Our model can identify the correct room type and its approximate location, though without actually entering a room, its exact footprint naturally remains ambiguous.

Conclusion and Future Work

We proposed a new research direction: audio-visual floorplan reconstruction from short video sequences. We developed a multi-modal model to estimate the floorplan around and far beyond the camera trajectory. Our AV-Map model successfully infers the structure and semantics of areas that are not visible, outperforming the state-of-the-art in extrapolated visual maps. In future work we plan to consider extensions to multi-level floorplans and connect our mapping idea to a robotic agent actively controlling the camera.

Acknowledgements

We would like to thank Unnat Jain, Changan Chen, and Santhosh Kumar Ramakrishnan for help with generating the audio simulations and providing help with code for this work. We would also like to thank Ruohan Gao for providing feedback on the text.

References

Appendix A Additional Interior Map Visualizations

Figure 7 presents additional AV-Map interior map prediction visualizations, like Fig 3 in the main text. We see again how our model sees beyond the visible portions (cyan) to more fully map the space. We also highlight our failure modes; see the mis-classified locations (circled) on the predicted maps. We observe that the errors often arise in challenging locations that are not visually covered, where the model relies on the audio signal (see Figure 7 sample 1,4,5). Some errors arise from noise in the scan of the environment (see Figure 7 sample 3 - missing point cloud) since the rendered RGB frames are noisy.

Appendix B Room Map Visualizations and Confusion Analysis

In Figure 8, we present additional visualizations for the estimated room maps. The room maps were generated by the AV-Map model operating in the environment generated all-room audio setting. Green dots on the ground truth indicate the camera positions. From these visualizations, we observe that the model can successfully identify the approximate locations of several rooms. Some sources of errors are errors in interior estimation (see Column 1, Row 4 and Column 2, Row 3) and errors in localization of the rooms (see Column 1, Row 3).

In Figure 9, we present a confusion matrix for the pixel-wise room label predictions. We observe that there is a bias towards predicting the “bathroom”, “hallway” and “bedroom” classes which are the three most frequent room labels. The two least frequent classes (“stairs” and “closet”) are almost never predicted. This indicates that our model could benefit from training on a larger, more diverse and more balanced dataset. We also find that the rooms that are usually in close proximity have slightly higher confusion rates - for example, bedroom vs bathroom, and dining room vs kitchen. This suggests that our model struggles to accurately localize the boundaries of rooms (as also indicated in the main text).

Appendix C Importance of Sequence Modeling

At each time step of a video, the audio clip $a_{i}$ is generated by convolving a downloaded audio clip $c$ with an impulse response $\omega_{i}$. Therefore, the audio clip can be expressed as:

These relative changes can also provide information about the geometric properties of the environment. For example, walking past a door of a room containing a sound source will see a large change in impulse response clearly indicating the presence of an opening. Note here that the inferred relative change in impulse response does not rely on the original audio clip $c$ anymore. This is also a favorable feature since downloaded audio clips are not 100% anechoic. So in practice the audio clips $c$ encode the acoustic characteristics of the recording environment i.e. $c=\hat{c}\circledast\omega_{rec}$ where $\hat{c}$ is the anechoic audio and $\omega_{rec}$ is the impulse response of the recording setup. While we do not explicitly enforce the AV-Map model to infer these relative changes, training with multiple audio clips forces the model to learn to disentangle the effect of the impulse response.

The proposed AV-Map model allows training and testing with video sequences of arbitrary length. During training, the primary bottleneck for using very long sequences is the memory footprint and speed of computation. Training with $t_{V}=1$ is equivalent to making independent predictions at each time step and pooling them to obtain the final interior map estimate. For each time step, such a model would not be able to make inferences using visual features in other time steps (for example, the fact that the camera entered a door in the first step provides additional context at the second time step). Furthermore, as explained above, making independent predictions does not allow us to model relative changes in the impulse responses. We observed that sequences of length $t_{V}=4$ provide the benefits of modeling sequences while maintaining a manageable training duration. As promised in the main paper, in Table 3, we show results with $t_{V}=1$ and compare to the $t_{V}=4$ setting to demonstrate the impact of sequence modeling.

In the main text, we presented a quantitative analysis of the AV-Map model trained to estimate interior maps for an area of 40$m^{2}$ around the camera at each time step (by setting hyper-parameters $H,W$). As promised in Section 4.1 of the main paper, here in Table 4, we present similar quantitative resultsNote that the positive and negative pixels are balanced by reweighting as discussed in Section 4.1 for a model trained to predict a 164$m^{2}$ area around the camera at each step. We observe similar results demonstrating the improved performance of the AV-Map model compared to the RGB-only model.

Appendix E Additional Dataset Details

We use the Matterport3D dataset to generate video sequences (see Sec 3.4 of the main text). We use the splits provided by the SoundSpaces dataset for training, validation, and testing. We include the environments in the splits here for reference:

Train environments: [’17DRP5sb8fy’, ’1LXtFkjw3qL’, ’1pXnuDYAj8r’, ’29hnd4uzFmX’, ’5LpN3gDmAk7’, ’5q7pvUzZiYa’, ’759xd9YjKW5’, ’7y3sRwLe3Va’, ’82sE5b5pLXE’, ’8WUmhLawc2A’, ’aayBHfsNo7d’, ’ac26ZMwG7aT’, ’B6ByNegPMKs’, ’b8cTxDM8gDG’, ’cV4RVeZvu5T’, ’D7N2EKCX4Sj’, ’e9zR4mvMWw7’, ’EDJbREhghzL’, ’GdvgFV5R1Z5’, ’gTV8FGcVJC9’, ’HxpKQynjfin’, ’i5noydFURQK’, ’JeFG25nYj2p’, ’JF19kD82Mey’, ’jh4fc5c5qoQ’, ’kEZ7cmS4wCh’, ’mJXqzFtmKg4’, ’p5wJjkQkbXX’, ’Pm6F8kyY3z2’, ’pRbA3pwrgk9’, ’PuKPg4mmafe’, ’PX4nDJXEHrG’, ’qoiz87JEwZ2’, ’rPc6DW4iMge’, ’s8pcmisQ38h’, ’S9hNv5qa7GM’, ’sKLMLpTHeUy’, ’SN83YJsR3w2’, ’sT4fr6TAbpF’, ’ULsKaCPVFJR’, ’uNb9QFRL6hY’, ’Uxmj2M2itWa’, ’V2XKFyX4ASd’, ’VFuaQ6m2Qom’, ’VVfe2KiqLaN’, ’Vvot9Ly1tCj’, ’vyrNrziPKCB’, ’VzqfbhrpDEA’, ’XcA2TqTSSAj’, ’D7G3Y4RVNrH’, ’E9uDoFAP3SH’, ’JmbYfDe2QKZ’, ’r1Q1Z4BcV1o’, ’r47D5H71a5s’, ’ur6pFq6Qu1A’, ’VLzqgDo317F’, ’YmJkqBEsHnH’, ’ZMojNkEp431’]

Val environments: [’2azQ1b91cZZ’, ’8194nk5LbLH’, ’EU6Fwq7SyZv’, ’oLBMNvg9in8’, ’QUCTc6BB5sX’, ’TbHJrupSAjP’, ’X7HyMhZNoso’, ’pLe4wQe7qrG’, ’x8F5xyUWy9e’, ’Z6MFQCViBuw’, ’zsNo4HB9uLZ’]

Test environments: [’5ZKStnWn8Zo’, ’ARNzJeq3xxb’, ’fzynW3qQPVF’, ’jtcxE69GiFV’, ’pa4otMbVnkk’, ’q9vSo1VnCiC’, ’rqfALeAoiTq’, ’UwV83HsGsw3’, ’wc2JMjhGNzB’, ’WYY7iVyf5p8’, ’YFuZgdQ5vWj’, ’yqstnuAEVhm’, ’gxdoqLR6rwA’, ’gYvKGZ5eRqb’, ’Vt2qJdWjCF2’]

Room types and associated sounds

For generating room maps, we choose the 13 most frequent room types. For each room type, we download sounds from www.freesound.org generated by objects (or people) that are unique to the room type. Here we present the list of rooms, their associated sounds, and the number of train/val/test sounds for each:

kitchen: blender (3/1/1), cabinet (1/1/1), dishwasher (3/2/2)

dining room: knife (4/1/1) , spoon(4/2/2)

laundryroom/mudroom: washing machine (5/3/3)

workout/gym/exercise: person panting (5/3/3)

Appendix F Implementation and Training Details

The AV-Map model is trained with a batchsize of 32 videos using 4 GPUs. Each sample in the batch is generated by randomly sampling a camera trajectory as described in the main text. We use the SGD optimizer with a starting learning rate of 0.1, momentum 0.9 and weight decay 0.00001. After 30000 SGD updates, we drop the learning rate to 0.01 and train for an additional 20000 SGD steps.

F.2 Positional Encoding

The positional encoding map added in the feature alignment stage (see Sec 3.2) is a 64-channel 2D map representing the position of each pixel with a 64 dimensional vector. For position (i,j) in the feature map, the positional encoding $PE(i,j)$ is computed as:

F.3 Feature Alignment

Here we present a pseudo-code to illustrate the feature alignment described in Section 3.2.