Image Matching across Wide Baselines: From Paper to Practice

Introduction

Matching two or more views of a scene is at the core of fundamental computer vision problems, including image retrieval Lowe04 ; Arandjelovic16 ; Radenovic16 ; Tolias2016 ; Noh17 , 3D reconstruction Agarwal09 ; Heinly15 ; Schoenberger16a ; Zhu_2018_CVPR , re-localization Sattler12 ; Sattler18 ; Lynen19 , and SLAM Mur15 ; DeTone17a ; DeTone17b . Despite decades of research, image matching remains unsolved in the general, wide-baseline scenario. Image matching is a challenging problem with many factors that need to be taken into account, e.g., viewpoint, illumination, occlusions, and camera properties. It has therefore been traditionally approached with sparse methods – that is, with local features.

Recent effort have moved towards holistic, end-to-end solutions Kendall15a ; Balntas_2018_ECCV ; Bui_2019_ICCV . Despite their promise, they are yet to outperform the separatists Sattler19 ; zhou2019learn that are based on the classical paradigm of step-by-step solutions. For example, in a classical wide baseline stereo pipeline Pritchett98a , one (1) extracts local features, such as SIFT Lowe04 , (2) creates a list of tentative matches by nearest-neighbor search in descriptor space, and (3) retrieves the pose with a minimal solver inside a robust estimator, such as the 7-point algorithm Hartley94_7pt in a RANSAC loop Fischler81 . To build a 3D reconstruction out of a set of images, same matches are fed to a bundle adjustment pipeline Hartley00 ; Triggs00 to jointly optimize the camera intrinsics, extrinsics, and 3D point locations. This modular structure simplifies engineering a solution to the problem and allows for incremental improvements, of which there have been hundreds, if not thousands.

New methods for each of the sub-problems, such as feature extraction and pose estimation, are typically studied in isolation, using intermediate metrics, which simplifies their evaluation. However, there is no guarantee that gains in one part of the pipeline will translate to the final application, as these components interact in complex ways. For example, patch descriptors, including very recent work He18b ; Wei18 ; Tian19 ; Mukundan19 , are often evaluated on Brown’s seminal patch retrieval database Brown10 , introduced in 2007. They show dramatic improvements – up to 39x relative Wei18 – over handcrafted methods such as SIFT, but it is unclear whether this remains true on real-world applications. In fact, we later demonstrate that the gap narrows dramatically when decades-old baselines are properly tuned.

We posit that it is critical to look beyond intermediate metrics and focus on downstream performance. This is particularly important now as, while deep networks are reported to outperform algorithmic solutions on classical, sparse problems such as outlier filtering Yi18a ; Ranftl18 ; Zhang19 ; Sun19 ; brachmann2019ngransac , bundle adjustment Tang19 ; shi2019selfsupervised , SfM Vijayna17 ; Deepsfm2019 and SLAM Tateno_2017_CVPR ; gradslam , our findings in this paper suggest that this may not always be the case. To this end, we introduce a benchmark for wide-baseline image matching, including:

A dataset with thousands of phototourism images of 25 landmarks, taken from diverse viewpoints, with different cameras, in varying illumination and weather conditions – all of which are necessary for a comprehensive evaluation. We reconstruct the scenes with SfM, without the need for human intervention, providing depth maps and ground truth poses for 26k images, and reserve another 4k for a private test set.

A modular pipeline incorporating dozens of methods for feature extraction, matching, and pose estimation, both classical and state-of-the-art, as well as multiple heuristics, all of which can be swapped out and tuned separately.

Two downstream tasks – stereo and multi-view reconstruction – evaluated with both downstream and intermediate metrics, for comparison.

A thorough study of dozens of methods and techniques, both hand-crafted and learned, and their combination, along with a recommended procedure for effective hyper-parameter selection.

The framework enables researchers to evaluate how a new approach performs in a standardized pipeline, both against its competitors, and alongside state-of-the-art solutions for other components, from which it simply cannot be detached. This is crucial, as true performance can be easily hidden by sub-optimal hyperparameters.

Related Work

The literature on image matching is too vast for a thorough overview. We cover relevant methods for feature extraction and matching, pose estimation, 3D reconstruction, applicable datasets, and evaluation frameworks.

Local features became a staple in computer vision with the introduction of SIFT Lowe04 . They typically involve three distinct steps: keypoint detection, orientation estimation, and descriptor extraction. Other popular classical solutions are SURF Bay06 , ORB Rublee11 , and AKAZE Alcantarilla13 .

Modern descriptors often train deep networks on pre-cropped patches, typically from SIFT keypoints (i.e.Difference of Gaussians or DoG). They include Deepdesc Simo-Serra15 , TFeat Balntas16b , L2-Net Tian17 , HardNet Mishchuk17 , SOSNet Tian19 , and LogPolarDesc Ebel19 – most of them are trained on the same dataset Brown10 . Recent works leverage additional cues, such as geometry or global context, including GeoDesc Luo18 and ContextDesc Luo19 . There have been multiple attempts to learn keypoint detectors separately from the descriptor, including TILDE Verdie15 , TCDet Zhang18bTCDet , QuadNet Savinov17 , and Key.Net BarrosoLaguna2019ICCVKeyNet . An alternative is to treat this as an end-to-end learning problem, a trend that started with the introduction of LIFT Yi16b and also includes DELF Noh17 , SuperPoint DeTone17b , LF-Net Ono18 , D2-Net Dusmanu19 and R2D2 R2D22019 .

2 Robust matching

Inlier ratios in wide-baseline stereo can be below 10% – and sometimes even lower. This is typically approached with iterative sampling schemes based on RANSAC Fischler81 , relying on closed-form solutions for pose solving such as the 5- Nister03 , 7- Hartley94_7pt or 8-point algorithm Hartley97 . Improvements to this classical framework include local optimization Chum03 , using likelihood instead of reprojection (MLESAC) Torr00 , speed-ups using probabilistic sampling of hypotheses (PROSAC) Chum05a , degeneracy check using homographies (DEGENSAC) Degensac2005 , Graph Cut as a local optimization (GC-RANSAC) gcransac2018 , and auto-tuning of thresholds using confidence margins (MAGSAC) magsac2019 .

As an alternative direction, recent works, starting with CNe (Context Networks) in Yi18a , train deep networks for outlier rejection taking correspondences as input, often followed by a RANSAC loop. Follow-ups include Ranftl18 ; Zhao19 ; Sun19 ; Zhang19 . Differently from RANSAC solutions, they typically process all correspondences in a single forward pass, without the need to iteratively sample hypotheses. Despite their promise, it remains unclear how well they perform in real-world settings, compared to a well-tuned RANSAC.

3 Structure from Motion (SfM)

In SfM, one jointly optimizes the location of the 3D points and the camera intrinsics and extrinsics. Many improvements have been proposed over the years Agarwal09 ; Heinly15 ; Cui_2017_CVPR_HSFM ; PSfMO2017 ; Zhu_2018_CVPR . The most popular frameworks are VisualSFM Wu13 and COLMAP Schoenberger16a – we rely on the latter, to both generate the ground truth and as the backbone of our multi-view task.

4 Datasets and benchmarks

Early works on local features and robust matchers typically relied on the Oxford dataset Miko04b , which contains 48 images and ground truth homographies. It helped establish two common metrics for evaluating local feature performance: repeatability and matching score. Repeatability evaluates the keypoint detector: given two sets of keypoints over two images, projected into each other, it is defined as the ratio of keypoints whose support regions’ overlap is above a threshold. The matching score (MS) is similarly defined, but also requires their descriptors to be nearest neighbours. Both require pixel-to-pixel correspondences – i.e., features outside valid areas are ignored.

A modern alternative to Oxford is HPatches Balntas17 , which contains 696 images with differences in illumination or viewpoint – but not both. However, the scenes are planar, without occlusions, limiting its applicability.

Other datasets that have been used to evaluate local features include DTU Aanaes12 , Edge Foci Zitnick11 , Webcam Verdie15 , AMOS Pultar19 , and Strecha’s Strecha08b . They all have limitations – be it narrow baselines, noisy ground truth, or a small number of images. In fact, most learned descriptors have been trained and evaluated on Brown07 , which provides a database of pre-cropped patches with correspondence labels, and measures performance in terms of patch retrieval. While this seminal dataset and evaluation methodology helped developed many new methods, it is not clear how results translate to different scenarios – particularly since new methods outperform classical ones such as SIFT by orders of magnitude, which suggests overfitting.

Datasets used for navigation, re-localization, or SLAM, in outdoor environments are also relevant to our problem. These include KITTI Geiger12 , Aachen Sattler12b , Robotcar Maddern17 , and CMU seasons Sattler18 ; Badino11 . However, they do not feature the wide range of transformations present in phototourism data. Megadepth Li18f is a more representative, phototourism-based dataset, which using COLMAP, as in our case – it could, in fact, easily be folded into ours.

Benchmarks, by contrast, are few and far between – they include VLBenchmark Lenc11 , HPatches Balntas17 , and SILDa SILDa2018 – all limited in scope. A large-scale benchmark for SfM was proposed in Schonberger17 , which built 3D reconstructions with different local features. However, only a few scenes contain ground truth, so most of their metrics are qualitative – e.g.number of observations or average reprojection error. Yi et al. Yi18a and Bian et al. Bian19 evaluate different methods for pose estimation on several datasets – however, few methods are considered and they are not carefully tuned.

We highlight some of these datasets/benchmarks, and their limitations, in Fig. 2. We are, to the best of our knowledge, the first to introduce a public, modular benchmark for 3D reconstruction with sparse methods using downstream metrics, and featuring a comprehensive dataset with a large range of image transformations.

The Image Matching Challenge PhotoTourism Dataset

While it is possible to obtain very accurate poses and depth maps under controlled scenarios with devices like LIDAR, this is costly and requires a specific set-up that does not scale well. For example, Strecha’s dataset Strecha08b follows that approach but contains only 19 images. We argue that a truly representative dataset must contain a wider range of transformations – including different imaging devices, time of day, weather, partial occlusions, etc. Phototourism images satisfy this condition and are readily available.

We thus build on 25 collections of popular landmarks originally selected in Heinly15 ; Thomee16 , each with hundreds to thousands of images. Images are downsampled with bilinear interpolation to a maximum size of 1024 pixels along the long-side and their poses were obtained with COLMAP Schoenberger16a , which provides the (pseudo) ground truth. We do exhaustive image matching before Bundle Adjustment – unlike Schoenberger16b , which uses only 100 pairs for each image – and thus provide enough matching images for any conventional SfM to return near-perfect results in standard conditions.

Our approach is to obtain a ground truth signal using reliable, off-the-shelf technologies, while making the problem as easy as possible – and then evaluate new technologies on a much harder problem, using only a subset of that data. For example, we reconstruct a scene with hundreds or thousands of images with vanilla COLMAP and then evaluate “modern” features and matchers against its poses using only two images (“stereo”) or up to 25 at a time (“multiview” with SfM). For a discussion regarding the accuracy of our ground truth data, please refer to Section 3.3.

In addition to point clouds, COLMAP provides dense depth estimates. These are noisy, and have no notion of occlusions – a depth value is provided for every pixel. We remove occluded pixels from depth maps using the reconstructed model from COLMAP; see Fig. 3 for examples. We rely on these “cleaned” depth maps to compute classical, pixel-wise metrics – repeatability and matching score. We find that some images are flipped 90∘, and use the reconstructed pose to rotate them – along with their poses – so they are roughly ‘upright’, which is a reasonable assumption for this type of data.

Out of the 25 scenes, containing almost 30k registered images in total, we select 2 for validation and 9 for testing. The remaining scenes can be used for training, if so desired, and are not used in this paper. We provide images, 3D reconstructions, camera poses, and depth maps, for every training and validation scene. For the test scenes we release only a subset of 100 images and keep the ground truth private, which is an integral part of the Image Matching Challenge. Results on the private test set can be obtained by sending submissions to the organizers, who process them.

The scenes used for training, validation and test are listed in Table 1, along with the acronyms used in several figures. For the validation experiments – Sections 5 and 7 – we choose two of the larger scenes, “Sacre Coeur” and “St. Peter’s Square”, which in our experience provide results that are quite representative of what should be expected on the Image Matching Challenge Dataset. These two subsets have been released, so that the validation results are reproducible and comparable. They can all be downloaded from the challenge website\@footnotemark.

2 Estimating the co-visibility between two images

When evaluating image pairs, one has to be sure that two given images share a common part of the scene – they may be registered by the SfM reconstruction without having any pixels in common as long as other images act as a ‘bridge’ between them. Co-visible pairs of images are determined with a simple heuristic. 3D model points, detected in both the considered images, are localised in 2D in each image. The bounding box around the keypoints is estimated. The ratio of the bounding box area and the whole image is the “visibility” of image $i$ in $j$ and $j$ in $i$ respectively; see Fig. 5 for examples. The minimum of the two “visibilities” is the “co-visibility” ratio $v_{i,j}\in$, which characterizes how challenging matching of the particular image pair is. The co-visibility varies significantly from scene to scene. The histogram of co-visibilities is shown in Fig. 4, providing insights into how “hard” each scene is – without accounting for some occlusions.

For stereo, the minimum co-visibility threshold is set to 0.1. For the multi-view task, subsets where at least 100 3D points are visible in each image, are selected, as in Yi18a ; Zhang19 . We find that both criteria work well in practice.

3 On the quality of the “ground-truth”

Our core assumption is that accurate poses can be obtained from large sets of images without human intervention. Such poses are used as the “ground truth” for evaluation of image matching performance on pairs or small subsets of images – a harder, proxy task. Should this assumption hold, the (relative) poses retrieved with a large enough number of images would not change as more images are added, and these poses would be the same regardless of which local feature is used.

To validate this, we pick the scene “Sacre Coeur” and compute SfM reconstructions with a varying number of images: 100, 200, 400, 800, and 1179 images (the entire “Sacre Coeur” dataset), where each set contains the previous one; new images are being added and no images are removed. We run each reconstruction three times, and report the average result of the three runs, to account for the variance inside COLMAP. The standard deviation among different runs is reported in Table 2. Note that these reconstructions are only defined up to a scale factor – we do not know the absolute scale that could be used to compare the reconstructions against each other. That is why we use a simple, pairwise metric instead. We pick all the pairs out of the 100 images present in the smallest subset, and compare how much their relative pose change with respect to their counterparts reconstructed using the entire set – we do this for every subset, i.e., 100, 200, etc. Ideally, we would like the differences between the relative poses to approach zero as more images are added. We list the results in Table 3, for different local feature methods. Notice how the poses converge, especially in terms of the median, as more images are used, for all methods – and that the reconstructions using only 100 images are already very stable. For SuperPoint we use a smaller number of features (2k per image), which is not enough to achieve pose convergence, but the error is still reduced as more images are used.

We conduct a second experiment to verify that there is no bias towards using SIFT features for obtaining the ground truth. We compare the poses obtained with SIFT to those obtained with other local features – note that our primary metric uses nothing but the estimated poses for evaluation. We report results in Table 4. We also show the histogram of pose differences in Fig. 7. The differences in pose due to the use of different local features have a median value below 0.1∘. In fact, the pose variation of individual reconstructions with SuperPoint is of the same magnitude as the difference between reconstructions from SuperPoint and other local features: see Table 3. We conjecture that the reconstructions with SuperPoint, which does not extract a large number of keypoints, are less accurate and stable. This is further supported by the fact that the point cloud obtained with the entire scene generated with SuperPoint is less dense (125K 3D points) than the ones generated with SIFT (438K) or R2D2 (317k); see Fig. 6. In addition, we note that SuperPoint keypoints have been shown to be less accurate when it comes to precise alignment (DeTone17b, , Table 4, $\epsilon=1$). Note also that the poses from R2D2 are nearly identical to those from SIFT.

These observations reinforce our trust in the accuracy of our ground truth – given a sufficient number of images, the choice of local feature is irrelevant, at least for the purpose of retrieving accurate poses. Our evaluation considers pose errors of up to 10∘, at a resolution of 1∘ – significantly smaller than the fluctuations observed here, which we consider negligible. Note that these conclusions may not hold on large-scale SfM requiring loop closures, but our dataset contains landmarks, which do not suffer from this problem.

In addition, we note that the use of dense, ground truth depth from SfM, which is arguably less accurate that camera poses, has been verified by multiple parties, for training and evaluation, including: CNe Yi18a , DFE Ranftl18 , LF-Net Ono18 , D2-Net Dusmanu19 , LogPolarDesc Ebel19 , OANet zhang2019oanet , and SuperGlue sarlin20superglue among others, suggesting it is sufficiently accurate – several of these rely on the data used in our paper.

As a final observation, while the poses are stable, they could still be incorrect. This can happen on highly symmetric structures: for instance, a tower with a square or circular cross section. In order to prevent such errors from creeping into our evaluation, we visually inspected all the images in our test set. Out of 900 of them, we found 4 misregistered samples, all of them from the same scene, “London Bridge”, which were removed from our data.

Pipeline

We outline our pipeline in Fig. 8. It takes as input $N{=}100$ images per scene. The feature extraction module computes up to $K$ features from each image. The feature matching module generates a list of putative matches for each image pair, i.e. $\frac{1}{2}N(N-1)=4950$ combinations. These matches can be optionally processed by an outlier pre-filtering module. They are then fed to two tasks: stereo, and multiview reconstruction with SfM. We now describe each of these components in detail.

We consider three broad families of local features. The first includes full, “classical” pipelines, most of them handcrafted: SIFT Lowe04 (and RootSIFT rootSIFT2012 ), SURF Bay06 , ORB Rublee11 , and AKAZE Alcantarilla13 . We also consider FREAK Alahi12 descriptors with BRISK Brisk2011 keypoints. We take these from OpenCV. For all of them, except ORB, we lower the detection threshold to extract more features, which increases performance when operating with a large feature budget. We also consider DoG alternatives from VLFeat vedaldi2010vlfeat : (VL-)DoG, Hessian Hessian78 , Hessian-Laplace Mikolajczyk04 , Harris-Laplace Mikolajczyk04 , MSER Matas04 ; and their affine-covariant versions: DoG-Affine, Hessian-Affine Mikolajczyk04 ; Baumberg00 , DoG-AffNet Mishkin18 , and Hessian-AffNet Mishkin18 .

The second group includes descriptors learned on DoG keypoints: L2-Net Tian17 , Hardnet Mishchuk17 , Geodesc Luo18 , SOSNet Tian19 , ContextDesc Luo19 , and LogPolarDesc Ebel19 .

The last group consists of pipelines learned end-to-end (e2e): Superpoint DeTone17b , LF-Net Ono18 , D2-Net Dusmanu19 (with both single- (SS) and multi-scale (MS) variants), and R2D2 Revaud19 .

Additionally, we consider Key.Net BarrosoLaguna2019ICCVKeyNet , a learned detector paired with HardNet and SOSNet descriptors – we pair it with original implementation of HardNet instead than the one provided by the authors, as it performs betterIn BarrosoLaguna2019ICCVKeyNet the models are converted to TensorFlow – we use the original PyTorch version.. Post-IJCV update. We have added 3 more local features to the benchmark after official publication of the paper – DoG-AffNet-HardNet Mishchuk17 ; Mishkin18 , DoG-TFeat Balntas16b and DoG-MKD-Concat mukundan2018understanding . We take them from the kornia eriba2019kornia library. Results of this features are included into the Tables and Figures, but not discussed in the text.

2 Feature matching

The “both” strategy is similar to the “symmetrical nearest neighbor ratio” (sNNR) bellavia2020NewAboutSIFT , proposed concurrently – SNNR combines the nearest neighbor ratio in both directions into a single number by taking the harmonic mean, while our test takes the maximum of the two values.

3 Outlier pre-filtering

Context Networks Yi18a , or CNe for short, proposed a method to find sparse correspondences with a permutation-equivariant deep network based on PointNet Qi17a , sparking a number of follow-up works Ranftl18 ; Dang18a ; Zhao19 ; Zhang19 ; Sun19 . We embed CNe into our framework. It often works best when paired with RANSAC Yi18a ; Sun19 , so we consider it as an optional pre-filtering step before RANSAC – and apply it to both stereo and multiview. As the published model was trained on one of our validation scenes, we re-train it on “Notre Dame Front Facade” and “Buckingham Palace”, following CNe training protocol, i.e., with 2000 SIFT features, unidirectional matching, and no ratio test. We evaluated the new model on the test set and observed that its performance is better than the one that was released by the authors. It could be further improved by using different matching schemes, such as bidirectional matching, but we have not explored this in this paper and leave as future work.

We perform one additional, but necessary, change: CNe (like most of its successors) was originally trained to estimate the Essential matrix instead of the Fundamental matrix Yi18a , i.e., it assumes known intrinsics. In order to use it within our setup, we normalize the coordinates by the size of the image instead of using ground truth calibration matrices. This strategy has also been used in Sun19 , and has been shown to work well in practice.

4 Stereo task

The list of tentative matches is given to a robust estimator, which estimates $\mathbf{F}_{i,j}$, the Fundamental matrix between $\mathbf{I}_{i}$ and $\mathbf{I}_{j}$. In addition to (locally-optimized) RANSAC Fischler81 ; LOransac2003 , as implemented in OpenCV OpenCV , and sklearn sklearn , we consider more recent algorithms with publicly available implementations: DEGENSAC Degensac2005 , GC-RANSAC gcransac2018 and MAGSAC magsac2019 . We use the original GC-RANSAC implementationhttps://github.com/danini/graph-cut-ransac/tree/benchmark-version, and not the most up-to-date version, which incorporates MAGSAC, DEGENSAC, and later changes.

For DEGENSAC we additionally consider disabling the degeneracy check, which theoretically should be equivalent to the OpenCV and sklearn implementations – we call this variant “PyRANSAC”. Given $\mathbf{F}_{i,j}$, the known intrinsics $\mathbf{K}_{\{i,j\}}$ were used to compute the Essential matrix $\mathbf{E}_{i,j}$, as $\mathbf{E}_{i,j}=\mathbf{K}_{j}^{T}\mathbf{F}_{i,j}\mathbf{K}_{i}$. Finally, the relative rotation and translation vectors were recovered with a cheirality check with OpenCV’s recoverPose.

5 Multiview task

Large-scale SfM is notoriously hard to evaluate, as it requires accurate ground truth. Since our goal is to benchmark local features and matching methods, and not SfM algorithms, we opt for a different strategy. We reconstruct a scene from small image subsets, which we call “bags”. We consider bags of 5, 10, and 25 images, which are randomly sampled from the original set of 100 images per scene – with a co-visibility check. We create 100 bags for bag sizes 5, 50 for bag size 10, and 25 for bag size 25 – i.e., 175 SfM runs in total.

We use COLMAP Schoenberger16a , feeding it the matches computed by the previous module – note that this comes before the robust estimation step, as COLMAP implements its own RANSAC. If multiple reconstructions are obtained, we consider the largest one. We also collect and report statistics such as the number of landmarks or the average track length. Both statistics and error metrics are averaged over the three bag sizes, each of which is in turn averaged over its individual bags.

6 Error metrics

Since the stereo problem is defined up to a scale factor Hartley00 , our main error metric is based on angular errors. We compute the difference, in degrees, between the estimated and ground-truth translation and rotation vectors between two cameras. We then threshold it over a given value for all possible – i.e., co-visible – pairs of images. Doing so over different angular thresholds renders a curve. We compute the mean Average Accuracy (mAA) by integrating this curve up to a maximum threshold, which we set to 10∘ – this is necessary because large errors always indicate a bad pose: 30∘ is not necessarily better than 180∘, both estimates are wrong. Note that by computing the area under the curve we are giving more weight to methods which are more accurate at lower error thresholds, compared to using a single value at a certain designated threshold.

This metric was originally introduced in Yi18a , where it was called mean Average Precision (mAP). We argue that “accuracy” is the correct terminology, since we are simply evaluating how many of the predicted poses are “correct”, as determined by thresholding over a given value – i.e., our problem does not have “false positives”.

The same metric is used for multiview. Because we do not know the scale of the scene a priori, it is not possible to measure translation error in metric terms. While we intend to explore this in the future, such a metric, while more interpretable, is not without problems – for instance, the range of the distance between the camera and the scene can vary drastically from scene to scene and make it difficult to compare their results. To compute the mAA in pose estimation for the multiview task, we take the mean of the average accuracy for every pair of cameras – setting the pose error to $\infty$ for pairs containing unregistered views. If COLMAP returns multiple models which cannot be co-registered (which is rare) we consider only the largest of them for simplicity.

For the stereo task, we can report this value for different co-visibility thresholds: we use $v=0.1$ by default, which preserves most of the “hard” pairs. Note that this is not applicable to the multiview task, as all images are registered at once via bundle adjustment in SfM.

Finally, we consider repeatability and matching score. Since many end-to-end methods do not report and often do not have a clear measure of scale – or support region – we simply threshold by pixel distance, as in Rosten10 . For the multiview task, we also compute the Absolute Trajectory Error (ATE) Sturm12 , a metric widely used in SLAM. Since, once again, the reconstructed model is scale-agnostic, we first scale the reconstructed model to that of the ground truth and then compute the ATE. Note that ATE needs a minimum of three points to align the two models.

7 Implementation

The benchmark code has been open-sourced\@footnotemark along with every method used in the paperhttps://github.com/vcg-uvic/image-matching-benchmark-baselines. The implementation relies on SLURM Yoo03 for scalable job scheduling, which is compatible with our supercomputer clusters – we also provide on-the-cloud, ready-to-go imageshttps://github.com/etrulls/slurm-gcp. The benchmark can also run on a standard computer, sequentially. It is computationally expensive, as it requires matching about 45k image pairs. The most costly step – leaving aside feature extraction, which is very method-dependent – is typically feature matching: 2–6 seconds per image pairTime measured on ‘n1-standard-2’ VMs on Google Cloud Compute: 2 vCPUs with 7.5 GB of RAM and no GPU., depending on descriptor size. Outlier pre-filtering takes about 0.5–0.8 seconds per pair, excluding some overhead to reformat the data into its expected format. RANSAC methods vary between 0.5–1 second – as explained in Section 5 we limit their number of iterations based on a compute budget, but the actual cost depends on the number of matches. Note that these values are computed on the validation set – for the test set experiments we increase the RANSAC budget, in order to remain compatible with the rules of the Image Matching Challenge\@footnotemark. We find COLMAP to vary drastically between set-ups. New methods will be continuously added, and we welcome contributions to the code base.

Details are Important

Our experiments indicate that each method needs to be carefully tuned. In this section we outline the methodology we used to find the right hyperparameters on the validation set, and demonstrate why it is crucial to do so.

Robust estimators are, in our experience, the most sensitive part of the stereo pipeline, and thus the one we first turn to. All methods considered in this paper have three parameters in common: the confidence level in their estimates, $\tau$; the outlier (epipolar) threshold, $\eta$; and the maximum number of iterations, $\Gamma$. We find the confidence value to be the least sensitive, so we set it to $\tau=0.999999$.

We evaluate each method with different values for $\Gamma$ and $\eta$, using reasonable defaults: 8k SIFT features with bidirectional matching with the “both” strategy and a ratio test threshold of 0.8. We plot the results in Fig. 9, against their computational cost – for the sake of clarity we only show the curve corresponding to the best reprojection threshold $\eta$ for each method.

Our aim with this experiment is to place all methods on an “even ground” by setting a common budget, as we need to find a way to compare them. We pick 0.5 seconds, where all methods have mostly converged. Note that these are different implementations and are obviously not directly comparable to each other, but this is a simple and reasonable approach. We set this budget by choosing $\Gamma$ as per Fig. 9, instead of actually enforcing a time limit, which would not be comparable across different set-ups. Optimal values for $\Gamma$ can vary drastically, from 10k for MAGSAC to 250k for PyRANSAC. MAGSAC gives the best results for this experiment, closely followed by DEGENSAC. We patch OpenCV to increase the limit of iterations, which was hardcoded to $\Gamma=1000$; this patch is now integrated into OpenCV. This increases performance by 10-15% relative, within our budget. However, PyRANSAC is significantly better than OpenCV version even with this patch, so we use it as our “vanilla” RANSAC instead. The sklearn implementation is too slow for practical use.

We find that, in general, default settings can be woefully inadequate. For example, OpenCV recommends $\tau=0.99$ and $\eta=3$ pixels, which results in a mAA at 10∘ of 0.3642 on the validation set – a performance drop of 29.3% relative.

2 RANSAC: One method at a time

The last free parameter is the inlier threshold $\eta$. We expect the optimal value for this parameter to be different for each local feature, with looser thresholds required for methods operating on higher recall/lower precision, and end-to-end methods trained on lower resolutions.

We report a wide array of experiments in Fig. 10 that confirm our intuition: descriptors learned on DoG keypoints are clustered, while others vary significantly. Optimal values are also different for each RANSAC variant. We use the ratio test with the threshold recommended by the authors of each feature, or a reasonable value if no recommendation exists, and the “both” matching strategy – this cuts down on the number of outliers.

3 Ratio test: One feature at a time

Having “frozen” RANSAC, we turn to the feature matcher – note that it comes before RANSAC, but it cannot be evaluated in isolation. We select PyRANSAC as a “baseline” RANSAC and evaluate different ratio test thresholds, separately for the stereo and multiview tasks. For this experiment, we use 8k features with all methods, except for those which cannot work on this regime – SuperPoint and LF-Net. This choice will be substantiated in Section 5.4. We report the results for bidirectional matching with the “both” strategy in Fig. 11, and with the “either” strategy in Fig. 12. We find that “both” – the method we have used so far – performs best overall. Bidirectional matching with the “either” strategy produces many (false) matches, increasing the computational cost in the estimator, and requires very small ratio test thresholds – as low as $r{=}0.65$. Our experiments with unidirectional matching indicate that is slightly worse, and it depends on the order of the images, so we did not explore it further.

As expected, each feature requires different settings, as the distribution of their descriptors is different. We also observe that optimal values vary significantly between stereo and multiview, even though one might expect that bundle adjustment should be able to better deal with outliers. We suspect that this indicates that there is room for improvement in COLMAP’s implementation of RANSAC.

Note how the ratio test is critical for performance, and one could arbitrarily select a threshold that favours one method over another, which shows the importance of proper benchmarking. Interestingly, D2-Net is the only method that clearly performs best without the ratio test. It also performs poorly overall in our evaluation, despite reporting state-of-the-art results in other benchmarks Miko04b ; Balntas17 ; Sattler12 ; taira2018inloc – without the ratio test, the number of tentative matches might be too high for RANSAC or COLMAP to perform well.

Additionally, we implement the first-geometric-inconsistent ratio threshold, or FGINN Mishkin15 . We find that although it improves over unidirectional matching, its gains mostly disappear against matching with “both”. We report these results in Section 7.2.

4 Choosing the number of features

The ablation tests in this section use (up to) $K{=}8000$ feature (2k for SuperPoint and LF-Net, as they are trained to extract fewer keypoints). This number is commensurate with that used by SfM frameworks Wu13 ; Schoenberger16a . We report performance for different values of $K$ in Fig. 13. We use PyRANSAC with reasonable defaults for all three matching strategies, with SIFT features.

As expected, performance is strongly correlated with the number of features. We find 8k to be a good compromise between performance and cost, and also consider 2k (actually 2048) as a ‘cheaper’ alternative – this also provides a fair comparison with some learned methods which only operate on that regime. We choose these two values as valid categories for the open challenge\@footnotemark linked to the benchmark, and do the same on this paper for consistency.

5 Additional experiments

Some methods require additional considerations before evaluating them on the test set. We briefly discuss them in this section. Further experiments are available in Section 7.

We consider three binary descriptors: ORB Rublee11 , AKAZE Alcantarilla13 , and FREAK Alahi12 Binary descriptor papers historically favour a distance threshold in place of the ratio test to reject non-discriminative matches Rublee11 , although some papers have used the ratio test for ORB descriptors saddle2019 . We evaluate both in Fig. 14 – as before, we use up to 8k features and matching with the “both” strategy. The ratio test works better for all three methods – we use it instead of a distance threshold for all experiments in the paper, including those in the previous sections.

On the influence of the detector (Fig. 15).

We embed several popular blob and corner detectors into our pipeline, with OpenCV’s DoG Lowe04 as a baseline. We combine multiple methods, taking advantage of the VLFeat library: Difference of Gaussians (DoG), Hessian Hessian78 , HessianLaplace Mikolajczyk04 , HarrisLaplace Mikolajczyk04 , MSER Matas04 , DoGAffine, Hessian-Affine Mikolajczyk04 ; Baumberg00 , DoG-AffNet Mishkin18 , and Hessian-AffNet Mishkin18 . We pair them with SIFT descriptors, also computed with VLFeat, as OpenCV cannot process affine keypoints, and report the results in Fig. 15. VLFeat’s DoG performs marginally better than OpenCV’s. Its affine version gives a small boost. Given the small gain and the infrastructure burden of interacting with a Matlab/C library, we use OpenCV’s DoG implementation for most of this paper.

On increasing the support region (Fig. 16).

The size (“scale”) of the support region used to compute a descriptor can significantly affect its performance Dong15 ; Zagoruyko15 ; Aanaes02 . We experiment with different scaling factors, using DoG with SIFT and HardNet Mishchuk17 , and find that 12$\times$ the OpenCV scale (the default value) is already nearly optimal, confirming the findings reported in Ebel19 . We show these results in Fig. 16. Interestingly, SIFT descriptors do benefit from increasing the scaling factor from 12 to 16, but the difference is very small – we thus use the recommended value of 12 for the rest of the paper. This, however, suggests that deep descriptors such as HardNet might be able to increase performance slightly by training on larger patches.

Establishing the State of the Art

With the findings and the optimal parameters found in Section 5, we move on to the test set, evaluating many methods with their optimal settings. All experiments in this section use bidirectional matching with the ‘’both” strategy. We consider a large feature budget (up to 8k features) and a smaller one (up to 2k), and evaluate many detector/descriptor combinations.

We make three changes with respect to the validation experiments of the previous section. (1) We double the RANSAC budget from 0.5 seconds (used for validation) to 1 second per image pair, and adjust the maximum number of iterations $\Gamma$ accordingly – we made this decision to encourage participants to the challenge based on this benchmark to use built-in methods rather than run RANSAC themselves to squeeze out a little extra performance, and use the same values in the paper for consistency. (2) We run each stereo and multiview evaluation three times and average the results, in order to decrease the potential randomness in the results – in general, we found the variations within these three runs to be negligible. (3) We use brute-force to match descriptors instead of FLANN, as we observed a drop in performance. For more details, see Section 6.6 and Table 12.

For stereo, we consider DEGENSAC and MAGSAC, which perform the best in the validation set, and PyRANSAC as a ‘baseline’ RANSAC. We report the results with both 8k features and 2k features in the following subsections. All observations are in terms of mAA, our primary metric, unless stated otherwise.

On the stereo task, deep descriptors extracted on DoG keypoints are at the top in terms of mAA, with SOSNet being #1, closely followed by HardNet. Interestingly, ‘HardNetAmos+’ HardNetAMOS2019 , a version trained on more datasets – Brown Brown10 , HPatches Balntas17 , and AMOS-patches HardNetAMOS2019 – performs worse than the original models, trained only on the “Liberty” scene from Brown’s dataset. On the multiview task, HardNet edges out ContextDesc, SOSNet and LogpolarDesc by a small margin.

We also pair HardNet and SOSNet with Key.Net, a learned detector, which performs worse than with DoG when extracting a large number of features, with the exception of Key.Net + SOSNet on the multiview task.

R2D2, the best performing end-to-end method, does well on multiview (#7), but performs worse than SIFT on stereo – it produces a much larger number of “inliers” (which may be correct or incorrect) than most other methods. This suggests that, like D2-Net, its lack of compatibility with the ratio test may be a problem when paired with sample-based robust estimators, due to a lower inlier ratio. Note that D2-net performs poorly on our benchmark, despite state-of-the-art results on others. On the multiview task it creates many more 3D landmarks than any other method. Both issues may be related to its poor localization (pixel) accuracy, due to operating on downsampled feature maps.

Out of the handcrafted methods, SIFT – RootSIFT specifically – remains competitive, being #10 on stereo and #9 on multiview, within 13.1% and 4.9% relative of the top performing method, respectively, while previous benchmarks report differences in performance of orders of magnitude. Other “classical” features do not fare so well. One interesting observation is that among these, their ranking on validation and test set is not consistent – Hessian is better on validation than DoG, but significantly worse on the test set, especially in the multiview setup. While this is a special case, this nonetheless demonstrates that a small-scale benchmark can be misleading, and a method needs to be tested on a variety of scenes, which is what our test set aims to provide.

Regarding the robust estimators, DEGENSAC and MAGSAC both perform very well, with the former edging out the latter for most local feature methods. This may be due to the nature of the scenes, which often contain dominant planes.

2 Results with 2k features — Tables 7 and 8

Results change slightly on the low-budget regime, where the top two spots on both tasks are occupied by Key.Net+SOSNet and Key.Net+HardNet. It is closely followed by LogPolarDesc (#3 on stereo and #4 on multiview), a method trained on DoG keypoints – but using a much larger support region, resampled into log-polar patches. R2D2 performs very well on the multiview task (#3), while once again falling a bit short on the stereo task (#8, and 14.5% relative below the #1 method), for which it retrieves a number of inliers significantly larger than its competitors. The rest of the end-to-end methods do not perform so well, other than SuperPoint, which obtains competitive results on the multiview task.

The difference between classical and learned methods is more pronounced than with 8k points, with RootSIFT once again at the top, but now within 31.4% relative of the #1 method on stereo, and 26.9% on multiview. This is somewhat to be expected, given that with fewer keypoints, the quality of each individual point matters more.

3 2k features vs 8k features — Figs. 17 and 18

We compare the results between the low- and high-budget regimes in Fig. 17, for stereo (with DEGENSAC), and Fig. 18, for multiview. Note how methods can behave quite differently. Those based on DoG significantly benefit from an increased feature budget, whereas those learned end-to-end may require re-training – this is exemplified by the difference in performance between 2k and 8k for Key.Net+Hardnet, specially on multiview, which is very narrow despite quadrupling the budget. Overall, learned detectors – KeyNet, SuperPoint, R2D2, LF-Net – show relatively better results on multiview setup than on stereo. Our hypothesis is that they have good robustness, but low localization precision, which is later corrected during bundle adjustment.

4 Outlier pre-filtering with deep networks — Table 9

Next, we study the performance of CNe Yi18a for outlier rejection, paired with PyRANSAC, DEGENSAC, and MAGSAC. Its training data does not use the ratio test, so we omit it here too – note that because of this, it expects a relatively large number of input matches. We thus evaluate it only for the 8k feature setting, while using the “both” matching strategy.

Our experiments with SIFT, the local feature used to train CNe, are encouraging: CNe aggressively filters out about 80% of the matches in a single forward pass, boosting mAA at 10∘ by 2-4% relative for stereo task and 8% for multiview task. In fact, it is surprising that nearly all classical methods benefit from it, with gains of up to 20% relative. By contrast, it damages performance with most learned descriptors, even those operating on DoG keypoints, and particularly for methods learned end-to-end, such as D2-Net and R2D2. We hypothesize this might be because the models performed better on the “classical” keypoints it was trained with – Sun19 reports that re-training them for a specific feature helps.

5 On the effect of local feature orientation estimation — Tables 10 and 11

In contrast with classical methods, which estimate the orientation of each keypoint, modern, end-to-end pipelines DeTone17b ; Dusmanu19 ; Revaud19 often skip this step, assuming that the images are roughly aligned (upright), with the descriptor shouldering the increased invariance requirements. As our images meet this condition, we experiment with setting the orientation of keypoints to a fixed value (zero). DoG often returns multiple orientations for the same keypoint, so we consider two variants: one where we simply remove keypoints which become duplicates after setting the orientation to a constant value (Upright), and a second one where we fill out the budget with new keypoints (Upright++). We list the results in Table 10, for stereo, and Table 11, for multiview. Performance increases across the board with Upright++ – albeit by a small margin.

6 On the effect of approximate nearest neighbor matching — Table 12

While it is known that approximate nearest neighbor search algorithms have non-perfect recall Muja09 , it is not clear how their usage influences downstream performance. We thus compare exact (brute-force) nearest neighbor search with a popular choice for approximate nearest neighbor search, FLANN Muja09 , as implemented in OpenCV. We experimented with different parameters and found that 4 trees and 128 checks provide a reasonable trade-off between precision and runtime. We report results with and without FLANN in Table 12. The performance drop varies for different methods: from a moderate 1$\%$ for RootSIFT, to 5-7$\%$ HardNet and SOSNet, and 18% for D2-Net.

7 Pose mAA vs. traditional metrics — Fig. 19

To examine the relationship between our pose metric and traditional metrics, we compare mAA against repeatability and matching score on the stereo task, with DEGENSAC. While the matching score seems to correlate with mAA, repeatability is harder to interpret. However, note that even for the matching score, which shows correlation, higher value does not guarantee high mAA – see e.g.RootSIFT vs ContextDesc. We remind the reader that, as explained in Section 4, our implementation differs from the classical formulation, as many methods do not have a strict notion of a support region. We compute these metrics at a 3-pixel threshold, and provide more granular results in Section 7.

As shown, all methods based on DoG are clustered, as they operate on the same keypoints. Key.Net obtains the best repeatability, but performs worse than DoG in terms of mAA with the same descriptors (HardNet). AKAZE and FREAK perform surprisingly well in terms of repeatability – #2 and #3, respectively – but obtain a low mAA, which may be related to their descriptors, which are binary. R2D2 shows good repeatability but a poor matching score and is outperformed by DoG-based features.

8 Breakdown by scene — Fig. 20

Results may vary drastically from scene to scene, as shown in Fig. 20. A given method may also perform better on some than others – for instance, D2-Net nears the state of the art on “Lincoln Memorial Statue”, but is 5x times worse on “British Museum”. AKAZE and ORB show similar behaviour. This can provide valuable insights on limitations and failure cases.

9 Breakdown by co-visibility — Figs. 21 and 22

Next, in Fig. 21 we evaluate stereo performance at different co-visibility thresholds, for several local feature methods, using DEGENSAC. Bins are encoded as $v$+, with $v$ the co-visibility threshold, and include all image pairs with a co-visibility value larger or equal than $v$. This means that the first bin may contain unmatchable images – we use 0.1+ for all other experiments in the paper. We do not report values above 0.6+ as there are only a handful of them and thus the results are very noisy.

Performance for all local features and RANSAC variants increases with the co-visibility threshold, as expected. Results are consistent, with end-to-end methods performing better at higher than co-visibility than lower, and single-scale D2-Net outperforming its multi-scale counterpart at 0.4+ and above, where the images are more likely aligned in terms of scale.

We also break down different RANSAC methods in Fig. 22, along with different local fetures, including AKAZE (binary), SIFT (also handcrafted), HardNet (learned descriptor on DoG points) and R2D2 (learned end-to-end). We do not observe significant variations in the trend of each curve as we swap RANSAC and local feature methods. DEGENSAC and MAGSAC show very similar performance.

10 Classical metrics vs. pixel threshold — Fig. 23

In Fig. 19 we plot repeatability and matching score against mAA at a fixed error threshold of 3 pixels. In Fig. 23 we show them at different pixel thresholds. End-to-end methods tend to perform better at higher pixel thresholds, which is expected – D2-Net in particular extracts keypoints from downsampled feature maps. These results are computed from the depth maps estimated by COLMAP, which are not pixel-perfect, so the results for very low thresholds are not completely trustworthy.

Note that repeatability is typically lower than matching score, which might be counter-intuitive as the latter is more strict – it requires two features to be nearest-neighbors in descriptor space in addition to being physically close (after reprojection). We compute repeatability with the raw set of keypoints, whereas the matching score is computed with optimal matching settings – bidirectional matching with the “both” strategy and the ratio test. This results in a much smaller pool – 8k features are typically narrowed down to 200–400 matches (see Table 13). This better isolates the performance of the detector and the descriptor where it matters.

11 Breakdown by angular threshold — Figs. 24 and 25

We summarize pose accuracy by mAA at 10∘ in order to have a single, easy-to-interpret number. In this section we show how performance varies across different error thresholds – we look at the average accuracy at every angular error threshold, rather than the mean Average Accuracy. Fig. 24 plots performance on stereo and multiview for different local feature types, showing that the ranks remain consistent across thresholds. Fig. 25 shows how it affects different RANSAC variants, with four different local features. Again, ranks do not change. DEGENSAC and MAGSAC perfom nearly identically for all features, except for R2D2. The consistency in the ranks demonstrate that summarizing the results with a single number, mAA at 10∘, is reasonable.

12 Qualitative results — Figs. 28, 29, 30, and 31

Figs. 28 and 29 show qualitative results for the stereo task. We draw the inliers produced by DEGENSAC and color-code them using the ground truth depth maps, from green (0) to yellow (5 pixels off) if they are correct, and in red if they are incorrect (more than 5 pixels off). Matches including keypoints which fall on occluded pixels are drawn in blue. Note that while the depth maps are somewhat noisy and not pixel-accurate, they are sufficient for this purpose. Notice how the best performing methods have more correct matches that are well spread across the overlapping region.

Fig. 30 shows qualitative results for the multiview task, for handcrafted detectors. We illustrate it by drawing keypoints used in the SfM reconstruction in blue and the rest in red. It showcases the importance of the detector, specially on unmatchable regions such as the sky. ORB and Hessian keypoints are too concentrated in the high-contrast regions, failing to provide evenly-distributed features. In contrast, SURF fails to filter-out background and sky points. Fig. 31 shows results for learned methods: DoG+HardNet, Key.Net+HardNet, SuperPoint, R2D2, and D2-Net (multiscale). They have different detection patterns: Key.Net resembles a “cleaned” version of DoG, while R2D2 seems to be evenly distributed. D2Net looks rather noisy, while SuperPoint fires precisely on corner points on structured parts, and sometimes form a regular grid on sky-like homogeneous regions, which might be due to the method running out of locations to place points at – note however that the best results were obtained with larger NMS (non-maxima suppression) thresholds.

Further results and considerations

In this section we provide additional results on the validation set. These include a study of the typical outlier ratios under optimal settings in Section 7.1, matching with FGINN in Section 7.2, image-preprocessing techniques for feature extraction in Section 7.3, and a breakdown of the optimal settings in Section 7.4, provided to serve as a reference.

We list the number of input matches and their resulting inliers for the stereo task, in Table 13. As before, we remind the reader that these inliers are what each method reports, i.e., the matches that are actually used to estimate the poses. We list the number of input matches, the number of inliers produced by each method (which may still contain outliers), their ratio, and the mAA at 10∘. We use PyRANSAC with optimal settings for each method, the ratio test, and bidirectional matching with the “both” strategy.

We see that inlier-to-outlier ratios hover around 35–40% for all features relying on classical detectors. Key.Net with HardNet descriptors sees a significant drop in inlier ratio and mAA, when compared to its DoG counterpart. D2-Net similarly has inlier ratios around 25%. R2D2 has the largest inlier ratio by far at 53%, but is outperformed by many other methods in terms of mAA, suggesting that many of these are not actual inliers. In general, we observe that the methods which produce a large number of matches, such as Key.Net (600+), D2-Net (600+) or R2D2 (900+) are less accurate in terms of pose estimation.

2 Feature matching with an advanced ratio test — Fig. 26

We also compare the benefits of applying first-geometrically-inconsistent-neighbor-ratio (FGINN) Mishkin15 to DoG/SIFT, DoG/HardNet and Key.Net/HardNet, against Lowe’s standard ratio test Lowe04 . FGINN performs the ratio-test with second-nearest neighbors that are “far enough” from the tentative match (10 pixels in Mishkin15 ). In other words, it loosens the test to allow for nearby-thus-similar points. We test it for 3 matching strategies: unidirectional (“uni”), “both” and “either”. We report the results in Fig. 26. As shown, FGINN provides minor improvements over the standard ratio test in case of unidirectional matching, and not as much when “both” is used. It also behaves differently compared to the standard strategy, in that performance at stricter thresholds degrades less.

3 Image pre-processing — Fig. 27

Contrast normalization is key to invariance against illumination changes – local feature methods typically apply some normalization strategy over small patches Lowe04 ; Mishchuk17 . Therefore, we experiment with contrast-limited adaptive histogram equalization (CLAHE) Pizer87 , as implemented in OpenCV. We apply it prior to feature detection and/or description with SIFT and several learned descriptors, and display the results in Fig. 27. Performance decreases for all learned methods, presumably because they are not trained for it. Contrary to our initial expectations, SIFT does not benefit much from it either: the only increase in performance comes from applying it for descriptor extraction, at 2.5% relative for stereo task and 0.56% relative for multi-view. This might be due to the small number of night-time images in our data. It also falls in line with the observations in Dong_2015_CVPR , which show that SIFT descriptors are actually optimal under certain assumptions.

4 Optimal settings breakdown — Tables 14 and 15

For the sake of clarity, we summarize the optimal hyperparameter combinations from Figs. 9, 10 and 11 in Table 14 (for 8k features) and Table 15 (for 2k features). We set the confidence value to $\tau{=}0.999999$ for all RANSAC variants. Notice how it is better to have more features and a stricter ratio test threshold to filter them out, than having fewer features from the beginning.

Conclusions

We introduce a comprehensive benchmark for local features and robust estimation algorithms. The modular structure of its pipeline allows to easily integrate, configure, and combine methods and heuristics. We demonstrate this by evaluating dozens of popular algorithms, from seminal works to the cutting edge of machine learning research, and show that classical solutions may still outperform the perceived state of the art with proper settings.

The experiments carried out through the benchmark and reported in this paper have already revealed unexpected, non-intuitive properties of various components of the SfM pipeline, which will benefit SfM development, e.g., the need to tune RANSAC to the particular feature detector and descriptor and to select specific settings for a particular RANSAC variant. Other interesting facts have been uncovered by our tests, such as that the optimal set-ups across different tasks (stereo and multiview) may differ, or that methods that perform better on proxy tasks, like patch retrieval or repeatability, may be inferior on the downstream task. Our work is open-sourced and makes the basis of an open challenge for image matching with sparse methods.

References