The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
Alex Zihao Zhu, Dinesh Thakur, Tolga Ozaslan, Bernd Pfrommer, Vijay Kumar, Kostas Daniilidis
I INTRODUCTION
EVENT based cameras sense the world by detecting changes in the log intensity of an image. By registering these changes with accuracy on the order of tens of microseconds and asynchronous, almost instant, feedback, they allow for extremely low latency responses compared to traditional cameras which typically have latencies on the order of tens of milliseconds. In addition, by tracking changes in log intensity, the cameras have very high dynamic range (130dB vs about 60dB with traditional cameras), which make them very useful for scenes with dramatic changes in lighting, such as indoor-outdoor transitions, as well as scenes with a strong light source, such as the sun.
However, most modern robotics algorithms have been designed for synchronous sensors, where measurements arrive at fixed time intervals. In addition, the generated events do not carry any intensity information on their own. As a result, new algorithms must be developed to fully take advantage of the benefits provided by this sensor. Unfortunately, due to the differences in measurements, we cannot directly take advantage of the enormous amounts of labeled data captured with traditional cameras. Such data has shown to be extremely important for providing realistic and consistent evaluations of new methods, training machine learning systems, and providing opportunities for new development for researchers who do not have access to these sensors.
In this work, we aim to provide a number of different sequences that will facilitate the research and development of novel solutions to a number of different problems. One main contribution is the first dataset with a synchronized stereo event camera system. A calibrated stereo system is useful for depth estimation with metric scale, which can contribute to problems such as pose estimation, mapping, obstacle avoidance and 3D reconstruction. There have been a few works in stereo depth estimation with event based cameras, but, due to the lack of accurate ground truth depth, the evaluations have been limited to small, disparate sequences, consisting of a few objects in front of the camera. In comparison, this dataset provides event streams from two synchronized and calibrated Dynamic Vision and Active Pixel Sensors (DAVIS-m346b), with long indoor and outdoor sequences in a variety of illuminations and speeds, along with accurate depth images and pose at up to 100Hz, generated from a lidar system rigidly mounted on top of the cameras, as in Fig 1, along with motion capture and GPS. We hope that this dataset can help provide a common basis for event based algorithm evaluation in a number of applications.
The full dataset can be found online at https://daniilidis-group.github.io/mvsec.
The main contributions from this paper can be summarized as:
The first dataset with synchronized stereo event cameras, with accurate ground truth depth and pose.
Event data from a handheld rig, a flying hexacopter, a car, and a motorcycle, in conjunction with calibrated sensor data from a 3D lidar, IMUs and frame based images, from a variety of different speeds, illumination levels and environments.
II RELATED WORK
At present, there are a number of existing datasets that provide events from monocular event based cameras in conjunction with a variety of other sensing modalities and ground truth measurements that are suitable for testing a number of different 3D perception tasks.
Weikersdorfer et al. combine the earlier eDVS sensor with 128x128 resolution, with a Primesense RGBD sensor, and provide a dataset of indoor sequences with ground truth pose from a motion capture system, and depth from the RGBD sensor.
Rueckauer et al. provide data from a DAVIS 240C camera undergoing pure rotational motion, as well as ground truth optical flow based on the angular velocities reported from the gyroscope, although this is subject to noise in the reported velocities.
Barranco et al. present a dataset with a DAVIS 240B camera mounted on top of a pan tilt unit, attached to a mobile base, along with a Microsoft Kinect sensor. The dataset provides sequences of the base moving with 5dof in a indoor environment, along with ground truth depth, and optical flow and pose from the wheel encoders on the base and the angles from the pan tilt unit. While the depth from the Kinect is accurate, the optical flow and pose are subject to drift from the position estimates of the base’s wheel encoders.
Mueggler et al. provide a number of handheld sequences intended for pose estimation in a variety of indoor and outdoor environments, generated from a DAVIS 240C. A number of the indoor scenes have provided pose ground truth, captured from a motion capture system. However, there are no outdoor sequences, or other sequences with a significant displacement, with ground truth information.
Binas et al. provide a large dataset of a DAVIS 346B mounted behind the windshield of a car, with 12 hours of driving, intended for end to end learning of various driving related tasks. The authors provide a number of auxiliary measurements from the vehicle, such as steering angle, accelerator pedal position, vehicle speed etc., as well as longitude and latitude from a GPS unit. However, no 6dof pose is provided, as only 2D translation can be inferred from the GPS output as provided.
These datasets provide valuable data for development and evaluation of event based methods. However, they have, to date, only monocular sequences, with ground truth 6dof pose limited to small indoor environments, with few sequences with ground truth depth. In contrast, this work provides stereo sequences with ground truth pose and depth images in a variety of indoor and outdoor settings.
II-B Event Based 3D Perception
Early works in , present stereo depth estimation results with a number of spatial and temporal costs. Later works in , and have adapted cooperative methods for stereo depth to event based cameras, due to their applicability to asynchronous, point based measurements. Similarly, and apply a set of temporal, epipolar, ordering and polarity constraints to determine matches, while compare this with matching based on the output of a bank of orientation filters. The authors in show a new method to determine the epipolar line, applied to stereo matching. In , the authors propose a novel context descriptor to perform matching, and the authors in use a stereo event camera undergoing pure rotation to perform depth estimation and panoramic stitching.
There are also a number of works on event based visual odometry and SLAM problems. The authors in and proposed novel methods to perform feature tracking in the event space, which they extended in and to perform visual and visual inertial odometry, respectively. In , the authors combine an event based camera with a depth sensor, to perform visual odometry and SLAM. The authors in use events to estimate angular velocity of a camera, while and perform visual odometry by building an up to a scale map. In addition, and also fuse events with measurements from an IMU to perform visual inertial odometry.
While the more recent works evaluate based on public datasets such as , the majority are evaluated on small datasets generated solely for the paper, making comparisons of performance difficult. This is particularly the case for stereo event based cameras. In this work, we try to generate more extensive ground truth, for more meaningful evaluations of new algorithms that can provide a basis for comparisons between methods.
III DATASET
For each sequence in this dataset, we provide the following measurements in ROS baghttp://wiki.ros.org/Bags format:
Events, APS grayscale images and IMU measurements from the left and right DAVIS cameras.
Images and IMU measurements from the VI Sensor.
Pointclouds from the Velodyne VLP-16 lidar.http://velodynelidar.com/vlp-16-lite.html
Ground truth reference poses for the left DAVIS camera.
Ground truth reference depth images for both left and right DAVIS cameras.
A list of sensors and their characteristics can be found in Table I. In addition, Fig 2(a) shows the CAD drawing of the sensor rig, with all sensor axes labeled, and Fig 2 shows how the sensors are mounted on each vehicle. The extrinsics between all sensors are estimated through calibration, as explained in Sec V.
For event generation, two experimental mDAVIS-346B cameras are mounted in a horizontal stereo setup. The cameras are similar to , but have a higher, 346x260 pixel, resolution, up to 50fps APS (frame based images) output, and higher dynamic range. The baseline of the stereo rig is 10cm, and the cameras are timestamp synchronized by using the trigger signal generated from the left camera (master) to deliver sync pulses to the right (slave) through an external wire. Both cameras have 4mm lenses with approximately 87 degrees horizontal field of view, with an additional IR cut filter placed on each one to suppress the IR flashes from the motion capture systems. The APS exposures are manually set (no auto exposure) depending on lighting conditions, but are always the same between the cameras. While the timestamps of the grayscale DAVIS images are synced, there is unfortunately no way to synchronize the image acquisition itself. Therefore, there may be up to 10ms of offset between the images.
To provide ground truth reference poses and depths (Sec IV), we have rigidly mounted a Velodyne Puck LITE above the stereo DAVIS cameras. The Velodyne lidar system provides highly accurate depth of a large number of points around the sensor. The lidar is mounted such that there is full overlap between the smaller vertical field of view of the lidar and that of the stereo DAVIS rig.
In the outdoor scenes, we have also mounted a GPS device for a second ground truth reference for latitude and longitude. Typically, the GPS is placed away from the sensor rig to avoid interference from the USB 3.0 data cables.
In addition, we have mounted a VI Sensor , originally developed by Skybotix for comparison with frame based methods. The sensor has a stereo pair with IMU, all synchronized. Unfortunately, the only mounting option was to mount the cameras upside down, but we provide the transform between them and the DAVIS cameras.
III-B Sequences
The full list of sequences with summary statistics can be found in Table II, and sample APS images with overlaid events can be found in Fig 3.
The sensor setup was mounted below the compute stack of the hexacopter, with a 25 degree downwards pitch, as in Fig 2(b). Two motion capture systems are used to generate sequences for this dataset, one indoors and one outdoors (Fig 4). The indoor area is instrumented with 20 Vicon Vantage VP-16 cameras. The outdoor netted area of is instrumented with an all-weather motion capture system comprised of 34 high resolution Qualisys Oqus 700 cameras. Both systems provide millimeter accuracy pose at 100Hz by emitting infrared strobes and tracking IR reflecting markers placed on the hexacopter. We provide sequences in each area, with flights of different length and speed.
III-B2 Handheld
In order to test performance in high dynamic range scenarios, the full sensor rig is carried in a loop through both outdoor and indoor environments, as well as indoor environments with and without external lighting. Ground truth pose and depth is provided by lidar SLAM.
III-B3 Outdoor Driving
For slow to medium speed sequences, the sensor rig is mounted on the sun roof of a sedan as in Fig 2(c), and driven around several West Philadelphia neighborhoods at speeds up to 12 m/s. Sequences are provided in both day and evening situations, including sequences with the sun directly in the cameras’ field of view. Ground truth is provided as depth images from a lidar map, as well as pose from loop closed lidar odometry and GPS.
For high speed sequences, the DAVIS stereo rig and VI Sensor are mounted on the handlebar of a motorcycle (Fig 2(d)), along with the GPS device. The sequences involve driving at up to 38m/s. Longitude and latitude, as well as relative velocity, are provided from the GPS.
IV GROUND TRUTH GENERATION
To provide ground truth poses, motion capture poses are used when available. Otherwise, if lidar is available, Cartographer is used for the driving sequences to fuse the lidar sweeps and IMU data into a loop-closed 2D pose of the lidar, which is transformed into the left DAVIS frame using the calibration in Sec V-D. For outdoor scenes, we also provide raw GPS readings.
For each sequence with lidar measurements, we run the Lidar Odometry and Mapping (LOAM) algorithm to generate dense 3D local maps, which are projected into each DAVIS camera to generate dense depth images at 20Hz, and to provide 3D pose for the handheld sequences.
Two separate lidar odometry algorithms are used as we noted that LOAM produces better, more well aligned, local maps, while Cartographer’s loop closure results in more accurate global poses with less drift for longer trajectories. While Cartographer only estimates a 2D pose, we believe that this is a valid assumption as the roads driven have, for the most part, a single consistent grade.
For the sequences in the indoor and outdoor motion capture arenas, the pose of the body frame of the sensor rig at each time is measured at 100Hz with millimeter level accuracy.
For outdoor sequences we rely on Cartographer to perform loop closure and fuse lidar sweeps and IMU data into a single loop-closed 2D pose of the body (lidar in this case) with minimal drift.
In order to provide a quantitative measure of the quality of the final pose, we align the positions with the GPS measurements, and provide both overlaid on top of satellite imagery for each outdoor sequence in the dataset, as well as the difference in position between the provided ground truth and GPS. Fig 7 provides a sample overlay for Car Day 2, where the average error between Cartographer and the GPS is consistently around 5m without drift. This error is consistent amongst all of the outdoor driving sequences, where the overall average error is 4.7m, and is on a similar magnitude to the error expected from GPS. Note that the spike in error around 440 seconds is due to significant GPS error, and corresponds to the section in bold on the top right of the overlay.
In both cases, the extrinsic transform, represented as a homogenous transform matrix , for each sequence that takes a point from the left DAVIS frame to the body frame is then used to estimate the pose of the left DAVIS at time with respect to the first left DAVIS pose at time :
IV-B Depth Map Generation
In each sequence where lidar is available, depth images for each DAVIS camera are generated for every lidar measurement. We first generate a local map by transforming each lidar pointcloud in a local window around the current measurement into the frame of the current measurement using the poses from LOAM. At each measurement, the window size is determined such that the distances between the current, and the first and last LOAM poses in the window are at least meters, and that there are at least seconds between the current, and first and last LOAM poses, where and are parameters tuned for each sequence. Examples of these maps can be found in Fig 5.
We then project each point, , in the resulting pointcloud into the image in each DAVIS camera, using the standard pinhole projection equation:
and is the camera intrinsics matrix for the rectified image (i.e. the top left of the projection matrix).
Any points falling outside the image bounds are discarded, and the closest point at each pixel location in the image is used to generate the final depth map, examples of which can be found in Fig 6.
In addition, we also provide raw depth images without any undistortion by unrectifying and distorting the rectified depth images using the camera intrinsics and OpenCV.
V CALIBRATION
In this section, we describe the various steps performed to calibrate the intrinsic parameters of each DAVIS and VI-Sensor camera, as well as the extrinsic transformations between each of the cameras, IMUs and the lidar. All of the calibration results are provided in yaml form.
The camera intrinsics, stereo extrinsics, and camera-IMU extrinsics are calibrated using the Kalibr toolboxhttps://github.com/ethz-asl/kalibr, , , the extrinsics between the left DAVIS camera and Velodyne lidar are calibrated using the Camera and Range Calibration Toolboxhttp://www.cvlibs.net/software/calibration/, and fine tuned manually, and the hand eye calibration between the mocap model pose in the motion capture world frame and the left DAVIS camera pose is performed using CamOdoCalhttps://github.com/hengli/camodocal . To compensate for changes in the mounted rig, each calibration is repeated each day data was collected, and every time the sensing payload was modified. In addition to the calibration parameters, the raw calibration data for each day is also available on demand for users to perform their own calibration, if desired.
The camera intrinsics and extrinsics are estimated using a grid of AprilTags that is moved in front of the sensor rig and calibrated using Kalibr. Each calibration provides the focal length and principal point of each camera as well as the distortion parameters and the extrinsics between the cameras.
In addition, we calibrate the temporal offset between the DAVIS stereo pair and the VI Sensor by finding the temporal offset that maximizes the cross correlation between the magnitude of the gyroscope angular velocities from the IMUs of the left DAVIS and the VI Sensor. The timestamps for the VI Sensor messages in the dataset are then modified to compensate for this offset.
V-B Camera to IMU Extrinsic Calibration
To calibrate the transformation between the camera and IMU, a sequence is recorded with the sensor rig moving in front of the AprilTag grid. The two calibration procedures are separated to optimize the quality of each individual calibration. The calibration sequences are once again run through Kalibr using the camera-IMU calibration to estimate the transformations between each camera and each IMU, given the prior intrinsic and camera-camera extrinsic calibrations.
V-C Motion Capture to Camera Extrinsic Calibration
Each motion capture system provides the pose of the mocap model in the motion capture frame at 100Hz. However, the mocap model frame is not aligned with any camera frame, and so a further calibration is needed to obtain the pose of the cameras from the motion capture system.
The sensor rig was statically held in front of an Aprilgrid at a variety of different poses. At each pose at time , the pose of the left DAVIS camera frame in the grid frame , as well as the pose of the mocap model (denoted body) in the mocap frame , were measured. These poses were then used to solve the handeye calibration problem for the transform that transforms a point in the left DAVIS frame into the model frame :
The optimization is performed using CamOdoCal, using the linear method in , and refining using a nonlinear optimization as described in .
V-D Lidar to Camera Extrinsic Calibration
The transformation that takes a point from the lidar frame to the left DAVIS frame was initially calibrated using the Camera and Range Calibration Toolbox . Four large checkerboard patterns are placed to fill the field of view of the DAVIS cameras, and a single pair of images from each camera is recorded, along with a full lidar scan. The calibrator then estimates the translation and rotation that aligns the camera and lidar observations of the checkerboards.
However, we found that the reported transform had up to five pixels of error when viewing the projected depth images (Fig 6). In addition, as the lidar and cameras are not hardware time synchronized, there was occasionally a noticeable and constant time delay between the two sensors. To improve the calibration, we fixed the translation based on the values from the CAD models, and manually fine tuned the rotation and time offset in order to maximize the overlap between the depth and event images. For visual confirmation, we provide the depth images with events overlaid for each camera. The timestamps of the lidar messages provided in the dataset are compensated for the time offset.
VI KNOWN ISSUES
The mapping used to generate the depth maps assumes that scenes are static, and typically does not filter out points on moving objects. As a result, the reported depth maps may have errors of up to two meters when tracking points on other cars, etc. However, these objects are typically quite rare compared to the total amount of data available. If desired, future work could involve classifying vehicles in the images and omitting these points from the depth maps.
VI-B Clock Synchronization
The motion capture and GPS are only synchronized to the rest of the system using the host computer’s time. This may incur an offset between the reported timestamps and the actual measurement time. We record all measurements on one computer to minimize this effect. In addition, there may be some delay between a lidar point’s measurement and the timestamp of the message due to the spin rate of the lidar.
VI-C DVS Biasing
Default biases for each camera were used when generating each sequence. However, it has been noted that, for the indoor flying sequences, the ratio of positive to negative events is higher than usual (2.5-5x). At this point, we are unaware of what may have caused this imbalance, or whether tuning the biases would have balanced it. We note that the imbalance is particularly skewed over the speckled floor. We advise researchers using the polarities of the events to be aware of this imbalance when working with these sequences.
VII CONCLUSION
We present a novel dataset for stereo event cameras, on a number of different vehicles and in a number of different environments, with ground truth 6dof pose and depth images. We hope that this data can provide one standard on which new event based methods can be evaluated and compared.