Data Viewer

The viewer enables an easy way of inspecting captured data from the recorded files and can visualize it with detected objects from post processing, real time algorithms or manual labels. With real time rendering and instant time axis scrolling even large recordings of hundreds of gigabytes can be viewed easily on any of the common desktop operating systems. With the viewer being free for everyone, recorded and processed data can be shared with and viewed by anyone. On Windows and MacOS, the viewer is distributed through their respective app stores. Download links for the supported platforms:

Navigating within the viewer has been implemented in a similar way as in the vast majority of 3D games. The user can move about in 3D space by using the mouse for (appropriately constrained 2 DOF) rotational motion and WASD for translational motion, with the addition of using the scroll-wheel for up/down translations in the current viewer position. Given that this method is considered mainstream in today's times, we can argue that our viewer provides an experience that is as ergonomic and intuitive as it can possibly get for viewing 3D Data. Particularly users that are familiar with playing 3D games should immediately feel very comfortable using our viewer.

The viewer works with efficiently represented data in our own format. The are applications in the bundle that convert the data from it's raw recorded form into this format. The camera stream for visualization is stored as separate video file and can optionally be left out too.

The following video demonstrates the usage of the data viewer in a single uninterrupted screen capture that iterates through the various capabilities of the viewer.

Automatic Post Processing

Simultaneous Localization and Mapping

The first step in our automatic processing is to use Google Cartographer. We are running it with specific settings and  parameters for obtaining robust and accurate localization for our system. In our cases, Cartographer is using the Livox Horizon's range and IMU data and allows for the integration of a GNSS data stream. We are recommending the inclusion of a GNSS signal, as our experiments have shown it can add significant benefit when driving along the same route more than once. Since Cartographer is using the GNSS data for its global SLAM module, the accuracy of a smartphone's signal seems to be enough for obtaining good localization results. We are using the output data from the GPS2IP App, which our software bundle seamlessly integrates with the rest of the data and processing algorithms. Since Google Cartographer has a ROS interface, other sources can be easily be integrated too.

With the localization produced by Cartographer and a few more of our processing steps within this module, we are producing an accumulated static map of the driven route, which is exposed as one of the outputs of the automatic processing of the software bundle. You can inspect a small piece of such a map in the 3D window below.

Deep Learning

Our software bundle offers the option to automatically compute semantic segmentations and object instance segmentations using pre trained versions of the Pyramid Scene Parsing Network and Mask R-CNN deep neural network and fuses these with the pointcloud.

Pyramid Scene Parsing Network

The PSPNet is a deep neural network that performs semantic segmentation, which has become widely adopted in the industry. It produces a semantic label for every pixel in the input images. Since in our sensor setup the camera's field of view fully incorporates the lidar's field of view, by projecting the points onto the corresponding images every point in the lidar's data stream gets a semantic label from the PSPNet. Hence, the PSPNet module of our software bundle is labeling all lidar points according to the corresponding PSPNet segmentations in both the raw lidar pointcloud stream and the accumulated map from the SLAM module.

Example outputs from the PSPNet on the camera data as well as  projected in the lidar points and static map can be seen in the videos and 3D window below.

Mask R-CNN

The Mask R-CNN can be used for both object detection as well as object instance segmentation tasks. Object instance segmentations are essentially segmentations within the object's bounding box. When projecting lidar points into the Mask R-CNN processed camera frames, the instance masks provide a better compatibility and tend to produce substantially fewer false positives than using bounding boxes. Since Mask R-CNN has become mainstream in the industry, we have integrated it directly into our automatic processing. In our use case, we are labeling all lidar points according to the corresponding Mask R-CNN detections and exposing them to our software's outputs.

Example outputs from the Mask R-CNN on the camera data as well as  projected in the lidar points can be seen in the videos below.

Calibration

Our software stack includes a module that we use to calibrate our sensor setup. We provide it to our customers within the software bundle, which they can use to recalibrate our roof mounted sensor solution or to calibrate lidar, camera and inertial sensors from external suppliers to the roof mounted sensors. The calibration module includes a semi-supervised suite of calibration tools. This comprises libraries and executables to calibrate lidar to lidar, lidar to IMU, and lidar to camera. The first two are fully automated and the quality of the results is visually inspected. The end result can be visualized with Meshlab, RViz, or our own visualization tool. For the camera and lidar registration we use a supervised method that utilizes a custom graphical user interface (GUI).