Monocular distance inference solution that eliminates the need for LiDAR | NEXTY Electronics, an electronic components and semiconductor trading company

Overview

Distance inference technology requires high accuracy in fields such as autonomous driving, robotics, and AR/VR.
Conventional methods have had significant challenges. For example, LiDAR, the mainstream method for measuring distance, is very expensive and has strict restrictions on where it can be installed, making cost and environmental adaptability major issues when it comes to practical use. Furthermore, stereo cameras also require precise calibration between multiple cameras, and have been pointed out as being vulnerable to fluctuations in lighting conditions and other factors.

Against this background, monocular distance inference algorithms that utilize AI, such as MiDAS, have been attracting attention in recent years.However, even with monocular distance inference technology, improving the accuracy of distance inference has always been a major challenge, and it has been difficult to achieve sufficient accuracy in calculating absolute distance using conventional approaches.

To solve these problems, we focused on the monocular distance inference technology of the Toyota Research Institute (TRI) and aimed to create a system that uses a single USB camera, is inexpensive, and requires minimal installation space.
We independently created a dataset with absolute distances, and used this data for training and accuracy verification. As a result, we confirmed promising accuracy compared to conventional technologies. In particular, we succeeded in improving the accuracy of absolute distance inference, which was a weakness of conventional technologies, suggesting the possibility of practical use.

The content of this article is based on an AI model developed by TRI.
For more information, please see the link below: "Monocular Depth in the Real World."

Monocular Depth in the Real World

About the AI model used

This distance inference technology aims to achieve high-precision distance inference without using distance data from LiDAR or other sensors as training data.
(For more information on the technology, see our article Monocular Depth in the Real World.)

As a first step, we aimed to maximize inference accuracy by creating a unique "training data creation system" that combines LiDAR and a camera, and we carried out everything from training data creation to inference.

In addition, the output of distance inference for an input image is distance information (absolute distance) for each pixel.

Considering actual use cases, this distance information can be used to calculate the distance to an obstacle, etc.

We believe that calculating distance from "pixel" units to "area" units such as object distances will make it easier to actually use the information.
We developed this demo system with the aim of being able to present information on the distance to each object by combining it with object detection (detecting rectangular areas around objects).
In this example, we used Yolo v7 as the object detection model.

Procedures and Configuration

Steps

Using cameras and LiDAR, training data is created on-site
study
Distance Reasoning
Inference result

Input and output of each phase

Creating training data

Inputs
- Range data from LiDAR
- Image data from a camera
Outputs
- A dataset in which image data and LiDAR data are time-synchronized

study

Inputs
- A dataset in which image data and LiDAR data are time-synchronized
Outputs
- Learning Model

Distance Reasoning

Inputs
- Learning Model
- Distance data from LiDAR *For accuracy measurement
- Image data from a camera (or file)
Outputs
- Distance information for each pixel
- Distance information (absolute value) for each object

System for creating training data

Our training data creation system uses a combination of a monocular camera and LiDAR. Training data is created by matching the distance information acquired by the LiDAR with each pixel in the camera image. This method allows us to obtain accurate distance data for the position indicated by each pixel in the camera image.

Accurate calibration is crucial when integrating camera and LiDAR data. By utilizing commercially available calibration tools, we streamline the calibration process and obtain highly accurate results.

Image of the system for creating training data

We can select the LiDAR and camera depending on the use case, create the housing, calibrate the LiDAR and camera, and build a system for creating training data.

Inference processing flow

Example of inference results

Here we will introduce a typical example of an outdoor intersection, where the scene was photographed and data was created from a pedestrian bridge.

The bounding box is displayed in a different color every 10 meters.

Please note that the colors are not based on class (people, cars, etc.).

Inference accuracy

The accuracy of distance inference for the above results is as follows:

Metric	Value
Mean Error (ME)	0.03 m
Mean Absolute Error (MAE)	0.12 m
Root Mean Squared Error (RMSE)	0.31 m
Absolute Relative Difference (abs_rel)	0.01
Squared Relative Difference (sqr_rel)	0.00
RMSE Log (rmse_log)	0.02
Accuracy a1	99.88%
Accuracy a2	100.00%
Accuracy a3	100.00%

Explanation of each indicator

3. Image of Root Mean Squared Error (RMSE)

4. Image of Absolute Relative Difference (abs_rel)

5. Image of Squared Relative Difference (sqr_rel)

evaluation

The tolerance for each will vary depending on the criteria required for the final use case,

Mean Error (ME) = 0.03 m
Because the signed mean error is very small, it is believed that there is little overall bias towards overestimation or underestimation.
Mean Absolute Error (MAE) = 0.12 m
The absolute value of the mean error is 0.12 m.
Although it depends on the distance range and the usage situation, the absolute error is around 10cm.
Root Mean Squared Error (RMSE) = 0.31 m
Even the RMSE, which reflects larger errors more strongly, remains at 0.31 m.
Naturally, it is slightly larger than MAE, but this value is also small.
It can be assumed that extremely large errors (outliers) do not occur very often.
Absolute Relative Difference (abs_rel) = 0.01
An average relative error of 1% indicates very high accuracy.
For example, if the actual distance is 10 m, the error is calculated to be about 0.1 m on average.
Squared Relative Difference (sqr_rel) = 0.00
A squared relative error of almost 0 means that the error is extremely small even at large distances.
When combined with abs_rel, it is believed that inference accuracy is high regardless of the distance.
RMSE Log (rmse_log) = 0.02
The small error even on the log scale suggests that it is relatively stable over short and long distances.
Since distance estimation tends to have larger errors the further away the object is, a small value is a desirable trend.
Accuracy a1 = 99.88%, a2 = 100.00%, a3 = 100.00%
The fact that it reaches approximately 99.88% at a1 (within 1.25 times) is evidence of extremely high accuracy.
Since a2 and a3 recorded 100%, it can be interpreted that all pixels are within the "error tolerance range" under somewhat looser standards.

Inference speed

Edge compatibility study

Considering actual use cases, it is not always possible to place a desktop PC on-site,
It is expected to be used as an edge device due to its price.

So we ran the same app on the following devices and measured the speed:

Jetson AGX Orin
Jetson Orin NX
Jetson Orin Nano

We also conducted measurements on a model that had been converted to TensorRT, a representative model that could be used to speed up the process.

Measurement Model
- Without TensorRT (FP32)
- With TensorRT (FP16)
Measurement conditions
- Uses a USB camera (resolution 640x480, 30 FPS)
- Measure FPS from image loading to rendering
- FRAMES rate varies depending on the number of objects detected by object detection, so it is fixed at 0.

Measurement results

(Unit: FPS)
	With TensorRT (FP16)	Without TensorRT (FP32)
Jetson AGX Orin	AVG: 15.0 (MAX: 21.1)	AVG: 7.5 (MAX: 7.9)
Jetson Orin NX	AVG: 8.7 (MAX: 9.1)	AVG: 3.6 (MAX: 3.7)
Jetson Orin Nano	AVG: 6.3 (MAX: 6.4)	AVG: 2.5 (MAX: 2.5)

By the way, the desktop PC used for learning (GPU: NVIDIA RTX6000 Ada)
So, the FP32 model runs at around 50 FPS.

Accuracy evaluation of model weight reduction

In this test, we describe the results of evaluating the degree of degradation in inference accuracy by changing the data precision of a distance inference model using Jetson AGX Orin.

Accuracy of the model used
- FP32
- TensorRT(FP16)
- TensorRT(INT8)
Regarding the data
Distance inference was performed on pre-photographed office scenes at each data accuracy level.
(The footage is not posted for privacy reasons.)

FP32

Metric	Value
Mean Error (ME)	-0.05 m
Mean Absolute Error (MAE)	0.20 m
Root Mean Squared Error (RMSE)	0.69 m
Absolute Relative Difference (abs_rel)	0.05
Squared Relative Difference (sqr_rel)	0.06
RMSE Log (rmse_log)	0.13
Accuracy a1	95.97%
Accuracy a2	98.19%
Accuracy a3	99.10%

TensorRT(FP16)

Metric	Value
Mean Error (ME)	-0.25 m
Mean Absolute Error (MAE)	0.42 m
Root Mean Squared Error (RMSE)	0.98 m
Absolute Relative Difference (abs_rel)	0.13
Squared Relative Difference (sqr_rel)	0.17
RMSE Log (rmse_log)	0.21
Accuracy a1	89.64%
Accuracy a2	96.46%
Accuracy a3	98.14%

TensorRT(INT8)

Metric	Value
Mean Error (ME)	-0.89 m
Mean Absolute Error (MAE)	1.25 m
Root Mean Squared Error (RMSE)	3.70 m
Absolute Relative Difference (abs_rel)	0.26
Squared Relative Difference (sqr_rel)	0.79
RMSE Log (rmse_log)	0.43
Accuracy a1	70.45%
Accuracy a2	87.59%
Accuracy a3	94.10%

Consideration

Mean Error (ME): -0.05 m for FP32 vs. -0.25 m for FP16, a large offset
Mean Absolute Error (MAE): 0.20 m for FP32 and 0.42 m for FP16, about twice the error.
abs_rel: FP32 increased to 0.05, and FP16 increased to 0.13.
Accuracy a1: FP32 95.97% vs. FP16 89.64%
The error may be due to the decrease in numerical accuracy associated with the FP16 conversion and the optimization process by TensorRT. However, metrics such as a2 and a3 are all above 96%, maintaining a fairly high level of accuracy in absolute terms.

Ideally, it would be desirable to compare speed and accuracy for the same data, but
Please note that these are individual comparisons at this time.

conclusion

Based on TRI's monocular distance inference technology, we have created a unique dataset with absolute distances, performed training on it, and by combining it with object detection technology, developed an application that can infer the absolute distance to a specific object with high accuracy.

Furthermore, the company has also succeeded in running the same application not only on desktop PCs but also on edge devices such as Jetson NX, making it a promising, practical solution that is inexpensive and has fewer restrictions on installation location.

The degree of error will vary depending on the conditions of the usage scenario.

Finally

If you are interested in learning more about this monocular distance inference technology or have ideas for applications,
Please feel free Inquiry using the Inquiry button on this page!