Get BEVFusion up and running

Introduction

Preparation (Building a Docker environment)

As a preliminary step, please install docker (with GPU environment).

Virtual environment with Docker

One of the reasons why it is difficult to run CUDA-BEVFusion is that it was released a few years ago, so if you try to run it now, you are a little late and are likely to encounter inconsistencies with other libraries that have been updated.
Therefore, it is necessary to prepare a "slightly older environment," and a virtual environment using Docker is useful in this case.

Docker image for CUDA-BEVFusion

Save the following text as "Dockerfile".
(It's long, but you can copy it to your clipboard using the copy button at the bottom right, so you can easily copy and paste it again without having to worry about prompts.)

The reason why the Dockerfile is like this is difficult to explain in detail as it is the culmination of blood, sweat, and tears of research and consideration, but we will provide a brief explanation below.

• I will also be running the bevfusion folder (which is like the original) later, so I have set up an older environment (cuda11.3) with that in mind.
• As mentioned in the issue, it seems that correct inference cannot be performed unless the version of tensorrt is old, so I installed it specifying the version.
• For OTHERS libraries, there are discrepancies between the notation used at the time and the current notation, so older ones are often specified.

Generate an image with the following command. Anything can be used for the -t part, but give the created image a nickname.
(In this article, we will use "cubev-nexty" as an example.)

Preparing the files

Get the repository

Create a working directory and get the repository under it.
(The working directory can be anywhere. In this article, we use /home/xxx/bev/.)

Get it using the git command. Since there is a reference type, add the recursive option.

Since recent versions may not work properly, we will revert to a version from around early 2024.

Let's run the sample

From here, we will follow the "Quick Start for Inference" in the CUDA-BEVFusion README and perform inference on the provided sample data (one set of camera + Lidar).

• "Quick Start for Inference" in the README

CUDA-BEVFusion sample data and model data

This is the procedure in the README section "1. Download models and data to CUDA-BEVFusion directory."
First, download the following two files from the links above:

• "1.Download models and datas to CUDA-BEVFusion directory"

• nuScenes-example_data.zip
• model.zip

Extract these to the specified directory: Lidar_AI_Solution/CUDA-BEVFusion/.

The directory structure will be as follows:

Download the nuScenes dataset

The video above uses the "v1.0 mini" set of the nuScenes dataset for autonomous driving.

• nuScenes

So, download the following from nuScenes Downloads.
(You must register for an account on nuscenes.org to download.)

• Download nuScenes

• Full Dataset (V1.0), mini (3.88GB)

v1.0-mini.tar

• Map expansion pack (v1.3) of Map Expansion

nuscenes-map-expansion-v1.3.zip

Deploying the nuScenes dataset

Since this is data preparation, the following will be done outside of Docker. Create a data storage location under Lidar_AI_Solution/CUDA-BEVFusion/.

Extract the data from nuscenes v1.0-mini.tar.

Extract nuscenes-map-expansion-v1.3.zip to data/nuscenes/maps.

After decompression, the file will look like this:

Create a symbolic link to this data/ directory so that it can also be referenced from the bevfusion/data/ directory.

The final folder structure will look like this:
(Because it is Access as data/ no matter where you look)

Preparation on the bevfusion (main) side

As a final preparation, copy configs/ to bevfusion/ in the Lidar_AI_Solution/CUDA-BEVFusion/ directory. This will also be done outside of Docker.
(The purpose is to copy the resnet50/ folder located at the end of a deep directory with the same name.)

This completes the file setup.

From now on, you will be working within Docker. Under the Lidar_AI_Solution/CUDA-BEVFusion/ directory, there is a directory called bevfusion, which is the original repository clone.
We will use the original tools, such as nuScenes data preparation, so we will move the directory within Docker.

Run the compilation of bevfusion.

Below is an example output. If the output at the end is similar, the compilation was successful.

Use create_data.py from the bevfusion (original) side to convert nuScenes into a format that bevfusion can use.
In this case, we are using v1.0-mini, so we will run the following script.

Here is an example of the output when run:

As a result, if the contents under data/nuscenes look like the following, then it's OK.

This completes the preparations on the original bevfusion side.

Inference on nuScenes mini with CUDA-BEVFusion

Within Docker, change the directory back to the original Lidar_AI_Solution/CUDA-BEVFusion/ directory.
Now, let's run the tool called dump-data.py as follows:

This is a tool that converts the data/nuscenes we prepared earlier into the format used by the single-image example.

An example of execution is as follows. It stops with an error on the 81st page, but it is normal (this is because the terminal implementation of this script is rough. It can be fixed, but for now it is as it is...)

As a result of executing this, 81 sets of folders, numbered 00000 to 00080, will be created under dump/.

Next, we will modify run.sh, which output the single sample image we created earlier, and create a script that performs inference on the 81 sets of data under dump/.
Change the last part of run.sh (the line ./build/bevfusion ...) to the following and save it with a different name, such as tools/run_nexty.sh.

When you run the created tool/run_nexty.sh, sequentially numbered jpg files will be output under out_dump/.

An example of the output of the first image (00000.jpg) is shown below.

The video from the sequential jpg files is created outside of Docker using ffmpeg etc. Please see the video results for yourself!

Preparation on the bevfusion (main) side

As a final preparation, copy configs/ to bevfusion/ in the Lidar_AI_Solution/CUDA-BEVFusion/ directory. This will also be done outside of Docker.
(The purpose is to copy the resnet50/ folder located at the end of a deep directory with the same name.)

This completes the file setup.

From now on, you will be working within Docker. Under the Lidar_AI_Solution/CUDA-BEVFusion/ directory, there is a directory called bevfusion, which is the original repository clone.
We will use the original tools, such as nuScenes data preparation, so we will move the directory within Docker.

Run the compilation of bevfusion.

Below is an example output. If the output at the end is similar, the compilation was successful.

Use create_data.py from the bevfusion (original) side to convert nuScenes into a format that bevfusion can use.
In this case, we are using v1.0-mini, so we will run the following script.

Here is an example of the output when run:

As a result, if the contents under data/nuscenes look like the following, then it's OK.

This completes the preparations on the original bevfusion side.

Inference on nuScenes mini with CUDA-BEVFusion

Within Docker, change the directory back to the original Lidar_AI_Solution/CUDA-BEVFusion/ directory.
Now, let's run the tool called dump-data.py as follows:

This is a tool that converts the data/nuscenes we prepared earlier into the format used by the single-image example.

An example of execution is as follows. It stops with an error on the 81st page, but it is normal (this is because the terminal implementation of this script is rough. It can be fixed, but for now it is as it is...)

As a result of executing this, 81 sets of folders, numbered 00000 to 00080, will be created under dump/.

Next, we will modify run.sh, which output the single sample image we created earlier, and create a script that performs inference on the 81 sets of data under dump/.
Change the last part of run.sh (the line ./build/bevfusion ...) to the following and save it with a different name, such as tools/run_nexty.sh.

When you run the created tool/run_nexty.sh, sequentially numbered jpg files will be output under out_dump/.

An example of the output of the first image (00000.jpg) is shown below.

The video from the sequential jpg files is created outside of Docker using ffmpeg etc. Please see the video results for yourself!

Preparation on the bevfusion (main) side

As a final preparation, copy configs/ to bevfusion/ in the Lidar_AI_Solution/CUDA-BEVFusion/ directory. This will also be done outside of Docker.
(The purpose is to copy the resnet50/ folder located at the end of a deep directory with the same name.)

This completes the file setup.

From now on, you will be working within Docker. Under the Lidar_AI_Solution/CUDA-BEVFusion/ directory, there is a directory called bevfusion, which is the original repository clone.
We will use the original tools, such as nuScenes data preparation, so we will move the directory within Docker.

Run the compilation of bevfusion.

Below is an example output. If the output at the end is similar, the compilation was successful.

Use create_data.py from the bevfusion (original) side to convert nuScenes into a format that bevfusion can use.
In this case, we are using v1.0-mini, so we will run the following script.

Here is an example of the output when run:

As a result, if the contents under data/nuscenes look like the following, then it's OK.

This completes the preparations on the original bevfusion side.

Inference on nuScenes mini with CUDA-BEVFusion

Within Docker, change the directory back to the original Lidar_AI_Solution/CUDA-BEVFusion/ directory.
Now, let's run the tool called dump-data.py as follows:

This is a tool that converts the data/nuscenes we prepared earlier into the format used by the single-image example.

An example of execution is as follows. It stops with an error on the 81st page, but it is normal (this is because the terminal implementation of this script is rough. It can be fixed, but for now it is as it is...)

As a result of executing this, 81 sets of folders, numbered 00000 to 00080, will be created under dump/.

Next, we will modify run.sh, which output the single sample image we created earlier, and create a script that performs inference on the 81 sets of data under dump/.
Change the last part of run.sh (the line ./build/bevfusion ...) to the following and save it with a different name, such as tools/run_nexty.sh.

When you run the created tool/run_nexty.sh, sequentially numbered jpg files will be output under out_dump/.

An example of the output of the first image (00000.jpg) is shown below.

The video from the sequential jpg files is created outside of Docker using ffmpeg etc. Please see the video results for yourself!