Introduction
Hello, this is a member of the Development Department at Nexty Electronics.
I am in charge of technical support for the GPU Advanced Test Drive (GAT), which allows for highly flexible trials of various GPU servers.
In recent years, in order to realize autonomous driving, it has become important to build a system that combines multiple sensors such as not only cameras but also Lidar and millimeter waves to detect and recognize the surroundings of the vehicle, and one such system that is attracting attention is BEVFusion, which performs sensor fusion in a BEV (bird's-eye view).We will not go into technical details here.
It is open source and anyone with a PC and GPU can try it out, but when you actually try to get it to work, there are some difficulties even just getting it to work.
In this column, I will explain how to actually run CUDA-BEVFusion, which is a TensorRT implementation of BEVFusion.
Required Hardware
PC + GPU
- In order to run CUDA, the GPU must be manufactured by NVIDIA.
- It has been proven to work with RTX3060 etc.
- The OS is based on Ubuntu, but it can run on Ubuntu with Windows WSL2.
Preparation (Building a Docker environment)
As a preliminary step, please install docker (with GPU environment).
1. For Ubuntu
- We recommend the official apt installation instructions.
- After that, you need to install nvidia-container-toolkit (which will allow you to use the GPU from Docker)
2. For Windows
- Please install docker-desktop (Windows version)
(If you enable "WSL2 Integration" during installation, you can use it from Ubuntu on WSL.) - On Windows, no additional installation of nvidia-container-toolkit is required.
- Please make it possible to use ubuntu20.04 or ubuntu22.04 with wsl2.
Virtual environment with Docker
One of the reasons why it is difficult to run CUDA-BEVFusion is that it was released a few years ago, so if you try to run it now, you are a little late and are likely to encounter inconsistencies with other libraries that have been updated.
Therefore, it is necessary to prepare a "slightly older environment," and a virtual environment using Docker is useful in this case.
Docker image for CUDA-BEVFusion
Save the following text as "Dockerfile".
(It's long, but you can copy it to your clipboard using the copy button at the bottom right, so you can easily copy and paste it again without having to worry about prompts.)
| Dockerfile FROM nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04 # Environment settings ENV DEBIAN_FRONTEND noninteractive ENV CONDA_DIR /opt/conda ENV PATH /usr/local/cuda/bin:$CONDA_DIR/bin:$PATH ENV FORCE_CUDA="1" ENV MMCV_WITH_OPS=1 # System dependencies RUN apt-get update && apt-get install -y \ wget \ build-essential g++ gcc \ libgl1-mesa-glx libglib2.0-0 \ openmpi-bin openmpi-common libopenmpi-dev libgtk2.0-dev git \ gnupg2 curl software-properties-common \ cmake\ protobuf-compiler\ libprotobuf-dev\ && rm -rf /var/lib/apt/lists/* # Add NVIDIA package repositories and install TensorRT RUN curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub | apt-key add - && \ curl -fsSL https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub | apt-key add - && \ echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" > /etc/apt/sources.list.d/cuda.list && \ echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/ /" > /etc/apt/sources.list.d/nvidia-ml.list && \ apt-get update && \ apt-get install -y --allow-downgrades cuda-toolkit-config-common=11.3.58-1 && \ apt-get install -y tensorrt=8.5.2.2-1+cuda11.8 \ libnvinfer8=8.5.2-1+cuda11.8 \ libnvinfer-plugin8=8.5.2-1+cuda11.8 \ libnvparsers8=8.5.2-1+cuda11.8 \ libnvonnxparsers8=8.5.2-1+cuda11.8 \ libnvinfer-bin=8.5.2-1+cuda11.8 \ libnvinfer-dev=8.5.2-1+cuda11.8 \ libnvinfer-plugin-dev=8.5.2-1+cuda11.8 \ libnvparsers-dev=8.5.2-1+cuda11.8 \ libnvonnxparsers-dev=8.5.2-1+cuda11.8 \ libnvinfer-samples=8.5.2-1+cuda11.8 && \ apt-get install -y python3-libnvinfer=8.5.2-1+cuda11.8 python3-libnvinfer-dev=8.5.2-1+cuda11.8 \ && rm -rf /var/lib/apt/lists/* # Install Miniconda RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \ /bin/bash ~/miniconda.sh -b -p $CONDA_DIR # Python packages RUN conda install -y python=3.8 pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch && \ pip install Pillow==8.4.0 tqdm torchpack mmcv==1.4.0 mmcv-full==1.4.0 mmdet==2.20.0 nuscenes-devkit mpi4py==3.0.3 numba==0.48.0 numpy==1.23.0 #Enviroment(for bevfusion) ENV CPATH /usr/local/cuda-11.3/include:$CPATH ENV LIBRARY_PATH /usr/local/cuda-11.3/lib64:$LIBRARY_PATH ENV LD_LIBRARY_PATH /usr/local/cuda-11.3/lib64:$LD_LIBRARY_PATH ENV PATH /usr/local/cuda-11.3/bin:$PATH ENV CUDA_HOME /usr/local/cuda-11.3 FROM nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04 # Environment settings ENV DEBIAN_FRONTEND noninteractive ENV CONDA_DIR /opt/conda ENV PATH /usr/local/cuda/bin:$CONDA_DIR/bin:$PATH # System dependencies RUN apt-get update && apt-get install -y \ wget \ build-essential g++ gcc \ libgl1-mesa-glx libglib2.0-0 \ openmpi-bin openmpi-common libopenmpi-dev libgtk2.0-dev git \ gnupg2 curl software-properties-common \ cmake\ libprotobuf-dev\ protobuf-compiler\ && rm -rf /var/lib/apt/lists/* # Add NVIDIA package repositories and install TensorRT RUN curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub | apt-key add - && \ curl -fsSL https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub | apt-key add - && \ echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" > /etc/apt/sources.list.d/cuda.list && \ echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/ /" > /etc/apt/sources.list.d/nvidia-ml.list && \ apt-get update && \ apt-get install -y --allow-downgrades cuda-toolkit-config-common=11.3.58-1 && \ apt-get install -y tensorrt=8.5.2.2-1+cuda11.8 \ libnvinfer8=8.5.2-1+cuda11.8 \ libnvinfer-plugin8=8.5.2-1+cuda11.8 \ libnvparsers8=8.5.2-1+cuda11.8 \ libnvonnxparsers8=8.5.2-1+cuda11.8 \ libnvinfer-bin=8.5.2-1+cuda11.8 \ libnvinfer-dev=8.5.2-1+cuda11.8 \ libnvinfer-plugin-dev=8.5.2-1+cuda11.8 \ libnvparsers-dev=8.5.2-1+cuda11.8 \ libnvonnxparsers-dev=8.5.2-1+cuda11.8 \ libnvinfer-samples=8.5.2-1+cuda11.8 && \ apt-get install -y python3-libnvinfer=8.5.2-1+cuda11.8 python3-libnvinfer-dev=8.5.2-1+cuda11.8 \ && rm -rf /var/lib/apt/lists/* # Install Miniconda RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \ /bin/bash ~/miniconda.sh -b -p $CONDA_DIR # Python packages RUN conda install -y python=3.8 pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch && \ pip install Pillow==8.4.0 tqdm torchpack nuscenes-devkit mpi4py==3.0.3 numba==0.48.0 numpy==1.23.0 #Enviroment(for bevfusion) ENV CPATH /usr/local/cuda-11.3/include:$CPATH ENV LIBRARY_PATH /usr/local/cuda-11.3/lib64:$LIBRARY_PATH ENV LD_LIBRARY_PATH /usr/local/cuda-11.3/lib64:$LD_LIBRARY_PATH ENV PATH /usr/local/cuda-11.3/bin:$PATH ENV CUDA_HOME /usr/local/cuda-11.3 #mmcv install RUN pip install --no-cache-dir mmcv==1.4.0 mmcv-full==1.4.0 mmdet==2.20.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10/index.html |
|---|
The reason why the Dockerfile is like this is difficult to explain in detail as it is the culmination of blood, sweat, and tears of research and consideration, but we will provide a brief explanation below.
- I will also be running the bevfusion folder (which is like the original) later, so I have set up an older environment (cuda11.3) with that in mind.
- As mentioned in the issue, it seems that correct inference cannot be performed unless the version of tensorrt is old, so I installed it specifying the version.
- For OTHERS libraries, there are discrepancies between the notation used at the time and the current notation, so older ones are often specified.
Generate an image with the following command. Anything can be used for the -t part, but give the created image a nickname.
(In this article, we will use "cubev-nexty" as an example.)
| docker build . -f Dockerfile -t cubev-nexty |
|---|
Preparing the files
Get the repository
Create a working directory and get the repository under it.
(The working directory can be anywhere. In this article, we use /home/xxx/bev/.)
| cd /home/xxx/ mkdir bev cd /bev |
|---|
Get it using the git command. Since there is a reference type, add the recursive option.
| git clone --recursive https://github.com/NVIDIA-AI-IOT/Lidar_AI_Solution |
|---|
Since recent versions may not work properly, we will revert to a version from around early 2024.
| cd Lidar_AI_Solution git checkout 8cd961f |
|---|
Let's run the sample
From here, we will follow the "Quick Start for Inference" in the CUDA-BEVFusion README and perform inference on the provided sample data (one set of camera + Lidar).
CUDA-BEVFusion sample data and model data
This is the procedure in the README section "1. Download models and data to CUDA-BEVFusion directory."
First, download the following two files from the links above:
- nuScenes-example_data.zip
- model.zip
Extract these to the specified directory: Lidar_AI_Solution/CUDA-BEVFusion/.
| cd Lidar_AI_Solution/CUDA-BEVFusion/ unzip hogehoge/nuScenes-example-data.zip unzip hogehoge/models.zip |
|---|
The directory structure will be as follows:
| Lidar_AI_Solution/CUDA-BEVFusion/ |-- example-data/ |-- model/ |
|---|
Compiling and running (in Docker)
From here, we will follow the steps from "2. Configure the environment.sh" to "5. Compile and run" in the README (for some reason there are some missing numbers).
Up until this point, we've simply been preparing files, so a local environment would have been fine, but from here on, we'll be working in a Docker virtual environment, as library environments and the like will be involved. So, we'll start the Docker image prepared above. This time, we'll do so using the following script:
| #!/bin/bash #edit according to your environment D_WORK_DIR=/home/xxx/bev/ #run docker run --name cbev --gpus all -it --network host -w "${D_WORK_DIR}" -v "${D_WORK_DIR}":"${D_WORK_DIR}" --shm-size 16g cubev-nexty /bin/bash |
|---|
- Please edit the working directory "D_WORK_DIR" to suit your environment. This is the initial directory at startup (-w) and the directory shared between Docker and outside (-v).
- --name cbev gives a nickname to the launched docker.
- --gpus all allows you to use all GPUs from within Docker.
- --shm-size 16g determines the amount of memory used by Docker. Adjust it to your environment.
- cubev-nexty is the name of the image generated from the Dockerfile. Please modify it if you have changed it.
If Docker is successfully started using the above startup script, a bash with administrator privileges (root) will be launched within the Docker virtual environment as shown below.
| root:/home/xxx/bev/ # |
|---|
In Docker, change the directory (this will be the base from now on)
| cd Lidar_AI_Solution/CUDA-BEVFusion/ |
|---|
From here, we will move on to the "2. Configure the environment.sh" section of the README. This will set the environment variables. Let's try running the following.
| bash tool/enviroment.sh |
|---|
I think the output will be something like this:
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion# bash tool/environment.sh ========================================================== || MODEL: resnet50int8 || PRECISION: int8 || DATA: example-data USEPython: OFF || || TensorRT: /path/to/TensorRT/lib || CUDA: /usr/local/cuda || CUDNN: /path/to/cudnn/lib ========================================================== Try to get the current device SM Current CUDA SM: 89 Configuration done! |
|---|
The CUDA_SM will be automatically calculated and displayed based on your GPU. Please note the following:
- If the "Current CUDA SM:" field (which varies depending on the GPU generation) is between 80 and 86, there is no problem, but if it is larger, such as 89, it will not work properly (with the version we are using this time), so you will need to overwrite it and lower it to 86.
Edit tool/environment.sh as follows:
| #export CUDASM=$cudasm export CUDASM=86 |
|---|
In this case, depending on your environment, it may be necessary to downgrade the NVIDIA driver.
- Model/precision etc. can be changed by editing tool/environment.sh.
Below is an example of changing to swint/fp16, which seems to have the best performance.
| # resnet50/resnet50int8/swint #export DEBUG_MODEL=resnet50int8 export DEBUG_MODEL=swint # fp16/int8 #export DEBUG_PRECISION=int8 export DEBUG_PRECISION=fp16 |
|---|
The path changes in environment.sh in the README are handled by Docker environment variables, so it should not be necessary this time, but please let us know if you encounter any problems.
Next, proceed to "5. Compile and run" in the README. First, "Convert the model to a TRT engine."
| bash tool/build_trt_engine.sh |
|---|
The following message will appear:
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion# bash tool/build_trt_engine.sh ========================================================== || MODEL: resnet50int8 || PRECISION: int8 || DATA: example-data USEPython: OFF || || TensorRT: /path/to/TensorRT/lib || CUDA: /usr/local/cuda || CUDNN: /path/to/cudnn/lib ========================================================== Try to get the current device SM Current CUDA SM: 89 Configuration done! Building the model: model/resnet50int8/build/camera.backbone.plan, this will take several minutes. Wait a moment ???~. Building the model: model/resnet50int8/build/fuser.plan, this will take several minutes. Wait a moment ???~. Building the model: model/resnet50int8/build/camera.vtransform.plan, this will take several minutes. Wait a moment ???~. Building the model: model/resnet50int8/build/head.bbox.plan, this will take several minutes. Wait a moment ???~. |
|---|
Next, "Compile & Infer Sample" is performed.
| bash tool/run.sh |
|---|
When you run this, an image called build/cuda-bevfusion.jpg will be generated. I've pasted it below, so please check if a similar image has been generated. With this startup method, you can view it from outside Docker, so please check it with any image viewer.
Make a video with nuScenes
We have now performed inference on CUDA-BEVFusion, but you may be wondering, "Huh? Only one image?" or "What about the flashy video in the Demonstration section?" (I was wondering too).
In fact, there are Python scripts and other tools for creating these, but since there is no explanation about them, it is usually difficult to understand.
So, from here on, I would like to explain the steps taken by the NEXTY Development Department to create a flashy Demonstration video.
Download the nuScenes dataset
The video above uses the "v1.0 mini" set of the nuScenes dataset for autonomous driving.
So, download the following from nuScenes Downloads.
(You must register for an account on nuscenes.org to download.)
- Full Dataset (V1.0), mini (3.88GB)
v1.0-mini.tar
- Map expansion pack (v1.3) of Map Expansion
nuscenes-map-expansion-v1.3.zip
Deploying the nuScenes dataset
Since this is data preparation, the following will be done outside of Docker. Create a data storage location under Lidar_AI_Solution/CUDA-BEVFusion/.
| cd Lidar_AI_Solution/CUDA-BEVFusion/ mkdir data mkdir data/nuscenes |
|---|
Extract the data from nuscenes v1.0-mini.tar.
| tar xvf v1.0-mini.tar -C data/nuscenes |
|---|
Extract nuscenes-map-expansion-v1.3.zip to data/nuscenes/maps.
| unzip -d data/nuscenes/maps/ nuscenes-map-expansion-v1.3.zip |
|---|
After decompression, the file will look like this:
| Lidar_AI_Solution/CUDA-BEVFusion/data/nuscenes/ |-- samples/ |-- sweeps/ |-- v1.0-mini/ |-- maps/ |-- basemap/ |-- expansion/ |-- prediction/ |-- xxxxxx.png 4 images |
|---|
Create a symbolic link to this data/ directory so that it can also be referenced from the bevfusion/data/ directory.
| cd bevfusion ln -s ../data/ . |
|---|
The final folder structure will look like this:
(Because it is Access as data/ no matter where you look)
| Lidar_AI_Solution/CUDA-BEVFusion/ |-- bevfusion/data/ (link) |-- data/ (real) |
|---|
Preparation on the bevfusion (main) side
As a final preparation, copy configs/ to bevfusion/ in the Lidar_AI_Solution/CUDA-BEVFusion/ directory. This will also be done outside of Docker.
(The purpose is to copy the resnet50/ folder located at the end of a deep directory with the same name.)
| cp -r configs/ bevfusion/ |
|---|
This completes the file setup.
From now on, you will be working within Docker. Under the Lidar_AI_Solution/CUDA-BEVFusion/ directory, there is a directory called bevfusion, which is the original repository clone.
We will use the original tools, such as nuScenes data preparation, so we will move the directory within Docker.
| cd bevfusion |
|---|
Run the compilation of bevfusion.
| python setup.py develop |
|---|
Below is an example output. If the output at the end is similar, the compilation was successful.
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion#python setup.py develop ...(snip)... Installed /home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion Processing dependencies for mmdet3d==0.0.0 Finished processing dependencies for mmdet3d==0.0.0 |
|---|
Use create_data.py from the bevfusion (original) side to convert nuScenes into a format that bevfusion can use.
In this case, we are using v1.0-mini, so we will run the following script.
| python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --version v1.0-mini --extra-tag nuscenes |
|---|
Here is an example of the output when run:
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion# python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --version v1.0-mini --extra-tag nuscenes ====== Loading NuScenes tables for version v1.0-mini... 23 categories, 8 attribute, 4 visibility, 911 instance, 12 sensors, 120 calibrated_sensor, 31206 ego_pose, 8 log, 10 scenes, 404 sample, 31206 sample_data, 18538 sample_annotation, 4 maps, Done loading in 0.313 seconds. ====== Reverse indexing... Done reverse indexing in 0.1 seconds. |
|---|
As a result, if the contents under data/nuscenes look like the following, then it's OK.
| data/nuscenes/ |-- maps/ |-- samples/ |-- sweeps/ |-- v1.0-mini/ |-- nuscenes_gt_database/ ←Add |-- nuscenes_dbinfos_train.pkl ←Add |-- nuscenes_infos_train.pkl ←Add |-- nuscenes_infos_val.pkl ←Add |
|---|
This completes the preparations on the original bevfusion side.
Inference on nuScenes mini with CUDA-BEVFusion
Within Docker, change the directory back to the original Lidar_AI_Solution/CUDA-BEVFusion/ directory.
Now, let's run the tool called dump-data.py as follows:
| python tool/dump-data.py |
|---|
This is a tool that converts the data/nuscenes we prepared earlier into the format used by the single-image example.
An example of execution is as follows. It stops with an error on the 81st page, but it is normal (this is because the terminal implementation of this script is rough. It can be fixed, but for now it is as it is...)
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion# python tool/dump-data.py Save tensor[(1, 6, 3, 256, 704), float16] to dump/00000/images.tensor Save tensor[(207998, 5), float16] to dump/00000/points.tensor Save tensor[(23, 9), float32] to dump/00000/gt_bboxes_3d.tensor Save tensor[(23,), int64] to dump/00000/gt_labels_3d.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00000/camera_intrinsics.tensor ... (snip) ... Save tensor[(1, 4, 4), float32] to dump/00079/lidar_aug_matrix.tensor Save tensor[(1, 6, 3, 256, 704), float16] to dump/00080/images.tensor Save tensor[(235496, 5), float16] to dump/00080/points.tensor Save tensor[(37, 9), float32] to dump/00080/gt_bboxes_3d.tensor Save tensor[(37,), int64] to dump/00080/gt_labels_3d.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera_intrinsics.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera2ego.tensor Save tensor[(1, 4, 4), float32] to dump/00080/lidar2ego.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/lidar2camera.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera2lidar.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/lidar2image.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/img_aug_matrix.tensor Save tensor[(1, 4, 4), float32] to dump/00080/lidar_aug_matrix.tensor Traceback (most recent call last): File "tool/dump-data.py", line 97, in dump_tensor(args) File "tool/dump-data.py", line 61, in dump_tensor data = next(data_iter) File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__ data = self._next_data() File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1176, in _next_data raise StopIteration StopIteration |
|---|
As a result of executing this, 81 sets of folders, numbered 00000 to 00080, will be created under dump/.
Next, we will modify run.sh, which output the single sample image we created earlier, and create a script that performs inference on the 81 sets of data under dump/.
Change the last part of run.sh (the line ./build/bevfusion ...) to the following and save it with a different name, such as tools/run_nexty.sh.
| #Comment out this original line and change it to the one below (continuously executes the dump below) #./build/bevfusion $DEBUG_DATA $DEBUG_MODEL $DEBUG_PRECISION #Create the final output destination mkdir -p out_dump #dump/The following 81 loops for i in {0..80} ; do TMP="00000${i}" ./build/bevfusion dump/${TMP: -5} $DEBUG_MODEL $DEBUG_PRECISION mv build/cuda-bevfusion.jpg out_dump/${TMP: -5}.jpg done Copy |
|---|
When you run the created tool/run_nexty.sh, sequentially numbered jpg files will be output under out_dump/.
| bash tool/run_nexty.sh |
|---|
An example of the output of the first image (00000.jpg) is shown below.
The video from the sequential jpg files is created outside of Docker using ffmpeg etc. Please see the video results for yourself!
Preparation on the bevfusion (main) side
As a final preparation, copy configs/ to bevfusion/ in the Lidar_AI_Solution/CUDA-BEVFusion/ directory. This will also be done outside of Docker.
(The purpose is to copy the resnet50/ folder located at the end of a deep directory with the same name.)
| cp -r configs/ bevfusion/ |
|---|
This completes the file setup.
From now on, you will be working within Docker. Under the Lidar_AI_Solution/CUDA-BEVFusion/ directory, there is a directory called bevfusion, which is the original repository clone.
We will use the original tools, such as nuScenes data preparation, so we will move the directory within Docker.
| cd bevfusion |
|---|
Run the compilation of bevfusion.
| python setup.py develop |
|---|
Below is an example output. If the output at the end is similar, the compilation was successful.
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion#python setup.py develop ...(snip)... Installed /home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion Processing dependencies for mmdet3d==0.0.0 Finished processing dependencies for mmdet3d==0.0.0 |
|---|
Use create_data.py from the bevfusion (original) side to convert nuScenes into a format that bevfusion can use.
In this case, we are using v1.0-mini, so we will run the following script.
| python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --version v1.0-mini --extra-tag nuscenes |
|---|
Here is an example of the output when run:
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion# python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --version v1.0-mini --extra-tag nuscenes ====== Loading NuScenes tables for version v1.0-mini... 23 categories, 8 attribute, 4 visibility, 911 instance, 12 sensors, 120 calibrated_sensor, 31206 ego_pose, 8 log, 10 scenes, 404 sample, 31206 sample_data, 18538 sample_annotation, 4 maps, Done loading in 0.313 seconds. ====== Reverse indexing... Done reverse indexing in 0.1 seconds. |
|---|
As a result, if the contents under data/nuscenes look like the following, then it's OK.
| data/nuscenes/ |-- maps/ |-- samples/ |-- sweeps/ |-- v1.0-mini/ |-- nuscenes_gt_database/ ←Add |-- nuscenes_dbinfos_train.pkl ←Add |-- nuscenes_infos_train.pkl ←Add |-- nuscenes_infos_val.pkl ←Add |
|---|
This completes the preparations on the original bevfusion side.
Inference on nuScenes mini with CUDA-BEVFusion
Within Docker, change the directory back to the original Lidar_AI_Solution/CUDA-BEVFusion/ directory.
Now, let's run the tool called dump-data.py as follows:
| python tool/dump-data.py |
|---|
This is a tool that converts the data/nuscenes we prepared earlier into the format used by the single-image example.
An example of execution is as follows. It stops with an error on the 81st page, but it is normal (this is because the terminal implementation of this script is rough. It can be fixed, but for now it is as it is...)
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion# python tool/dump-data.py Save tensor[(1, 6, 3, 256, 704), float16] to dump/00000/images.tensor Save tensor[(207998, 5), float16] to dump/00000/points.tensor Save tensor[(23, 9), float32] to dump/00000/gt_bboxes_3d.tensor Save tensor[(23,), int64] to dump/00000/gt_labels_3d.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00000/camera_intrinsics.tensor ... (snip) ... Save tensor[(1, 4, 4), float32] to dump/00079/lidar_aug_matrix.tensor Save tensor[(1, 6, 3, 256, 704), float16] to dump/00080/images.tensor Save tensor[(235496, 5), float16] to dump/00080/points.tensor Save tensor[(37, 9), float32] to dump/00080/gt_bboxes_3d.tensor Save tensor[(37,), int64] to dump/00080/gt_labels_3d.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera_intrinsics.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera2ego.tensor Save tensor[(1, 4, 4), float32] to dump/00080/lidar2ego.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/lidar2camera.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera2lidar.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/lidar2image.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/img_aug_matrix.tensor Save tensor[(1, 4, 4), float32] to dump/00080/lidar_aug_matrix.tensor Traceback (most recent call last): File "tool/dump-data.py", line 97, in dump_tensor(args) File "tool/dump-data.py", line 61, in dump_tensor data = next(data_iter) File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__ data = self._next_data() File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1176, in _next_data raise StopIteration StopIteration |
|---|
As a result of executing this, 81 sets of folders, numbered 00000 to 00080, will be created under dump/.
Next, we will modify run.sh, which output the single sample image we created earlier, and create a script that performs inference on the 81 sets of data under dump/.
Change the last part of run.sh (the line ./build/bevfusion ...) to the following and save it with a different name, such as tools/run_nexty.sh.
| #Comment out this original line and change it to the one below (continuously executes the dump below) #./build/bevfusion $DEBUG_DATA $DEBUG_MODEL $DEBUG_PRECISION #Create the final output destination mkdir -p out_dump #dump/The following 81 loops for i in {0..80} ; do TMP="00000${i}" ./build/bevfusion dump/${TMP: -5} $DEBUG_MODEL $DEBUG_PRECISION mv build/cuda-bevfusion.jpg out_dump/${TMP: -5}.jpg done Copy |
|---|
When you run the created tool/run_nexty.sh, sequentially numbered jpg files will be output under out_dump/.
| bash tool/run_nexty.sh |
|---|
An example of the output of the first image (00000.jpg) is shown below.
The video from the sequential jpg files is created outside of Docker using ffmpeg etc. Please see the video results for yourself!
Preparation on the bevfusion (main) side
As a final preparation, copy configs/ to bevfusion/ in the Lidar_AI_Solution/CUDA-BEVFusion/ directory. This will also be done outside of Docker.
(The purpose is to copy the resnet50/ folder located at the end of a deep directory with the same name.)
| cp -r configs/ bevfusion/ |
|---|
This completes the file setup.
From now on, you will be working within Docker. Under the Lidar_AI_Solution/CUDA-BEVFusion/ directory, there is a directory called bevfusion, which is the original repository clone.
We will use the original tools, such as nuScenes data preparation, so we will move the directory within Docker.
| cd bevfusion |
|---|
Run the compilation of bevfusion.
| python setup.py develop |
|---|
Below is an example output. If the output at the end is similar, the compilation was successful.
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion#python setup.py develop ...(snip)... Installed /home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion Processing dependencies for mmdet3d==0.0.0 Finished processing dependencies for mmdet3d==0.0.0 |
|---|
Use create_data.py from the bevfusion (original) side to convert nuScenes into a format that bevfusion can use.
In this case, we are using v1.0-mini, so we will run the following script.
| python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --version v1.0-mini --extra-tag nuscenes |
|---|
Here is an example of the output when run:
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion/bevfusion# python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --version v1.0-mini --extra-tag nuscenes ====== Loading NuScenes tables for version v1.0-mini... 23 categories, 8 attribute, 4 visibility, 911 instance, 12 sensors, 120 calibrated_sensor, 31206 ego_pose, 8 log, 10 scenes, 404 sample, 31206 sample_data, 18538 sample_annotation, 4 maps, Done loading in 0.313 seconds. ====== Reverse indexing... Done reverse indexing in 0.1 seconds. |
|---|
As a result, if the contents under data/nuscenes look like the following, then it's OK.
| data/nuscenes/ |-- maps/ |-- samples/ |-- sweeps/ |-- v1.0-mini/ |-- nuscenes_gt_database/ ←Add |-- nuscenes_dbinfos_train.pkl ←Add |-- nuscenes_infos_train.pkl ←Add |-- nuscenes_infos_val.pkl ←Add |
|---|
This completes the preparations on the original bevfusion side.
Inference on nuScenes mini with CUDA-BEVFusion
Within Docker, change the directory back to the original Lidar_AI_Solution/CUDA-BEVFusion/ directory.
Now, let's run the tool called dump-data.py as follows:
| python tool/dump-data.py |
|---|
This is a tool that converts the data/nuscenes we prepared earlier into the format used by the single-image example.
An example of execution is as follows. It stops with an error on the 81st page, but it is normal (this is because the terminal implementation of this script is rough. It can be fixed, but for now it is as it is...)
| root:/home/xxx/bev/Lidar_AI_Solution/CUDA-BEVFusion# python tool/dump-data.py Save tensor[(1, 6, 3, 256, 704), float16] to dump/00000/images.tensor Save tensor[(207998, 5), float16] to dump/00000/points.tensor Save tensor[(23, 9), float32] to dump/00000/gt_bboxes_3d.tensor Save tensor[(23,), int64] to dump/00000/gt_labels_3d.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00000/camera_intrinsics.tensor ... (snip) ... Save tensor[(1, 4, 4), float32] to dump/00079/lidar_aug_matrix.tensor Save tensor[(1, 6, 3, 256, 704), float16] to dump/00080/images.tensor Save tensor[(235496, 5), float16] to dump/00080/points.tensor Save tensor[(37, 9), float32] to dump/00080/gt_bboxes_3d.tensor Save tensor[(37,), int64] to dump/00080/gt_labels_3d.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera_intrinsics.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera2ego.tensor Save tensor[(1, 4, 4), float32] to dump/00080/lidar2ego.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/lidar2camera.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/camera2lidar.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/lidar2image.tensor Save tensor[(1, 6, 4, 4), float32] to dump/00080/img_aug_matrix.tensor Save tensor[(1, 4, 4), float32] to dump/00080/lidar_aug_matrix.tensor Traceback (most recent call last): File "tool/dump-data.py", line 97, in dump_tensor(args) File "tool/dump-data.py", line 61, in dump_tensor data = next(data_iter) File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__ data = self._next_data() File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1176, in _next_data raise StopIteration StopIteration |
|---|
As a result of executing this, 81 sets of folders, numbered 00000 to 00080, will be created under dump/.
Next, we will modify run.sh, which output the single sample image we created earlier, and create a script that performs inference on the 81 sets of data under dump/.
Change the last part of run.sh (the line ./build/bevfusion ...) to the following and save it with a different name, such as tools/run_nexty.sh.
| #Comment out this original line and change it to the one below (continuously executes the dump below) #./build/bevfusion $DEBUG_DATA $DEBUG_MODEL $DEBUG_PRECISION #Create the final output destination mkdir -p out_dump #dump/The following 81 loops for i in {0..80} ; do TMP="00000${i}" ./build/bevfusion dump/${TMP: -5} $DEBUG_MODEL $DEBUG_PRECISION mv build/cuda-bevfusion.jpg out_dump/${TMP: -5}.jpg done Copy |
|---|
When you run the created tool/run_nexty.sh, sequentially numbered jpg files will be output under out_dump/.
| bash tool/run_nexty.sh |
|---|
An example of the output of the first image (00000.jpg) is shown below.
The video from the sequential jpg files is created outside of Docker using ffmpeg etc. Please see the video results for yourself!
Conclusion
In this article, we explained the steps to actually run CUDA-BEVFusion on your own. Although it is originally an autonomous driving technology for in-vehicle use, running it on a PC simulation can help you gain a deeper understanding and see the finer details.
The NEXTY Electronics Development Department investigates cutting-edge technology trends and conducts numerous trials at the edge using PCs, GPU servers, NVIDIA DRIVE Orin, and other devices.
Furthermore, Nexty Electronics provides technical support related to autonomous driving and AI, as well as server trial environment services (GPU Advanced Test Drive: GAT) such as the DGX-H100/A100 for training purposes. Please feel free Inquiry.
