Isaac ROS Depth Segmentation

Overview

Isaac ROS Depth Segmentation
Isaac ROS Depth Segmentation provides NVIDIA hardware-accelerated packages for depth segmentation
-The isaac_ros_bi3d package uses the optimized Bi3D DNN model to perform stereo-depth estimation via binary classification, which is used for depth segmentation
-Depth segmentation can be used to determine whether an obstacle is within a proximity field and to avoid collisions with obstacles during navigation

-Bi3D is used in a graph of nodes to provide depth segmentation from a time-synchronized input left and right stereo image pair
-Images to Bi3D need to be rectified and resized to the appropriate input resolution. The aspect ratio of the image needs to be maintained;
hence, a crop and resize may be required to maintain the input aspect ratio
-The graph for DNN encode, to DNN inference, to DNN decode is part of the Bi3D node
Inference is performed using TensorRT, as the Bi3D DNN model is designed to use optimizations supported by TensorRT
-Compared to other stereo disparity functions, depth segmentation provides a prediction of whether an obstacle is within a proximity field, as opposed to continuous depth
while simultaneously predicting freespace from the ground plane, which other functions typically do not provide
-Also unlike other stereo disparity functions in Isaac ROS, depth segmentation runs on NVIDIA DLA(deep learning accelerator)
which is separate and independent from the GPU -For more information on disparity, refer to this page

Quickstarts

Isaac ROS Depth Segmentation
(1)Set up your development environment by following the instructions here
=> Already Done
(2)Clone isaac_ros_common and this repository under ${ISAAC_ROS_WS}/src

(3)Pull down a rosbag of sample data

(4)Launch the Docker container using the run_dev.sh script

(5)Install this package’s dependencies

(6)Download model files for Bi3D (refer to the Model Preparation section for more information):

(7)Convert the .onnx model files to TensorRT engine plan files (refer to the Model Preparation section for more information)

(8)Run the launch file to spin up a demo of this package:

(9)Open a second terminal inside the Docker container:

(10)Play the rosbag file to simulate image streams from the cameras:

(11)Open two new terminals inside the Docker container for visualization:


(12)Visualize the output
(12)-1 Start disparity visualizer:

Result:

(12)-2 Start image visualizer:

Result:

Isaac ROS NITROS Acceleration

Package is powered by NVIDIA Isaac Transport for ROS (NITROS)
Which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes

Packages

isaac_ros_bi3d

Supported Platforms

Package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU

Docker

Strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps
This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms

Reference:
https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_depth_segmentation/isaac_ros_bi3d/index.html#quickstart