Isaac ROS Stereo Depth

Overview

Isaac ROS DNN Stereo Depth
:GPU-accelerated package for DNN-based stereo disparity
-Stereo disparity is calculated from a time-synchronized image pair sourced from a stereo camera and is used to produce a depth image or a point cloud for a scene
-The isaac_ros_ess package uses the ESS DNN model to perform stereo depth estimation via continuous disparity prediction
-Given a pair of stereo input images, the package generates a disparity map of the left input image

-ESS is used in a graph of nodes to provide a disparity prediction from an input left and right stereo image pair
-Images to ESS need to be rectified and resized to the appropriate input resolution
The aspect ratio of the image is recommended to be maintained, so the image may need to be cropped and resized to maintain the input aspect ratio
-The graph for DNN encode, DNN inference, and DNN decode is included in the ESS node
Inference is performed using TensorRT, as the ESS DNN model is designed with optimizations supported by TensorRT
ESS node is agnostic to the model dimension and disparity output has the same dimension as the ESS model

Footnote:
Stereo disparity:
Difference in coordinates of similar features within two stereo images
When capturing a scene from slightly different viewpoints(similar to how human eyes work), the same object will appear at different positions in the two images
This difference in position is what we call disparity
-In the context of computer vision and depth sensing, stereo disparity is a crucial concept used to infer depth information from two-dimensional images
-By analyzing the disparity between a pair of stereo images, algorithms can calculate the distance of objects from the camera, creating a depth map of the scene
The fundamental principle behind this process is triangulation, where the geometry of the camera setup and the disparity values are used to compute the depth for each point of interest
-Disparity is typically measured in pixels and can vary inversely with the object’s distance from the camera; closer objects will have a larger disparity, while objects farther away will have smaller disparity
This information is foundational for numerous applications, including 3D reconstruction, autonomous vehicle navigation, and augmented reality, among others

QuickStarts

Isaac ROS ESS

(1)Set up your development environment by following the instructions here
=> Already Done
(2)Clone isaac_ros_common and this repository under ${ISAAC_ROS_WS}/src

(3)Pull down a ROS Bag of sample data:

(4)Launch the Docker container using the run_dev.sh script:

(5)Install this package’s dependencies

(6)Download the pre-trained ESS model from the ESS model page:
For ESS:


For Light ESS:


(7)Convert the encrypted model (.etlt) to a TensorRT engine plan:

For ESS:
Error:

[WARNING] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[INFO] Detected input dimensions from the model: (1, 3, 576, 960)
[INFO] Detected input dimensions from the model: (1, 3, 576, 960)
[INFO] Model has no dynamic shape.
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +854, GPU +368, now: CPU 1731, GPU 4051 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +125, GPU +60, now: CPU 1856, GPU 4111 (MiB)
[INFO] Local timing cache in use. Profiling results in this builder pass will not be stored.
[WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: autotuning: CUDA error 3 allocating 0-byte buffer:
[WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: autotuning: CUDA error 3 allocating 0-byte buffer:
[WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: autotuning: CUDA error 3 allocating 0-byte buffer:
[ERROR] 10: [optimizer.cpp::computeCosts::3712] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[853…Softmax_136]}.)
[ERROR] Unable to create engine
Segmentation fault (core dumped)
For Light ESS:


Error:

Solution: (Searching for it…)
[WARNING] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32
[INFO] Detected input dimensions from the model: (1, 3, 288, 480)
[INFO] Detected input dimensions from the model: (1, 3, 288, 480)
[INFO] Model has no dynamic shape.
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +854, GPU +348, now: CPU 1729, GPU 4043 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +126, GPU +60, now: CPU 1855, GPU 4103 (MiB)
[INFO] Local timing cache in use. Profiling results in this builder pass will not be stored.
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes.
[WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: autotuning: CUDA error 3 allocating 0-byte buffer:
[WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: autotuning: CUDA error 3 allocating 0-byte buffer:
[WARNING] Skipping tactic 0x0000000000000000 due to Myelin error: autotuning: CUDA error 3 allocating 0-byte buffer:
[ERROR] 4: [optimizer.cpp::computeCosts::3710] Error Code 4: Internal Error (Could not find any implementation for node {ForeignNode[853…Softmax_136]} due to insufficient workspace. See verbose log for requested sizes.)
[ERROR] Unable to create engine
Segmentation fault (core dumped)