Test Isaac_ROS_Nvblox Repository
Launch File
isaac_sim_example.launch.py, isaac_sim_humans_example.launch.py, isaac_sim_dynamics_example.launch.py, realsense_example.launch.py
realsense_humans_example.launch.py, realsense_dynamics_example.launch.py, record_realsense.launch.py, zed_example.launch.py
Static Reconstruction in Isaac Sim
Example to run with Isaac Sim
Static Reconstruction in Isaac Sim
Generating a 3D reconstruction with nvblox using image and LiDAR data from Isaac Sim
The reconstruction is displayed as a 3D mesh in RVIZ.
Nvblox also converts the reconstruction into a 2D costmap, which is fed into the Nav2 stack for path planning.
Footnote
3D mesh: Collection of vertices, edges, and faces that defines the shape of a three-dimensional object in computer graphics and 3D modeling
Vertices: Points in 3D space
Edges: Straight line segment connecting two vertices
Faces: Flat surfaces that make up the 3D shape. Formed by connecting three or more vertices
In most 3D meshes, faces are typically triangles or quadrilaterals
-When displaying a 3D reconstruction as a mesh in RViz (a common visualization tool in robotics), the mesh acts as a visual representation of the object or environment
This can be particularly useful in robotics for understanding and analyzing the shape and features of physical spaces or objects that a robot might interact with
-2D Costmap:
costmap: grid that represents a robot’s environment, where each cell in the grid corresponds to a specific area in the real world
Each cell in the costmap has a value or “cost” associated with it, indicating how desirable or undesirable it is for the robot to travel over that cell
For example, areas with obstacles would have a high cost, making them less desirable paths, while free space would have a low cost.
The costmap can integrate data from various sensors (like LiDAR, cameras) to determine the cost values
Tutorial Walkthrough
(1)Complete the Nvblox Quickstart
=> Already Done
(2)Install Isaac Sim, open the Isaac Sim terminal and export the necessary environment variables as explained in steps 1-3 of the Isaac Sim Setup Guide
Install Isaac Sim, choosing the appropriate working environment:
=> Native, already done
(3)Start the simulation by running the following command in the terminal
(4)In another terminal, launch the Docker container using the run_dev.sh script
(5)Inside the container, build and source the workspace:
Error:
2024-01-08 12:24:23 [18,263ms] [Warning] [omni.isaac.dynamic_control.plugin] Failed to find articulation at ‘/World/Carter_ROS/chassis_link’
2024-01-08 12:24:23 [18,263ms] [Error] [omni.isaac.dynamic_control.plugin] DcGetArticulationRootBody: Invalid or expired articulation handle
2024-01-08 12:24:23 [18,263ms] [Warning] [omni.physx.tensors.plugin] Failed to find articulation at ‘/World/Carter_ROS/chassis_link’
2024-01-08 12:24:23 [18,263ms] [Error] [omni.physx.tensors.plugin] Pattern ‘/World/Carter_ROS/chassis_link’ did not match any articulations
2024-01-08 12:24:24 [18,474ms] [Warning] [omni.graph.core.plugin] /World/Carter_ROS/ActionGraph/articulation_controller: [/World/Carter_ROS/ActionGraph] OmniGraph Warning: ‘NoneType’ object has no attribute ‘is_homogeneous’
(from compute() at line 112 in /home/rosexpert1/.local/share/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core_nodes/omni/isaac/core_nodes/ogn/python/nodes/OgnIsaacArticulationController.py)
2024-01-08 12:24:24 [18,475ms] [Error] [omni.graph.core.plugin] /World/Carter_ROS/ActionGraph/isaac_compute_odometry_node: [/World/Carter_ROS/ActionGraph] prim is not a valid rigid body or articulation root
2024-01-08 12:24:24 [18,480ms] [Error] [omni.graph.core.plugin] /World/Carter_ROS/ActionGraph/isaac_read_lidar_point_cloud_node: [/World/Carter_ROS/ActionGraph] Prim is not a Lidar Prim
2024-01-08 12:24:24 [18,812ms] [Warning] [omni.isaac.dynamic_control.plugin] Failed to find articulation at ‘/World/Carter_ROS/chassis_link’
2024-01-08 12:24:24 [18,812ms] [Error] [omni.isaac.dynamic_control.plugin] DcGetArticulationRootBody: Invalid or expired articulation handle
2024-01-08 12:24:24 [18,812ms] [Warning] [omni.physx.tensors.plugin] Failed to find articulation at ‘/World/Carter_ROS/chassis_link’
2024-01-08 12:24:24 [18,812ms] [Error] [omni.physx.tensors.plugin] Pattern ‘/World/Carter_ROS/chassis_link’ did not match any articulations
2024-01-08 12:24:24 [18,813ms] [Error] [omni.isaac.dynamic_control.plugin] DcGetRigidBodyPose: Invalid or expired body handle
2024-01-08 12:24:24 [18,813ms] [Error] [omni.isaac.dynamic_control.plugin] DcGetRigidBodyLocalLinearVelocity: Invalid or expired body handle
2024-01-08 12:24:24 [18,813ms] [Error] [omni.isaac.dynamic_control.plugin] DcGetRigidBodyAngularVelocity: Invalid or expired body handle
terminate called without an active exception
Fatal Python error: Aborted
Solution: (Searching for it)
https://forums.developer.nvidia.com/t/static-reconstruction-in-isaac-sim-tutorial-error/279656
=> NVIDA’s Error! Now Fixed
Correct Version:
(6)Launch the pre-composed graph launch file
(7)Click on the 2D Goal Pose button. You should see the mesh, costmap, and the robot moving towards the goal location, as shown at the top of this page
Successful Result:
Omniverse Screen
Isaac Sim Screen
Video
https://drive.google.com/file/d/12f0ArTzN-RE0IBAUIa_-tmfKkwzLIYPS/view?usp=sharing
Reference
https://nvidia-isaac-ros.github.io/concepts/scene_reconstruction/nvblox/tutorials/tutorial_isaac_sim.html
Human Reconstruction in Isaac Sim
Example to run with Isaac Sim including human reconstruction
Human Reconstruction in Isaac Sim
Demonstrates how to perform human reconstruction in nvblox with Isaac Sim
The detected humans are then visible in a separate dynamic costmap that can be used for navigation with Nav2.
If you want to know more about human reconstruction works, see Technical Details.
Technical Details
Human Reconstruction
Additional options for mapping scenes containing people exist.
-Humans are separated from the TSDF reconstruction into a separate layer containing an occupancy grid representing reconstructed humans
-Human segmentation is applied to each processed color frame with Isaac ROS Image Segmentation
-The depth masker module uses the segmentation mask from the color image to separate the depth-map into human and non-human parts.
-While the non-human labeled part of the depth frame is still forwarded to TSDF mapping, the human labeled part is processed to an occupancy grid map.
-To relax the assumption that occupancy grid maps only capture static objects, an occupancy decay step must be applied. At a fixed frequency, all voxel occupancy probabilities are decayed towards 0.5 over time. This means that the state of the map (occupied or free) becomes less certain after it has fallen out of the field of view, until it becomes unknown (0.5 occupancy probability)
Tutorial Walkthrough
Use Isaac Sim for human reconstruction in nvblox. nvblox uses the ground truth semantic segmentation coming from the simulation as input to detect humans
This demonstration relies on the extensions omni.anim.people and omni.anim.navigation to make humans navigate in the environment while the robot is moving
Running the Demonstration Scene
Before continuing this example, you must have successfully completed the Static Reconstruction in Isaac Sim
=> Successfully Completed
(1)Open the Isaac Sim terminal and export the necessary environment variables as explained in steps 2-3 of the Isaac Sim Setup Guide
(1)-1 Follow the Running Native ROS instructions to ensure that ROS 2 and the required packages are installed to successfully enable the ROS 2 Bridge in Omniverse Isaac Sim
=> Completed
(1)-2 Follow the instructions to launch Isaac Sim App Selector window
=> Completed
(2)Start the simulation by running the following command in the terminal
(3)In another terminal, run the ROS Docker container using the run_dev.sh script:
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
./scripts/run_dev.sh
(4)Inside the container, build and source the workspace:
cd /workspaces/isaac_ros-dev && \
colcon build --symlink-install && \
source install/setup.bash
Error:
Fixed Version:
(5)Launch nvblox configured for human mapping:
ros2 launch nvblox_examples_bringup isaac_sim_humans_example.launch.py
Error:
Fixed Version:
Solution for steps (4) & (5): (Searching for it…)
https://forums.developer.nvidia.com/t/encountering-file-and-package-location-issues-with-human-reconstruction-running-the-demonstration-scene-in-isaac-sim/281057
Deleted the Ros Benchmark Repository
Fixed Result:
Omniverse Screen
Terminal
Rviz
Error Again:
Solution: (Searching for it…)
https://forums.developer.nvidia.com/t/encountering-file-and-package-location-issues-with-human-reconstruction-running-the-demonstration-scene-in-isaac-sim/281057
Correct version for step (5):
ros2 launch nvblox_examples_bringup isaac_sim_dynamics_example.launch.py run_vslam:=True global_frame:=odom
Successful Result:
Omniverse Screen:
Rviz:
Video:
https://drive.google.com/file/d/13DloUuPTPgX6bbWMfTPeUD22Avtsx4Vb/view?usp=sharing
Running with Custom Human Paths
To have non-default paths taken by the humans, you can
Change the default human animation file directly on the server
Use the randomization options of the above script. To do so:
Reference
https://nvidia-isaac-ros.github.io/concepts/scene_reconstruction/nvblox/tutorials/tutorial_human_reconstruction_isaac_sim.html
Dynamic Reconstruction in Isaac Sim
Example to run with Isaac Sim including general dynamic reconstruction
This tutorial demonstrates how to perform general dynamic reconstruction in nvblox with Isaac Sim.
-For more about general dynamic reconstruction in nvblox see Technical Details.
Dynamic Reconstruction
-Human reconstruction pipeline employs a Deep Neural Network(DNN) to generate a mask image for separating human detections into a dynamic occupancy layer
-General dynamic reconstruction pipeline maintains a (high-confidence) freespace layer dedicated to detecting dynamic objects
-Whenever an object enters freespace, it is identified as dynamic and then integrated into the dynamic occupancy layer, similar to the human reconstruction pipeline described above.
This enables the pipeline to separately reconstruct humans (or other specific objects that the DNN was trained for) and all moving objects regardless of their class or category
Tutorial Walkthrough
Before continuing this example, you must have successfully completed the Static Reconstruction in Isaac Sim
=> Successfully Completed
(1)Open the Isaac Sim terminal and export the necessary environment variables as explained in steps 2-3 of the Isaac Sim Setup Guide
(1)-1 Follow the Running Native ROS instructions to ensure that ROS 2 and the required packages are installed to successfully enable the ROS 2 Bridge in Omniverse Isaac Sim
=> Completed
(1)-2 Follow the instructions to launch Isaac Sim App Selector window
=> Completed
(2)Start the simulation by running the following command in the terminal
(3)In another terminal, run the ROS Docker container using the run_dev.sh script:
(4)Inside the container, install isaac_ros_visual_slam
(5)Inside the container, build and source the workspace: (New Step)
cd /workspaces/isaac_ros-dev && \
> colcon build --symlink-install && \
> source install/setup.bash
(6)Launch the pre-composed graph launch file:
ros2 launch nvblox_examples_bringup isaac_sim_dynamics_example.launch.py run_vslam:=True global_frame:=odom_vslam
Error:
Solution: (Searching for it…)
https://forums.developer.nvidia.com/t/difficulty-locating-the-nvblox-examples-bringup-package-for-dynamic-reconstruction-in-isaac-sim/281052
=> Delete Benchmark Package and source the workspace
Correct Version:
After fixing Error:
Omniverse Screen
Terminal
Rviz
Error Again:
Solution: (Searching for it…)
https://forums.developer.nvidia.com/t/difficulty-locating-the-nvblox-examples-bringup-package-for-dynamic-reconstruction-in-isaac-sim/281052
=> Nvidia’s bug
Correct Version:
ros2 launch nvblox_examples_bringup isaac_sim_dynamics_example.launch.py run_vslam:=True global_frame:=odom
Successful Result:
Omniverse Screen
Rviz
Video:
https://drive.google.com/file/d/1knmg5Dn-jy_7c966EfzqtrJq1FkuvsPY/view?usp=sharing
Reference
https://nvidia-isaac-ros.github.io/concepts/scene_reconstruction/nvblox/tutorials/tutorial_dynamic_reconstruction_isaac_sim.html
Nvblox ROS 2 integration for local 3D scene reconstruction and mapping
Integration for Local 3D Scene Reconstruction and Mapping:
Nvblox is being used in conjunction with ROS 2 to perform the task reconstructing and mapping local 3D scenes.
Integrated system can take sensor data (like images or laser scans) from an environment and use it to build a three-dimensional model of that area. This is a crucial task in robotics for understanding and navigating the surrounding space.