ROS2 Cameras

Learning Objectives

-Add additional cameras to the scene and onto the robot
-Add camera publishers
-Send ground truth synthetic perception data through rostopics

Getting Started

Prerequisite
-Completed ROS & ROS 2 Installation: installed ROS2, enabled the ROS2 extension, built the provided Isaac Sim ROS2 workspace, and set up the necessary environment variables
=> Already Done
-It is also helpful to have some basic understanding of ROS topics and how publisher and subscriber works
=> Already Done
-Completed tutorial on OmniGraph and Add Camera and Sensors
=> Already Done
-Completed URDF Import: Turtlebot so that there is a Turtlebot ready on stage
=> Already Done

Camera Publisher

Setting Up Cameras

The default camera displayed in the Viewport is the Perspective camera
You can verify that by the Camera button on the top left hand corner inside the Viewport display
Click on the Camera button and you will see there are a few other preset camera positions: Top, Front, and Right side views
-Perspective

-Top

-Front

-Right

For the purpose of this tutorial, let’s add two stationary cameras, naming them Camera_1 and Camera_2, viewing the room from two different perspectives

Left: Camera_1
Right: Camera_2
Camera_1 Configuration:

Camera_2 Configuration:

Add Camera and Sensors

Adding Camera

To add a camera, go to the Menu Bar and select Create > Camera
A camera will appear on the stage tree, and a grey wireframe representing the camera’s view will appear on the stage
Camera Inspector Extension
Once our cameras are added in the scene, we can use the Camera Inspector Extension to manage our cameras
The Camera Inspector Extension allows us to easily create multiple viewports for each camera, check camera coverage, as well as get/set camera poses in the desired frames
-Launching Extension
To open the Camera Inspector extension, go to the Menu Bar and select Isaac Utils > Workflows > Camera Inspector
See our camera selected already in the dropdown
When we add a new camera, be sure to click the Refresh button to ensure that the extension finds this new camera
-Camera State Textbox

The Camera State textbox near the top of the extension provides a convenient way to copy the position and orientation of your camera directly into code
Simply click the copy icon on the right of the textbox to copy to your clipboard
Creating Viewport

With our camera selected, we can create a new viewport for our camera
To do so, click on the ‘Create Viewport’ button to the right of the camera dropdown menu
By default, this creates a new viewport and assigns the current selected Camera to it
We can assign different cameras to different viewports using the two dropdown menus and buttons in the extension:
Example:
Viewport1

Viewport2

Add Camera to Turtlebot3 Robot

Create > Camera
Rename turtle_Camera

Move the turtle_Camera under the base_link frame

Adjust the Camer’s Transform

Simulation of moving forward
https://drive.google.com/file/d/1Af55DsNLKV5q7HeTwIw0JXyQy06y2En7/view?usp=sharing

Building the Graph for a RGB publisher

(1)Open Visual Scripting: Window > Visual Scripting > Action Graph
(2)Click on the New Action Graph Icon in middle of the Action Graph Window, or Edit Action Graph if you want to append the camera publisher to an existing action graph

(3)Build an Action Graph with the nodes and connection of the following image, and parameters using the table below

Turn on the Ros2 Bridge Extension
Parameters:
-Isaac Create Render Product

cameraPrim: /World/Camera_1
enabled: True
-ROS2 Camera Helper

Graph Explained
-Isaac Create Render Product:
Creating a render product prim which acquires the rendered data from the given camera prim and outputs the path to the render product prim
Rendering can be enabled/disabled on command by checking/unchecking the enabled field
-ROS2 Camera Helper:
Indicating which type of data to publish, and which rostopic to publish it on
Camera Helper Node
The Camera Helper Node is abstracting a complex postprocessing network from the users
Once you press Play with a Camera Helper Node connected, you may see that in the list of Action Graphs when you click on the icon on the upper left corner of the Action Graph window, a new one appears: /Render/PostProcessing/SDGPipeline
This graph is automatically created by the Camera Helper Node
The pipeline retrieves relevant data from the renderer, process it, and send them to the corresponding ROS publisher
This graph is only created in the session you are running. It will not be saved as part of your asset and will not appear in the Stage tree

Verifying ROS connection

Terminal1:

#Move Turtlebot Forward <br/>
ros2 topic pub /cmd_vel geometry_msgs/Twist '{linear:  {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0,y: 0.0,z: 0.0}}'

Terminal2:

#Verify whether the rgb image is being published
ros2 topic list

#Visualize messages received by the topic /rgb 
ros2 run rqt_image_view rqt_image_view /rgb

Visualize rgb image using the rqt_image_view method again

Reference:
https://docs.omniverse.nvidia.com/isaacsim/latest/ros2_tutorials/tutorial_ros2_camera.html
https://docs.omniverse.nvidia.com/isaacsim/latest/gui_tutorials/tutorial_gui_camera_sensors.html#isaac-sim-app-tutorial-gui-camera-sensors