Introductory

Add Camera and Sensors

Omniverse Isaac Sim provides a variety of sensors that can be used to sense the environment and robot’s state
Cover the simple example of attaching a camera sensor to our mock robot, a process that can be generalized to other sensors
(Details regarding the camera and other types of sensors can be found in our Advance Tutorials and Sensor Extensions under Manuals)

Learning Objectives

Add cameras
Attach cameras to geometries

Getting Started

Prerequisites
Introductory Tutorials, Previous tutorial in the GUI Tutorials, as well as the introduction to camera frames and axes

Sensor Axes Representation (LiDAR, Cameras)

Cameras in Isaac Sim are subject to three different types of axes definition, depending on the context of use
Introduce the three conventions and how it’s used in different contexts
World axes
World axes uses the +X forward, +Z up convention
The origin of the world prim is always represented in the World axes
Camera prim, represented in the world axes, is shown in the figure below

USD Axes
USD convention is used in the computer graphics community
The USD axes uses the +Y up, -Z forward convention
Property panel displays the poses of objects in the USD stage in an Isaac Sim application
Poses of all objects in the stage are displayed in the world axes, with the exception of camera prims, which is displayed in the +Y up, -Z forward convention
=> The convention is referred to as USD Axes in the context of camera prims. Camera prim, represented in the USD axes convention, is shown in the figure below

ROS Axes
ROS axes uses the -Y up, +Z forward convention
=>Any camera data including transforms published to ROS(ROS Cameras) or ROS2(ROS2 Cameras) will be represented in this convention
The camera prim, represented in the ROS axes convention, is shown in the figure below

Adding Camera

(1)Create camera and identify it in the stage
Go to the Menu Bar and select Create > Camera
A camera will appear on the stage tree, and a grey wireframe representing the camera’s view will appear on the stage

You can move and rotate the camera’s transform just like any other objects on the stage

Camera Inspector Extension

Once cameras are added in the scene, Camera Inspector Extension can be used to manage our cameras
Camera Inspector Extension:
Allows us to easily create multiple viewports for each camera, check camera coverage, as well as get/set camera poses in the desired frames
(1)Launching Extension
Open the Camera Inspector extension
Go to the Menu Bar and select Isaac Utils > Workflows > Camera Inspector
=> Should be able to see our camera selected already in the dropdown
Be sure to click the Refresh button to ensure that the extension finds this new camera When a new camera is added

(2)Camera State Textbox
Provides a convenient way to copy the position and orientation of your camera directly into code
Click the copy icon on the right of the textbox to copy to your clipboard

(3)Creating Viewport
A new viewport for our camera can be created with our camera selected
(3)-1 Click on the ‘Create Viewport’ button to the right of the camera dropdown menu (This creates a new viewport and assigns the current selected Camera to it by default)

Different cameras to different viewports can be assigned using the two dropdown menus and buttons in the extension
(3)-2 Going to viewport, the resolution using the menu can also be changed in the top left
Omniverse Kit only supports square pixels. Meaning that when changing the resolution, resolution aspect ratio must be the same as the aperture ratio

(4)Camera Status Panel

camera status panel: the camera and viewport dropdown menus
-Panel displays the location and orientation of the camera in both the local coordinate system and also the world coordinate system
Default: Displays the coordinates in the World Axis
However, using the Camera Axes dropdown, this can be changed to be in the USD Axis or the ROS Axis
-Camera status panel will update automatically if the camera is moved in the stage using the gizmo. If the location or orientation of the camera is updated in the property panel camera status panel will be also updated.

Attach Camera to Robot

(1)Rename the newly added camera to car_camera so we can keep track of it

(2)-1 Would be easier to place the camera if we could see both the desired camera input stream, as well as where it is relative to the robot from an outside camera

(2)-2 Open up a second viewport window by going to the Menu Bar and click Window > Viewport > Viewport 2

New viewport appears; dock it wherever you’d like
(3)Keep one of the viewport in Perspective camera view, and change the other one to car_camera view

Find the Cameras menu on the top edge of the viewport, and switch to Camera > car_camera

Have both the view of the onboard camera, as well as an overview of the scene
(4)Attach the camera to the robot’s body by dragging the prim under body
Now the camera will move together with the body. (May need to switch the camera view for the viewport again)
=> Already Done
(5)Point the camera slightly down and make it face forward so we can see the car as well as the ground

Set the camera
transform translation: x=-0.6,y=0,z=2.2
orientation: x=0,y=-80,z=-90
scale: x=1,y=1,z=1

Should see the viewport showing the onboard camera view:
Splitting the window between the robot’s body and the ground and the relative position and orientation of the camera to the robot in the Perspective camera viewport
(6)Press Play
https://drive.google.com/file/d/1LtE9GTCewsYqNO7Nrryp_VD_Ra_91bnW/view?usp=sharing
The camera onboard the robot should now move with the robot
(Similar strategy is used to apply other onboard sensors)

Summary

Learned how to use the Camera Inspector Extension. Additionally, learned how to add a camera to the robot

Reference:
https://docs.omniverse.nvidia.com/isaacsim/latest/gui_tutorials/tutorial_gui_camera_sensors.html#isaac-sim-app-tutorial-gui-camera-sensors
https://docs.omniverse.nvidia.com/isaacsim/latest/concepts/cameras/index.html#isaac-sim-cameras