IsaacSim Semantic Segmentation

Synthetic Data Pipeline Part1: Semantic Information

Select Sensors

Choose SemanticSegmentation in RenderVar

Click on Semantic Data Recorder

Choose the _03_cracker_box mesh

Configure New Semantic Type as class and New Semantic data as box
Click on Add Entry On All Selected Prims
Do the same thing for the other cracker_box2

The crackerboxes which appeared black now appear in shades of hot pink

Choose the _05_tomato_soup_can mesh
Configure New Semantic Type as class and New Semantic data as can
Click on Add Entry On All Selected Prims
Do the same thing for the other _05_tomato_soup_can

The soup cans which appeared black now appear in shades of deep blue and the cracker boxes which appeared to be in shades of hot pink now appears to appear in shades of bright red

Synthetic Data Recorder

A GUI-based extension in Isaac Sim to record synthetic data using Replicator

Learning Objectives

This tutorial introduces the Synthetic Data Recorder for Isaac Sim, a GUIextension for recording synthetic data with the possibility of using custom Replicator writers to record the data in any custom format
Getting Started

Replicator > Synthetic Data Recorder

Basic Usage

The recorder is split into two main parts:
the Writer frame - containing sensor, data, and output parameters; and the Control frame - containing the recording functionalities such as start, stop, pause, and parameters such as the number of frames to execute

Writer Parameters
The Render Products frame creates a list of render product entries using the Add New Render Product button
By default a new entry is added to the list using the active viewport camera as its camera path (see left figure)
If however cameras are selected in the stage viewer, these are added to the render products list (see right figure)
The render products list can contain the same camera path multiple times, however each time with a different resolution
All the entry values (camera path or resolution) can also be manually edited in the input fields

Click Add New Render Product Entry to add turtle_Camera


The Parameters frame gives the possibility to choose between the default built-in Replicator writer (BasicWriter) or to choose a custom writer
The default writer parameters (mostly annotators) can be selected from the checkbox list
As custom writers have unknown parameters, these can be provided by the user in form of a json file containing all the required parameters
The path to the json file can be added in the Parameters Path input field

Click to check Semantic_Segmentation

The Output frame (left figure) contains the working directory path where the data will be saved together with the folder name used for the current recording
The output folder name will be incremented in case of conflict
The recorder can also write to S3 buckets by checking Use S3 and providing the required fields and having the AWS credentials set up

The Config frame (right figure) can load and save the GUI writer state as a json config file. By default the extension loads the previously used configuration state
Control

The Control frame contains the recording functionalities such as Start/Stop and Pause/Resume, and parameters such as the number of frames to record, or the number of subframes to render for each recorded frame
The Start button will create a writer given the selected parameters and start the recording
The Stop button will stop the recording and clear the writer
The Pause button will pause the recording without clearing the writer, and the Resume button will resume the recording
The Number of frames input field will set the number of frames to record, after which the recorder will be stopped and the writer cleared
If the value is set to 0, the recording will run indefinitely until the Stop button is pressed
The RTSubframes field will set the number of additional subframes to render for each per frame
This can be used if randomized materials are not loaded in time or if temporal rendering artifacts (such as ghosting) are present due to objects being teleported
The Control Timeline checkbox will start/stop/pause/resume the timeline together with the recorder, after each recording the timeline is moved to timestamp 0

Record Turtlebot Semantic data using Synthetic Data Recorder

(1)Choose Parameters

Chose rgb, colorize_semantic_segmentation, instance_id_segmentation, and colorize_instance_id_segmentation
(2)Writer Parameters

Stored data in /home/rosexpert1/Semantic_data directory _out_sdrec folder
(3)Control Parameters

(4)Drive Turtlebot
(4)-1 To make it easier for us to move the Turtlebot around, install the teleop_twist_keyboard by running the following command:

#sudo apt-get install ros-$ROS_DISTRO-teleop-twist-keyboard
sudo apt-get install ros-humble-teleop-twist-keyboard

(4)-2 Enable driving using the keyboard by running:

ros2 run teleop_twist_keyboard teleop_twist_keyboard

Record Turtlebot Semantic data using Ros2 bags

(1)Fix the ActionGraph01
(1)-1 Change configuration of Ros2 Camera Helper Node

Modify the topic name to semantic_segmentation

Reference: https://docs.omniverse.nvidia.com/isaacsim/latest/replicator_tutorials/tutorial_replicator_recorder.html