Object Detection Synthetic Data Generation

omni.replicator.object is a no-code-change-required extension that generates synthetic data for model training, that can be used on a range of tasks from retail object detection to robotics
It takes a description file in YAML that describes a mutable scene, or a hierarchy of such stacked description files as input, and outputs a description file along with graphics content including RGB, 2D/3D bounding boxes, segmentation masks, etc
../_images/overview.png

Run Instructions

The extension can be run from the UI, or isaac-sim container

Run from the UI

In Omniverse Kit or Isaac Sim, enable omni.replicator.object from Windows -> Extensions -> THIRD PARTY
A window will pop up


Clicking on the folder icon or the Visual Studio Code icon below,

The root folder of the extension will be open

Under source/extensions/omni.replicator.object/omni/replicator/object/configs there are many description files in YAML format


It’s recommended to start with demo_kaleidoscope.yaml
To do so, go to global.yaml and update output_path to any local folder to store the simulation output


Next, put demo_kaleidoscope in the Description File textbox, and hit Simulate to start simulation;
alternatively, you can also put the absolute path …/source/extensions/omni.replicator.object/omni/replicator/object/configs/demo_kaleidoscope.yaml, with … being the local path
Results will be stored in output_path



Errors in loading the environment, so I applied the simple_room.usd instead

Concepts

The description file

The description file is a YAML file that has a key omni.replicator.object, and the value corresponding to that key is the description
-The description consists of key-value pairs
-Each key-value pair is a Mutable, a Harmonizer, or a Setting
-The description generates frames as the user demands
Each frame, the scene is randomized, and graphics content is captured and output to disk
-Settings describe how the scene is configured and how data is output
For example, you can set the number of frames to output, whether or not to output 2D bounding boxes, or set the gravity and friction of physics simulation
-The description composes the scene populated with objects that are called mutables
Mutables randomize every frame
-Sometimes we want to constrain how they randomize
For example, to know how other mutables are randomizing and randomize correspondingly
To do so, we can define harmonizers

Simulation Workflow

Every time a simulation is launched, the description is parsed, and then the scene along with the mutables and harmonizers are initialized
Every frame it goes through the following stages
Free randomize
Mutable attributes that doesn’t depend on harmonizer
Randomizes freely
Harmonize
The harmonizers randomize, and collect information from free randomized mutable attributes
The collected information are called pitches
Just like real harmonizers, the pitches are collected and propagated back to all relevant mutables
Harmonized randomize
Using the return information from the harmonizer, the harmonized mutable attributes randomizes
Physics resolution
Physics is resolved so that objects move away from each other if they overlap, or drop onto a surface if gravity is turned on
For more details, refer to Physics simulation explained
Capture
Graphics content is captured
Logging
The state of the scene in this frame is recorded into a description file, such that later on it can be restored or inspected

Scene restoration

In the output content, you can use the output description file of a specific frame to generate the exact same graphics content as when this frame was generated, once again
Or you can slightly modify it to have something different but everything else the same
In this way we support multiple-sampling for pretrained models

Convention in this documentation

Type in the tables indicate the expected data type.
Where a type is expected, a macro string can be used in place to be later on evaluated for that specific type
For example, if we expect int in a value, we can either give an int or something like $[index]. See Macro for details
Within a mutable, aside from these 2 options, you can also specify a Mutable attribute to evaluate to this type
And numeric means literal or evaluated float or int

Catalog

Reference: https://docs.omniverse.nvidia.com/isaacsim/latest/replicator_tutorials/tutorial_replicator_object.html