Nvidia IssacSim Synthetic Data Generation

Synthetic Data

Collecting and labeling this data in the real world is time-consuming and expensive. This can hinder the development of AI models and slow down the time to solution.
Generated by computer simulations, synthetic data is comprised of 2D images or text, and can be used in conjunction with real-world data to train AI models. Synthetic data generation (SDG) can save significant time and greatly reduce costs.
Footnote
The Need for Large and Diverse Datasets in AI Training:
AI models, especially those based on ML and DL, require extensive datasets for training. These datasets must be both large(thousands to millions of elements) and diverse to ensure the AI can recognize and process a wide range of scenarios.
The datasets need to cover a broad spectrum, sometimes even beyond what we can visually perceive(like infrared data, for example), to be effective in various applications.
Challenges in Real-World Data Collection:
Collecting such vast and diverse datasets from the real world is a daunting task. It involves not just gathering the data but also labeling it accurately, which is crucial for supervised learning models. This process is often expensive and time-consuming. It also might be limited by practical constraints(like accessibility to certain types of data) and ethical concerns(like privacy issues).
The Role of Synthetic Data:
To address these challenges, synthetic data comes into play. Synthetic data is artificially generated data that mimics real-world data. It usually consists of 2D images or text but can also include other types of data.
The key advantage of synthetic data is that it can be generated under controlled conditions, allowing for a specific focus on certain scenarios or variations that might be rare or hard to capture in the real world.
Synthetic Data Generation (SDG):
SDG refers to the process of creating this artificial data. Advanced computer simulations and algorithms are used to generate data that is realistic enough to be used in AI training.
The beauty of SDG is that it can be tailored to specific needs. For instance, if you’re developing an AI model for medical diagnosis, you can create synthetic medical images that cover a range of conditions, even rare ones.
Benefits of Using Synthetic Data in AI Training:
Time and Cost Efficiency: Generating synthetic data is generally faster and less expensive than collecting and labeling real-world data. Enhanced Data Privacy: It eliminates privacy concerns associated with using real-world data, especially in sensitive fields like healthcare. Comprehensive Training: It allows the creation of diverse and comprehensive datasets that might not be possible with real-world data alone. This leads to more robust and versatile AI models.
Combining with Real-World Data: Often, synthetic data is used in conjunction with real-world data to create a more complete and effective training dataset.
In summary, synthetic data is a powerful tool in the field of AI, enabling the development of sophisticated models without the prohibitive costs and time associated with real-world data collection and labeling. It plays a crucial role in overcoming the limitations of real-world datasets, especially in terms of diversity, volume, and ethical concerns.

What is synthetic data?

Synthetic data : Annotated information that computer simulations or algorithms generate as an alternative to real-world data.
-Put another way, synthetic data is created in digital worlds rather than collected from or measured in the real world.
-It may be artificial, but synthetic data reflects real-world data, mathematically or statistically. Research demonstrates it can be as good or even better for training an AI model than data based on actual objects, events or people.
-That’s why developers of deep neural networks increasingly use synthetic data to train their models. Indeed, a survey of the field calls use of synthetic data “one of the most promising general techniques on the rise in modern deep learning, especially computer vision” that relies on unstructured data like images and video.
-The rise of synthetic data comes as AI pioneer Andrew Ng is calling for a broad shift to a more data-centric approach to machine learning. He’s rallying support for a benchmark or competition on data quality which many claim represents 80 percent of the work in AI.
“Most benchmarks provide a fixed set of data and invite researchers to iterate on the code … perhaps it’s time to hold the code fixed and invite researchers to improve the data,” he wrote in his newsletter, The Batch.
-In a report on synthetic data, Gartner predicted by 2030 most of the data used in AI will be artificially generated by rules, statistical models, simulations or other techniques.
“The fact is you won’t be able to build high-quality, high-value AI models without synthetic data,” the report said.

Why Is Synthetic Data So Important?

Problem:
Developers need large, carefully labeled datasets to train neural networks. More diverse training data generally makes for more accurate AI models.
The problem is gathering and labeling datasets that may contain a few thousand to tens of millions of elements is time consuming and often prohibitively expensive.
-Enter synthetic data. A single image that could cost $6 from a labeling service can be artificially generated for six cents, estimates Paul Walborsky, who co-founded one of the first dedicated synthetic data services.
Cost savings are just the start. Synthetic data can address privacy issues and reduce bias by ensuring users have the data diversity to represent the real world.
-Because synthetic datasets are automatically labeled and can deliberately include rare but crucial corner cases, it’s sometimes better than real-world data. For example, in the video below NVIDIA Omniverse Replicator generates synthetic data to train autonomous vehicles to navigate safely amid shopping carts and pedestrians in a simulated parking lot.

Footnote
Synthetic Data Defined:
Synthetic Data: This is data that is artificially generated rather than obtained by direct measurement or collection from real-world events or processes. Annotated Information: In the context of synthetic data, this refers to the additional information or labels that are attached to the data, which are crucial for training machine learning models. For example, in a synthetic image of a street scene, each object (like cars, pedestrians, traffic lights) might be labeled for a computer vision model.
How Synthetic Data is Generated:
Computer Simulations: These are programs that replicate real-world processes or systems. For instance, a simulation might generate synthetic images of various weather conditions for training autonomous vehicle systems.
Algorithms: These are sets of rules or instructions designed to perform a specific task. Algorithms can generate synthetic data by following predefined patterns or rules. For instance, an algorithm can create synthetic textual data for natural language processing.
Purpose of Synthetic Data:
Alternative to Real-World Data: One of the main purposes of synthetic data is to serve as a substitute for real-world data. This is particularly useful in situations where real-world data is scarce, difficult, or expensive to obtain, or when using real-world data raises privacy concerns (such as in healthcare). Training AI Models: Synthetic data is extensively used in training AI and machine learning models. Since it can be generated in large quantities and with diverse scenarios, it is valuable for training models to be accurate, reliable, and robust.
Benefits of Using Synthetic Data:
Controlled Environment: You can generate synthetic data under specific conditions or parameters, which is not always possible with real-world data.
Scalability and Diversity: It’s easier to create large and diverse datasets, which are essential for training effective AI models.
Privacy and Ethical Considerations: Synthetic data eliminates many privacy concerns associated with using real personal data, especially in sensitive fields.
Cost-Effectiveness: Collecting and annotating real-world data can be expensive and time-consuming. Synthetic data can be a more cost-effective solution.

Augmented and Anonymized Versus Synthetic Data

Data augmentation: Technique that involves adding new data to an existing real-world dataset.
Data anonymization: Given concerns and government policies about privacy, removing personal information from a dataset is another increasingly common practice. This is called data anonymization, and it’s especially popular for text, a kind of structured data used in industries like finance and healthcare.
Augmented and anonymized data are not typically considered synthetic data. However, it’s possible to create synthetic data using these techniques. For example, developers could blend two images of real-world cars to create a new synthetic image with two cars.

Synthetic Data Use cases

Ford, BMW Generate Synthetic Data
Car makers use synthetic data today
To optimize the process of how it makes cars, BMW created a virtual factory using NVIDIA Omniverse, a simulation platform that lets companies collaborate using multiple tools. The data BMW generates helps fine tune how assembly workers and robots work together to build cars efficiently.
Amazon Robotics use Synthetic Data
In logistics, Amazon Robotics uses synthetic data to train robots to identify packages of varying types and sizes. Food and beverage giant PepsiCo employs Omniverse Replicator to generate synthetic data it uses to train AI models in NVIDIA TAO, making its operations more efficient.
Synthetic Data at the Hospital, Bank and Store
Healthcare providers in fields such as medical imaging use synthetic data to train AI models while protecting patient privacy.
For example, startup Curai trained a diagnostic model on 400,000 simulated medical cases. “GAN-based architectures for medical imaging, either generating synthetic data or adapting real data from other domains … will define the state of the art in the field for years to come,” said Nikolenko in his 2019 survey. GANs are getting traction in finance, too. American Express studied ways to use GANs to create synthetic data, refining its AI models that detect fraud.
In retail, companies such as startup Caper use 3D simulations to take as few as five images of a product and create a synthetic dataset of a thousand images. Such datasets enable smart stores where customers grab what they need and go without waiting in a checkout line.

Replicator

Omniverse Replicator: Highly extensible framework built on a scalable Omniverse platform that enables physically accurate 3D synthetic data generation to accelerate training and performance of AI perception networks.
-Omniverse Replicator provides deep learning engineers and researchers with a set of tools and workflows to bootstrapping model training, improve the performance of existing models or develop a new type of models that were not possible due to the lack of datasets or required annotations. It allows users to easily import simulation-ready assets to build contextually aware 3D scenes to unleash a data-centric approach by creating new types of datasets and annotations previously not available.
-Built on open-source standards like Universal Scene Description(USD), PhysX, Material Definition Language(MDL), Omniverse Replicator can be easily integrated or connected to existing pipelines via extensible Python APIs.
-Omniverse Replicator is built on the highly extensible OmniGraph architecture that allows users to easily extend the built-in functionalities to create datasets for their own needs. It provides an extensible registry of annotators and writers to address custom requirements around type of annotations and output formats needed to train AI models. In addition, extensible randomizers allow the creation of programmable datasets that enable a data-centric approach to training these models.
-Omniverse Replicator is exposed as a set of extensions, content, and examples in Omniverse Code
Footnote
Understanding the Omniverse Replicator
Omniverse Platform:
The Omniverse Replicator is built on the Omniverse platform, which is a scalable and versatile framework. This platform is designed to support complex and large-scale 3D simulations, making it ideal for generating synthetic data.
Purpose:
Its primary function is to facilitate the generation of physically accurate 3D synthetic data. This data is crucial for training AI perception networks, which are used in applications like autonomous vehicles, robotics, and more.
Use Cases for Deep Learning Engineers and Researchers:
Bootstrapping Model Training: It helps in initiating the training process of AI models, especially when real-world data is insufficient or unavailable.
Enhancing Existing Models: It provides the means to improve the performance of current AI models by providing additional, diverse training data.
Developing New Models: The tool allows for the creation of new types of AI models that were previously unfeasible due to dataset constraints. Features of Omniverse Replicator:
Simulation-Ready Assets: Users can import assets that are ready for simulation to create contextually rich 3D scenes.
Data-Centric Approach: It supports the creation of novel datasets and annotations, expanding the possibilities for AI training and development.
Built on Open-Source Standards: Omniverse Replicator utilizes Universal Scene Description (USD), PhysX, and Material Definition Language (MDL), ensuring compatibility and integration with existing workflows.
Extensible Python APIs: These APIs allow for seamless integration or connection with existing pipelines, enhancing workflow efficiency.

Theory behind training with synthetic data(Expensive manual process)

Typical process to train a deep neural network(DNN) for perception tasks

(1)Manual collection of data(images in most cases)
(2)Manual process of annotating these images and optional augmentations
(3)These images are then converted into the format usable by the DNNs.
(4)DNN is then trained for the perception tasks
(5)Hyperparameter tuning or changes in network architecture are typical steps to optimize network performance
(6)Analysis of the model performance may lead to potential changes in the dataset however this may require another cycle of manual data collection and annotation.
Synthetic data generation enables large scale training data generation with accurate annotations in a cost-effective manner.
Furthermore, synthetic data generation also addresses challenges related to long tail anomalies, bootstraps model training where no training data is available as well as online reinforcement learnings.
Some more difficult perception tasks require annotations of images that are extremely difficult to do manually (e.g. images with occluded objects).
Programmatically generated synthetic data can address this very effectively since all generated data is perfectly labeled. The programmatic nature of data generation also allows the creation of non-standard annotations and indirect features that can be beneficial to DNN performance.

Set of challenges that need to be addressed for it to be effective for Synthetic data generation
Synthetic data sets are generated using simulation; hence it is critical that we close the gap between the simulation and real world. This gap is called the domain gap, which can be divided into two parts:
Appearance gap: Set of pixel level differences between real and synthetic images. These differences can be a result of differences in object detail, materials, or in the case of synthetic data, differences in the capabilities of the rendering system used
Content gap: Difference between the domains. This includes factors like the number of objects in the scene, the diversity in type and placement, and similar contextual information.
-Appearance gap can be further addressed with high fidelity 3D assets and ray-tracing or path-tracing based rendering, using physically based materials such as those defined with the MDL material language. Validated sensor models and domain randomization of their parameters can also help here.
-Critical tool for overcoming these domain gaps: domain randomization. Domain randomization increases the size of the domain that we generate for a synthetic dataset to try to ensure that we include the range that best matches reality including long tail anomalies. By generating a wider distribution of data than we might find in reality, a neural network may be able to learn to better generalize across the full scope of the problem.
-On the content side, a large pool of assets relevant to the scene is needed. Omniverse provides a wide variety of connectors available to other 3D applications. Developers can also write tools to generate diverse domain scenes applicable to their specific domain.
-These challenges introduce a layer of complexity to training with synthetic data, since it is not possible to know if the randomizations done in the synthetic dataset were able to encapsulate the real domain. To successfully train a network with synthetic data, the network has to be tested on a real dataset. To address any model performance issues, we adopt a data-centric approach as a first step where we tune our dataset before attempting to change model architecture or hyperparameters.

NVIDIA Omniverse Replicator for Isaac Sim

Consistent with the growing focus on data quality, NVIDIA is releasing the new Omniverse Replicator for Isaac Sim application, which is based on the recently announced Omniverse Replicator synthetic data-generation engine. These new capabilities in Isaac Sim enable ML engineers to build production-quality synthetic datasets to train robust deep-learning perception models. “Replicating” the inherent distribution of the model’s target domain is the key to maximizing model performance.
NVIDIA Omniverse Replicator for Isaac Sim
Integration with Isaac Sim:
The Omniverse Replicator for Isaac Sim is an application that integrates with Isaac Sim. Isaac Sim is NVIDIA’s advanced simulation platform designed specifically for robotics and AI applications.
This integration brings the capabilities of synthetic data generation, a key feature of the Omniverse Replicator, directly into the robotics simulation environment.
Focus on Data Quality:
There’s an increasing emphasis in the AI and ML community on the quality of data used for training models. High-quality, diverse, and realistic datasets are crucial for training robust and reliable AI models.
The Replicator for Isaac Sim addresses this need by enabling the generation of production-quality synthetic datasets. These datasets closely mimic real-world scenarios, providing a rich resource for training deep-learning perception models.
Building Synthetic Datasets:
ML engineers can use the Omniverse Replicator for Isaac Sim to create detailed and contextually accurate synthetic datasets.
These datasets are particularly valuable in situations where collecting real-world data is challenging, risky, expensive, or ethically questionable.
Replicating Target Domain Distribution:
A critical aspect of training effective AI models is the ability to replicate the inherent distribution of the model’s target domain. This means that the synthetic data should closely resemble the actual conditions and scenarios in which the AI model will operate.
For instance, if developing a perception model for a self-driving car, the synthetic data should include varied road conditions, weather scenarios, pedestrian interactions, and other real-world driving situations.
Maximizing Model Performance:
By accurately replicating these real-world conditions, Omniverse Replicator for Isaac Sim helps in maximizing the performance of the AI models.
This approach ensures that the AI models are not only theoretically accurate but also practically effective in real-world applications.
Advantages for ML Engineers:
This tool provides ML engineers with a powerful resource to train, test, and refine perception models more efficiently and effectively.
It reduces the dependency on hard-to-obtain real-world data and speeds up the development cycle of AI models, particularly in the robotics and autonomous systems domain.
Conclusion
In summary, the NVIDIA Omniverse Replicator for Isaac Sim is a significant development in the field of AI and robotics. It provides an advanced platform for generating high-quality synthetic data, crucial for training sophisticated deep-learning perception models. By facilitating the creation of datasets that closely mirror real-world conditions, it plays a pivotal role in enhancing the performance and reliability of AI models in various applications.

Omniverse Replicator for Isaac Sim advantages

Generates datasets to achieve stochastic, controlled, and bounded distributions set as targets by the developer.
Ensures that datasets contain targeted corner and test cases.
Contains camera-relative field of view placement for objects, lighting, and the scene.
Works at scale on edge- and cloud-based systems.
Traces tools and parameters used in each dataset to drive iterative processes and support quality audits on production datasets.

Figure 1. Example synthetic data generation workflow in Isaac Sim

This figure illustrates a workflow for generating synthetic data using Isaac Sim, which is a simulation tool that’s part of NVIDIA’s Omniverse platform. Here’s a step-by-step explanation of the process depicted:

Parameter File and Asset List:
The process starts with two key inputs: a parameter file and an asset list.
parameter file: specifications and variables that dictate how the synthetic data should be generated.
asset list: Compilation of 3D models and environmental elements that can be used to create the scenes for data generation.
Input Parsing:
These inputs are then parsed. Input parsing is the process of interpreting and checking the parameter file and asset list to ensure they are in the correct format and contain valid data for the synthetic data generation process.
Sampling:
Sampling is a step where specific parameters and assets are selected based on the requirements detailed in the parameter file. This might involve choosing certain types of assets or scene configurations from the broader list.
Scene Generation:
With the selected parameters and assets, a 3D scene is generated. This is the synthetic environment where the data will be captured.
It might include various elements like vehicles, pedestrians, buildings, etc., arranged according to the parsed parameters.
Data Capture:
Once the scene is generated, data is captured from it. This could involve taking images from different angles, capturing video sequences, or obtaining sensor data like LIDAR or depth measurements.
DNN Data Formatting:
The captured data is then formatted for deep neural networks (DNNs). This typically involves annotating the data with labels and converting it into a format that can be ingested by machine learning algorithms for training.
Dataset:

The formatted data is compiled into a dataset. This dataset is what will be used to train machine learning models. Copy of Input Files and Log of Sampling:

The workflow also includes creating a copy of the input files and maintaining a log of the sampling process. This is likely for record-keeping, reproducibility, and to ensure consistency across different data generation runs. Output:

The final output of the workflow is a dataset ready for use in training DNNs. This dataset should reflect the variety and complexity of the real world as specified by the parameters. The workflow described here is typical for the generation of synthetic data in a controlled and repeatable manner, ensuring that the resulting datasets are useful for training AI models to perform tasks in simulated environments that resemble real-world conditions.

OmniGraph Architecture: This architecture underpins the Replicator, enabling users to extend built-in functionalities for their specific dataset requirements. Custom Annotations and Formats: Users can customize annotations and output formats to meet the specific needs of their AI training models. Programmable Datasets with Extensible Randomizers: This feature allows for the creation of datasets that can be tailored programmatically, supporting a data-centric approach in model training. Availability:

Omniverse Replicator is available as a set of extensions, content, and examples in Omniverse Code, making it accessible for users to explore and utilize in their AI development endeavors. Conclusion The Omniverse Replicator represents a significant advancement in synthetic data generation. Its ability to create detailed, accurate 3D environments and scenarios is crucial for the development and enhancement of AI perception networks. Its extensibility and integration capabilities make it a valuable tool for researchers and engineers working in AI and deep learning, providing them with the resources to push the boundaries of what’s possible in AI model training and development.

Isaac Sim/Robotics Weekly Livestream: Synthetic Data Generation

For Synthetic data generation, Omniverse has the extension called replicator, and basically provides a means to collect data for learning. It can randomize objects in the scene, and use various annotaters to collect various data and provides writers how to write this data to either disk or the cloud
Core componenets is the sementic schema editor, so it is important to annotate things in your scene so you basically label them with their classes this also shows up in the forum that users don’t get data and that happens because the scene is not annotated.

Core components of a Replicator

Semantic Schema Editor

Important to annotate things in your scene, basically label them, sometimes shows up in the forum, because users don’t get data, but this is because the thing is not annotated.
Selecting an annotater we want to see => Insert segmentation which requires the scene to be enabled (ex. Instance Segmentation)
-> We can see that there will be an action. For every print, you will have a different color
If the scene is not labeled
Create a cube, show sementic segmentation, you do not get anything and then if we would not enable them, it appears!

Visualizer

Click button Sensors -> Can choose various annotaters what you want to visualize
-LdrColor(Can visualize color), Distance to image plane(depth), Distance to camera or focal point
-Whenever you get data from an annotator and you don’t know what shape or type it is, look over the documentation and it will provide all the information
-Annotator Documentation: Go over all annotators and it provides extra data and demos and what to expect the output data to look like
https://docs.omniverse.nvidia.com/extensions/latest/ext_replicator/annotators_details.html
-Annotator example
Ldr Color (Get channels(R,G,B,A), and what type of data will come out(np.uint8)

Annotator Registry

Replicator uses omnigraph to generate this. Many users did not really understand the workflow how this happens, so basically replicator replicator has the python script and and these scripts generate the graph which will then collect the data

Isaac Replicator

Builds on top of the replicator extension and has various example tutorials which makes more sense on the simulation part and also various UIs like helper functions or extensions built on top of replicator to annotate the data, it is important to have your scene labeled. UI helps automatically either frames, and then click add and it will label them given some rules or quickly annotate whole scenes given the prem names.
-Gives an example how you can traverse your stage and apply various roles on your scene to label it.

The Synthetic Data Recorder

Recorder uses a writer from Replicator, it wraps it as an extension and then in UI you can select various cameras the ovs you want to output, the number of frames, and it will generate images in the folder you want to do so. So this is a useful tool for quickly debugging on how the data looks like or testing how it will look like as an end product